Method and electronic device for realizing robot imitation learning trajectory

Abstract

The present application discloses a method and an electronic device for implementing a robot imitation learning trajectory, the method comprising: obtaining an initial trajectory, and obtaining a basis function weight group corresponding to the initial trajectory; generating a reward corresponding to the initial trajectory; generating a current basis function weight group according to the basis function weight group corresponding to the initial trajectory and the corresponding reward; generating a current trajectory according to the current basis function weight group and a dynamic motion primitive algorithm; generating a reward corresponding to the current trajectory; generating a first weight gradient group according to a basis function weight group corresponding to a plurality of first trajectories selected from a plurality of target trajectories that the robot has already run; in response to the first weight gradient group meeting a preset requirement, stopping the iteration process and generating a final trajectory; otherwise, updating the current basis function weight group, and returning to execute the step of generating the current trajectory according to the current basis function weight group and a dynamic motion primitive algorithm. The method of the present application can improve the efficiency of the robot imitation learning trajectory.

Description

Technical Field

[0001] The present application relates to the field of robots, and in particular to a method and electronic device for implementing robot imitation learning trajectories. Background Art

[0002] In recent years, robots have been gradually applied to various fields, such as industrial fields and daily life fields. The widespread application of robots also requires robots to have higher intelligence, such as being able to complete more complex tasks by learning skills. Imitation learning is an effective method to simplify robot skill learning, which can avoid complex manual programming. The inventor of this application found that the current robot imitation learning process is inefficient and needs further improvement. Summary of the invention

[0003] The present application provides a method and electronic device for implementing a robot imitation learning trajectory, which can improve the efficiency of the robot imitation learning trajectory.

[0004] A first aspect of an embodiment of the present application provides a method for implementing a robot imitation learning trajectory, the method comprising: obtaining an initial trajectory, and encoding the initial trajectory through a dynamic motion primitive algorithm to obtain a basis function weight group corresponding to the initial trajectory, the basis function weight group including multiple basis function weight values; generating a reward corresponding to the initial trajectory, wherein the reward corresponding to the initial trajectory represents the probability of the robot completing the target task after moving along the initial trajectory; generating a current basis function weight group based on the basis function weight group corresponding to the initial trajectory and the reward corresponding to the initial trajectory, the current basis function weight group including multiple basis function weight values; generating a current trajectory based on the current basis function weight group and the dynamic motion primitive algorithm; generating a reward corresponding to the current trajectory, wherein the reward corresponding to the current trajectory represents the probability of the robot completing the target task after moving along the current trajectory The probability of the target task; according to the rewards corresponding to the multiple target trajectories that the robot has already run, a plurality of first trajectories are screened out from the multiple target trajectories; according to the basis function weight groups corresponding to the screened first trajectories, a first weight gradient group is generated, and the first weight gradient group includes a plurality of first weight gradient values; whether the first weight gradient group meets the preset requirements is judged; in response to the first weight gradient group meeting the preset requirements, the current trajectory is determined as the final trajectory; otherwise, a first noise item is obtained, a second weight gradient group is generated according to the first weight gradient group and the first noise item, and then the current basis function weight group of the current trajectory is updated according to the second weight gradient group, and the step of generating the current trajectory according to the current basis function weight group and the dynamic motion primitive algorithm is returned to execute; wherein, the first noise items obtained twice adjacently are different.

[0005] A second aspect of an embodiment of the present application provides an electronic device, which includes a processor, a memory, and a communication circuit, wherein the processor is coupled to the memory and the communication circuit, respectively, and program data is stored in the memory. The processor implements the steps in the above method by executing the program data in the memory.

[0006] A third aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to implement the steps in the above method.

[0007] The beneficial effect is: the present application uses a dynamic motion primitive algorithm to parameterize the trajectory strategy and obtains a basis function weight group corresponding to the trajectory. At the same time, in the process of iteratively updating the basis function weight group, multiple target trajectories that the robot has already run are screened according to the rewards corresponding to the trajectory, and the target trajectories that the robot has already run are selectively reused, which can improve the efficiency of the robot's imitation learning trajectory. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work, among which:

[0009] Figure 1 This is a schematic diagram of the state of the robot of the present application before it moves;

[0010] Figure 2 yes Figure 1 Schematic diagram of the robot's state after movement;

[0011] Figure 3 It is a flowchart of an implementation method of the present application for implementing a robot imitation learning trajectory;

[0012] Figure 4 It is a flowchart of another implementation method of the method for implementing robot imitation learning trajectory in the present application;

[0013] Figure 5 It is a flowchart of another implementation method of the method for implementing robot imitation learning trajectory in the present application;

[0014] Figure 6 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0015] Figure 7It is a structural schematic diagram of an implementation method of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0017] It should be noted that the terms "first" and "second" in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" can explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0018] The purpose of the method for implementing robot imitation learning trajectory in this application is to allow the robot to continuously imitate and learn, and finally obtain an ideal trajectory that can complete the target task. Figure 1 As shown, a cup is fixed at the end of the robot, and a small ball is connected to the bottom of the cup by a rope. The purpose of this application is to finally obtain an ideal trajectory in which the robot can make the ball fall into the cup, wherein the state of the ball falling into the cup is as shown in FIG. Figure 2 shown.

[0019] See also Figure 3 , Figure 3 This is a flow chart of an implementation method of a robot imitation learning trajectory in the present application, the method comprising:

[0020] S110: Acquire an initial trajectory, and encode the initial trajectory using a dynamic motion primitive algorithm to obtain a basis function weight group corresponding to the initial trajectory.

[0021] The robot does not need to complete the task after executing the initial trajectory, e.g. Figure 1 and Figure 2In the application scenario, after the robot moves along the initial trajectory, the ball may not fall into the cup. In one embodiment, the initial trajectory may be generated by teaching, specifically, the operator manually drags the robot to move to generate the teaching trajectory, and the teaching trajectory is determined as the initial trajectory; in another embodiment, the initial trajectory may also be obtained by programming, that is, a human expert generates a programming trajectory by programming, and the programming trajectory is determined as the initial trajectory. In short, this application does not limit the generation process of the initial trajectory.

[0022] At the same time, the Dynamic Movement Primitives (DMP) algorithm is a method for trajectory imitation learning. With its highly nonlinear characteristics and high real-time performance, it is applied to various fields of robotics. The model of the Dynamic Motion Primitives algorithm framework originally came from the second-order spring damping system. The characteristic of the entire system is to converge toward the target position. However, although the second-order spring damping system can make the system converge to the target state, it cannot control the convergence process quantity, such as the shape of the trajectory. Therefore, a nonlinear function is superimposed on the second-order spring damping system to control the convergence process quantity. In other words, the introduction of a nonlinear function can change the shape of the trajectory to the desired shape, where the nonlinear function The expression is as follows:

[0023]

[0024] in, As the basis function, in general, the Gaussian basis function can be used. is the weight corresponding to the basis function, is the number of basis functions.

[0025] From the above content, it can be seen that after each basis function and the corresponding weight are given, the final trajectory can be weighted to obtain. Conversely, after the trajectory and each basis function are given, the weights corresponding to each basis function can be generated.

[0026] Therefore, after obtaining the initial trajectory, the dynamic motion primitive algorithm can be used to encode and decompose the initial trajectory to obtain the weights of each basis function, and the array composed of the weights corresponding to each basis function is determined as the basis function weight group corresponding to the initial trajectory, that is, the basis function weight group corresponding to the initial trajectory includes multiple function weight values.

[0027] S120: Generate a reward corresponding to the initial trajectory, wherein the reward corresponding to the initial trajectory represents the probability of the robot completing the target task after moving along the initial trajectory.

[0028] Specifically, the robot is controlled to automatically move along the initial trajectory, and then the probability of the robot completing the target task after moving along the initial trajectory is determined according to the result of the robot movement, and then the reward corresponding to the initial trajectory is determined according to the probability. The process of determining the probability of the robot completing the target task can be: according to the distance between the position of the TCP point after the robot moves along the initial trajectory and the target position, the probability of the robot completing the target task is determined, for example, the smaller the distance, the greater the probability, or the probability of the robot completing the target task can also be determined according to the gap between the final state and the initial state of the target object involved in the robot's movement process along the initial trajectory. For example, if the robot's task is to rotate the handle and open the door, the probability of the robot completing the target task can be determined according to the degree of closure of the door at the end.

[0029] In one embodiment, the greater the probability that the robot completes the target task after moving along the initial trajectory, the greater the corresponding reward, that is, the reward is positively correlated with the probability of completing the target task; in another embodiment, the greater the probability that the robot completes the target task after moving along the initial trajectory, the smaller the corresponding reward, that is, the reward is negatively correlated with the probability of completing the target task. For the sake of convenience, the following description is based on the assumption that the greater the probability that the robot completes the target task after moving along the initial trajectory, the greater the corresponding reward.

[0030] The rules for generating the rewards corresponding to the initial trajectory can be preset, for example, Figure 1 and Figure 2 In the application scenario, after the robot moves along the initial trajectory, the height of the ball from the plane where the cup mouth is located in the vertical direction and the distance from the center of the cup mouth in the horizontal plane can be obtained during the movement of the ball, and the reward corresponding to the initial trajectory can be determined based on the obtained height and distance.

[0031] It can be understood that the reward corresponding to the initial trajectory can also represent the degree to which the initial trajectory is close to the ideal trajectory. For example, when the reward is positively correlated with the probability of completing the target task, the larger the reward corresponding to the initial trajectory, the closer the initial trajectory is to the ideal trajectory.

[0032] S130: Generate a current basis function weight group according to the basis function weight group corresponding to the initial trajectory and the reward corresponding to the initial trajectory.

[0033] It can be understood that the reward corresponding to the initial trajectory represents the degree to which the initial trajectory is close to the ideal trajectory, and can also reflect the degree to which the basis function weight group corresponding to the initial trajectory is close to the basis function weight group corresponding to the ideal trajectory. Therefore, according to the reward corresponding to the initial trajectory, the basis function weight group can be updated to generate a new basis function weight (i.e., the current basis function weight group), so that the new basis function weight group is getting closer and closer to the basis function weight group corresponding to the ideal trajectory. Among them, the current basis function weight also includes multiple basis function weight values, and the multiple basis function weight values ​​in the current basis function weight correspond to the multiple basis function weight values ​​in the basis function weight group corresponding to the initial trajectory one-to-one, and the two one-to-one corresponding basis function weight values ​​are the weights of the same basis function.

[0034] In one embodiment, an amplitude group including multiple amplitudes can be generated according to the size of the reward (the multiple amplitudes correspond to the multiple basis function weight values ​​in the basis function weight group corresponding to the initial trajectory one by one), and then the corresponding amplitudes are added to the multiple basis function weight values ​​in the basis function weight group corresponding to the initial trajectory to obtain multiple new basis function weight values, which constitute the current basis function weight group. In one embodiment, the larger the reward corresponding to the initial trajectory, the smaller the average value of the corresponding multiple amplitudes generated.

[0035] The reward may be substituted into a pre-fitted function to obtain multiple amplitudes, or multiple amplitudes corresponding to the reward may be generated according to a pre-established corresponding relationship.

[0036] Of course, in other implementations, step S130 may also use other specific methods to generate the current basis function weight group, which may be described below for details.

[0037] S140: Generate a current trajectory according to the current basis function weight group and the dynamic motion primitive algorithm.

[0038] Specifically, the current trajectory is obtained by weighting according to each basis function and the current basis function weight group.

[0039] S150: Generate a reward corresponding to the current trajectory, wherein the reward corresponding to the current trajectory represents the probability of the robot completing the target task after moving along the current trajectory.

[0040] Specifically, the robot is controlled to automatically move along the current trajectory, and then the reward corresponding to the current trajectory is determined according to the probability of the robot completing the target task after the movement. The process of generating the reward corresponding to the current trajectory is the same as the process of generating the reward corresponding to the initial trajectory in step S120. For details, please refer to the above-mentioned related content.

[0041] S160: Filtering out a plurality of first trajectories from the plurality of target trajectories according to rewards corresponding to the plurality of target trajectories that the robot has already run.

[0042] In one embodiment, all trajectories that the robot has run can be determined as target trajectories, that is, step S160 is: according to the rewards corresponding to each trajectory, a number of trajectories are selected from all trajectories that the robot has run as the first trajectory.

[0043] In one embodiment, when the reward corresponding to the trajectory is positively correlated with the probability of the robot completing the target task after moving along the trajectory, step S160 includes: according to the rewards corresponding to multiple target trajectories, a first number of target trajectories with the largest corresponding rewards are selected from the multiple target trajectories as a plurality of first trajectories.

[0044] Specifically, the larger the reward, the greater the possibility that the robot will complete the target task after moving along the corresponding target trajectory, and the basis function weight group corresponding to the target trajectory is more conducive to the iterative convergence process. Therefore, from multiple target trajectories, N target trajectories with the largest rewards are screened, and these N target trajectories are all determined as the first trajectories, where N is an integer greater than or equal to 2, and its value can be set according to actual needs, for example, N is equal to 2, 5, 10 or a larger value.

[0045] In another embodiment, the process of step S160 may further include: calculating the reward average value according to the rewards corresponding to the multiple target trajectories; and selecting the trajectories whose corresponding rewards are greater than or equal to the reward average value from the multiple target trajectories as the first trajectories. Specifically, at this time, the reward average value is calculated first, and then the target trajectories whose corresponding rewards are greater than or equal to the reward average value are determined as the first trajectories.

[0046] In another embodiment, the process of step S160 may further include: selecting a target trajectory whose corresponding reward is greater than a reward threshold from a plurality of target trajectories as a first trajectory, wherein the reward threshold is preset and can be adjusted according to demand.

[0047] In summary, the present application does not limit the specific process of how to select a plurality of first trajectories from a plurality of target trajectories in step S160.

[0048] S170: Generate a first weight gradient group according to the basis function weight groups corresponding to the selected plurality of first trajectories.

[0049] Specifically, the first weight gradient group includes multiple first weight gradient values, and the multiple first weight gradient values ​​correspond to multiple basis functions one by one. The meaning of the first weight gradient value is the change in the weight corresponding to the corresponding basis function. Weight learning is performed based on the basis function weight group corresponding to the selected first trajectories to generate a first weight gradient group, wherein the generated first weight gradient group needs to satisfy: when the reward corresponding to the trajectory is positively correlated with the probability of the robot completing the target task after moving along the trajectory, the reward corresponding to the current trajectory in the next iteration process is greater than the reward corresponding to the current trajectory in the current iteration process.

[0050] In one embodiment, step S170 specifically includes: generating a first weight gradient group according to a basis function weight group corresponding to the selected plurality of first trajectories and an expectation-maximization algorithm.

[0051] Specifically, the expectation-maximum algorithm (EM) is a type of optimization algorithm that performs maximum likelihood estimation (MLE) through iteration. The standard calculation framework of the EM algorithm consists of alternating E-step (Expectation-step) and M-step (Maximization step). The convergence of the algorithm can ensure that the iteration at least approaches the local maximum. By using the EM algorithm to calculate the first weight gradient group, the efficiency can be improved and the accuracy of the algorithm can be guaranteed.

[0052] S180: Determine whether the first weight gradient group meets preset requirements.

[0053] The preset requirement is a preset convergence condition, which can be set in advance by the designer according to the requirements. If it is determined that the first weight gradient group can meet the preset requirement, it is determined that the basis function weight group reaches a convergence state, and the iteration process is no longer performed, that is, step S190 is executed, but if it is determined that the first weight gradient group cannot meet the preset requirement, step S200 is executed.

[0054] Among them, the preset requirement can be that multiple first weight gradient values ​​in the first weight gradient group are all less than a threshold, or that the average value of multiple first weight gradient values ​​is less than a threshold, and details can be seen in the following introduction.

[0055] S190: Determine the current trajectory as the final trajectory.

[0056] Specifically, at this time, the current trajectory can be directly determined as the final trajectory, or the first noise term can be obtained, and a second weight gradient group can be generated according to the first weight gradient group and the first noise term, and then the current basis function weight group is updated again according to the second weight gradient group, and the trajectory corresponding to the updated current basis function weight group is determined, and finally the trajectory is determined as the final trajectory. The process of the second weight gradient group is described below.

[0057] S200: Acquire a first noise term, generate a second weight gradient group according to the first weight gradient group and the first noise term, and then update a current basis function weight group of the current trajectory according to the second weight gradient group.

[0058] Specifically, in order to avoid the same current trajectory generated twice in adjacent times, the present application introduces a first noise term. The first noise terms obtained in two adjacent times are different. Therefore, even if the first weight gradient group generated in two adjacent iterations is the same, because of the existence of the first noise term, the second weight gradient group generated in the two adjacent iterations is different, thereby ensuring that the current basis function weights of the current trajectory in two adjacent iterations are different, and ultimately ensuring that the current trajectory in two adjacent iterations is different, avoiding falling into an infinite loop.

[0059] Among them, assuming that the first weight gradient group includes N first weight gradient values, the second weight gradient group will also include N weight gradient values. If the first weight gradient group is represented by a vector L and the second weight gradient group is represented by a vector P (vector L and vector P are both N-dimensional vectors), the relationship between the first weight gradient group L and the second weight gradient group P can be expressed by the following formula:

[0060] M=N+f(e)

[0061] Among them, f(e) is the noise function, which generates an N-dimensional vector, that is, the first noise term.

[0062] It can be understood that the meaning of the second weight gradient group is also the weight change amount, so according to the current basis function weight group and the second weight gradient group, a new basis function weight group can be generated, and the new basis function weight group continues to be the current basis function weight group. Among them, the multiple basis function weight values ​​in the current basis function weight group are added to the multiple weight gradient values ​​in the second weight gradient group one by one, so as to obtain multiple new basis function weight values, and the multiple new basis function weight values ​​constitute a new basis function weight group.

[0063] After executing step S200, return to execute step S140. In the process of returning to execute, continue to generate new trajectories and calculate the rewards of the new trajectories, then generate new weight gradient groups, and continue to judge whether the new weight gradient groups meet the preset requirements. If they do, the convergence is completed. If not, continue to repeatedly update the current basis function weight group, and continue to iterate until the latest weight gradient group can meet the preset requirements.

[0064] See also Figure 4 In another embodiment of the present application, a method for implementing a robot imitation learning trajectory includes:

[0065] S110: Acquire an initial trajectory, and encode the initial trajectory using a dynamic motion primitive algorithm to obtain a basis function weight group corresponding to the initial trajectory.

[0066] S120: Generate a reward corresponding to the initial trajectory, wherein the reward corresponding to the initial trajectory represents the probability of the robot completing the target task after moving along the initial trajectory.

[0067] S130: Generate a current basis function weight group according to the basis function weight group corresponding to the initial trajectory and the reward corresponding to the initial trajectory.

[0068] S140: Generate a current trajectory according to the current basis function weight group and the dynamic motion primitive algorithm.

[0069] S150: Generate a reward corresponding to the current trajectory, wherein the reward corresponding to the current trajectory represents the probability of the robot completing the target task after moving along the current trajectory.

[0070] S160: Filtering out a plurality of first trajectories from the plurality of target trajectories according to rewards corresponding to the plurality of target trajectories that the robot has already run.

[0071] S170: Generate a first weight gradient group according to the basis function weight groups corresponding to the selected plurality of first trajectories.

[0072] S180: Determine whether the first weight gradient group meets preset requirements.

[0073] If it is determined that the first weight gradient group can meet the preset requirement, step S190 is executed, but if it is determined that the first weight gradient group cannot meet the preset requirement, step S210 is executed.

[0074] S190: Determine the current trajectory as the final trajectory.

[0075] S210: Determine whether the currently screened first trajectories include the current trajectory.

[0076] If the judgment result is not included, step S200 is executed; if the judgment result is included, step S220 is executed.

[0077] S200: Acquire a first noise term, generate a second weight gradient group according to the first weight gradient group and the first noise term, and then update a current basis function weight group of the current trajectory according to the second weight gradient group.

[0078] After executing step S200 , the process returns to executing step S140 .

[0079] S220: Update the current basis function weight group of the current trajectory according to the first weight gradient group.

[0080] After executing step S220, the process returns to executing step S140.

[0081] Specifically, Figure 3 The difference between the implementation modes is that, in this implementation mode, before step S200, it is also determined whether the currently selected first trajectories include the current trajectory. If they do, it means that the calculation process of the first weight gradient group is related to the current trajectory, and the current basis function weight group of the current trajectory can be directly updated according to the first weight gradient group. However, if the currently selected first trajectories do not include the current trajectory, in order to avoid the current trajectory being determined twice in an adjacent manner being the same, the second weight gradient group is first determined, and then the current basis function weight group of the current trajectory is updated according to the second weight gradient group.

[0082] Among them, the process of updating the current basis function weight group of the current trajectory according to the first weight gradient group is similar to the process of updating the current basis function weight group of the current trajectory according to the second weight gradient group. For example, the multiple basis function weight values ​​in the current basis function weight group are added to the multiple weight gradient values ​​in the first weight gradient group one by one to obtain multiple new basis function weight values, and the multiple new basis function weight values ​​constitute a new basis function weight group.

[0083] In one embodiment, the process of step S180 determining whether the first weight gradient group meets the preset requirements includes: determining whether each first weight gradient value in the first weight gradient group is less than its corresponding gradient threshold; if so, determining that the first weight gradient group meets the preset requirements, otherwise, determining that the first weight gradient group does not meet the preset requirements.

[0084] Specifically, each first weight gradient value corresponds to a gradient threshold, and the gradient thresholds corresponding to different first weight gradient values ​​may be equal or may not be equal, and can be set according to actual needs.

[0085] If each first weight gradient value in the first weight gradient group is less than the corresponding gradient threshold, it means that even if the current basis function weight group is updated according to the first weight gradient group and the corresponding trajectory is obtained, the reward corresponding to the trajectory will not have much room for improvement, and the first weight gradient group is determined to meet the preset requirements, that is, the convergence is determined to be completed, otherwise it is determined that the first weight gradient group does not meet the preset requirements, and the iteration process continues. The gradient threshold is preset.

[0086] In other embodiments, the process of determining whether the first weight gradient group meets the preset requirements in step S180 may also be: determining whether each first weight gradient value in the first weight gradient group is less than the corresponding gradient threshold value (the preset threshold value may be equal to or different from the above-mentioned gradient threshold value) for H consecutive times, if the determination result is yes, then determining that the first weight gradient group meets the preset requirements, otherwise, determining that the first weight gradient group does not meet the preset requirements. Wherein H is an integer greater than the latter and equal to 2, which can be specifically set according to requirements.

[0087] From the above content, it can be seen that the present application uses a dynamic motion primitive algorithm to parameterize the trajectory strategy and obtain a basis function weight group corresponding to the trajectory. At the same time, in the process of iteratively updating the basis function weight group, the robot's multiple target trajectories that have been run are screened according to the rewards corresponding to the trajectory, and the robot's target trajectories that have been run are selectively reused, which can improve the efficiency of the robot's imitation learning trajectory.

[0088] In one embodiment, step S130 includes:

[0089] S131: applying a plurality of second noise terms to the basis function weight groups corresponding to the initial trajectory to generate a plurality of noise weight groups.

[0090] Specifically, each second noise term includes a plurality of second noise values, and the plurality of second noise values ​​correspond one to one with the basis function weight values ​​in the basis function weight group. Among them, the plurality of second noise terms can be randomly generated, for example, the plurality of second noise values ​​in each second noise term can be generated using a noise function, and the plurality of second noise values ​​in the second noise term can also be generated according to a normal distribution. Specifically, the plurality of second noise values ​​in the second noise term conform to a normal distribution. In summary, the present application does not impose any specific restrictions on the method for generating the plurality of second noise terms.

[0091] For each second noise term, multiple basis function weight values ​​in the basis function weight group corresponding to the initial trajectory are added to the second noise values ​​corresponding to the second noise term to obtain a new weight group, namely the noise weight group.

[0092] The above steps are performed for each second noise item, thereby obtaining multiple noise groups.

[0093] It can be understood that the multiple second noise terms are not the same.

[0094] S132: Generate multiple test trajectories according to multiple noise weight groups and dynamic motion primitive algorithms respectively.

[0095] Specifically, for each noise weight group, a test trajectory is generated according to the noise weight group and the dynamic motion primitive algorithm, thereby obtaining a test trajectory corresponding to each noise weight. The process of generating the test trajectory is the same as the process of generating the current trajectory, and details can be found in the above-mentioned related content.

[0096] S133: Generate rewards corresponding to each test trajectory respectively, wherein the rewards corresponding to the test trajectory represent the probability of the robot completing the target task after moving along the test trajectory.

[0097] Specifically, for each test trajectory, a corresponding reward is generated, wherein the process of generating the reward corresponding to the test trajectory is the same as the process of generating the reward corresponding to the current trajectory and generating the reward corresponding to the initial trajectory. For details, please refer to the above-mentioned related content.

[0098] S134: Filtering out a plurality of second trajectories from the initial trajectory and the plurality of test trajectories according to the reward corresponding to the initial trajectory and the reward corresponding to each test trajectory.

[0099] Specifically, the selected second trajectories may include the initial trajectory and the test trajectory, or may only include the test trajectory without the initial trajectory.

[0100] In one embodiment, the process of screening the second trajectory in step S134 is similar to the process of screening the first trajectory in step S160 .

[0101] For example, in one embodiment, when the reward corresponding to the trajectory is positively correlated with the probability of the robot completing the target task after moving along the trajectory, step S134 includes: according to the rewards corresponding to the multiple trajectories that the robot has already run, from the multiple trajectories that the robot has already run, select the second number of trajectories with the largest corresponding rewards as a number of second trajectories, that is, from the initial trajectory and multiple test trajectories, select L target trajectories with the largest rewards, and determine these L target trajectories as second trajectories, where L is an integer greater than or equal to 2, and its value can be set according to actual needs, for example, L is equal to 2, 5, 10 or a larger value. At the same time, the second number can be equal to the above-mentioned first number, or it can be unequal.

[0102] For another example, in another embodiment, when the reward corresponding to the trajectory is positively correlated with the probability of the robot completing the target task after moving along the trajectory, step S134 includes: calculating an average value based on the rewards corresponding to multiple trajectories that the robot has already run; and selecting, from the multiple trajectories that the robot has already run, trajectories whose corresponding rewards are greater than or equal to the average value as several first trajectories.

[0103] In one embodiment, the strategy for selecting trajectories in step S134 and step S160 is the same, for example, both select several trajectories with the largest corresponding rewards.

[0104] S135: Generate a third weight gradient group according to the basis function weight groups corresponding to the selected plurality of second trajectories.

[0105] The process of generating the third weight gradient group is the same as the process of generating the first weight gradient group, and reference may be made to the above-mentioned related contents.

[0106] S136: Obtain a current basis function weight group according to the basis function weight group corresponding to the initial trajectory and the third weight gradient group.

[0107] Specifically, the third weight gradient group includes multiple gradient values ​​corresponding to multiple basis functions one by one, and the meaning of the gradient value in the third weight gradient is also the weight change amount. Therefore, according to the basis function weight group corresponding to the initial trajectory and the third weight gradient group, a new basis function weight group can be generated, that is, the current basis function weight group. This process is the same as the process of generating the current basis function weight group mentioned above, and the details can be referred to the above related content.

[0108] In one embodiment, before step S160, the method further includes: determining the first track and the current track selected in the most recent track selection as the target track. Specifically, the first tracks selected in the previous selection are all tracks whose basis function weight groups are beneficial to the iterative process. Therefore, in order to reduce the algorithm processing amount and avoid re-selection, the track selected in the previous selection and the current track are determined as the target track before step S160, and then step S160 continues to select tracks in the track selected in the previous selection and the current track.

[0109] It should be noted that, when the first plurality of first trajectories are screened for the first time in the whole process, the first plurality of trajectories may be screened from all trajectories that the robot has already run, that is, the plurality of target trajectories that the robot has already run at this time are all trajectories that the robot has already run.

[0110] In another embodiment, before step S160, it can also be: first select M trajectories with the largest corresponding rewards from all trajectories that the robot has run except the current trajectory, and then determine these M trajectories and the current trajectory as target trajectories, where M is an integer greater than the latter and equal to 2.

[0111] In another embodiment, before step S160, the method may further include: selecting the most recently run P trajectories from the trajectories that the robot has already run as target trajectories, where P is an integer greater than or equal to 2.

[0112] In summary, there is no restriction on the specific process of determining the target trajectory.

[0113] To better understand the solution of this application, see Figure 5 The solution of the present application is described in detail below with reference to a specific example. In this example, the greater the probability that the robot completes the target task after moving along the trajectory, the greater the corresponding reward. The method in this example includes:

[0114] S1: Determine the teaching trajectory generated by teaching as the initial trajectory, or determine the programming trajectory generated by programming as the initial trajectory.

[0115] S2: Encode the initial trajectory through the dynamic motion primitive algorithm to obtain the basis function weight group corresponding to the initial trajectory.

[0116] S3: Control the robot to move along the initial trajectory, and generate a reward corresponding to the initial trajectory based on the probability of the robot completing the target task after movement.

[0117] S4: applying a plurality of second noise terms to the basis function weight group corresponding to the initial trajectory to generate a plurality of noise weight groups, and generating a test trajectory corresponding to each noise weight group, and then generating a reward corresponding to each test trajectory respectively.

[0118] S5: According to the reward corresponding to each trajectory, select N trajectories with the largest corresponding rewards from multiple test trajectories and initial trajectories.

[0119] S6: Generate a third weight gradient group according to the basis function weight groups corresponding to the screened N trajectories and using the EM algorithm.

[0120] S7: Obtain a current basis function weight group according to the basis function weight group corresponding to the initial trajectory and the third weight gradient group.

[0121] S8: Generate the current trajectory according to the current basis function weight group and the dynamic motion primitive algorithm.

[0122] S9: Control the robot to move along the current trajectory, and generate a reward corresponding to the current trajectory based on the robot's movement results.

[0123] S10: From the N trajectories selected last time and the current trajectory, select the N trajectories with the largest rewards.

[0124] S11: Generate a first weight gradient group according to the basis function weight group corresponding to the latest selected N trajectories and using the EM algorithm.

[0125] S12: Determine whether each first weight gradient value in the first weight gradient group is less than the corresponding gradient threshold.

[0126] If it is determined that each first weight gradient value in the first weight gradient group is smaller than the corresponding gradient threshold, step S13 is executed; otherwise, step S14 is executed.

[0127] S13: Determine that convergence is complete, determine the current trajectory as the final trajectory, or obtain the first noise term, generate a second weight gradient group based on the first weight gradient group and the first noise term, then update the current basis function weight group based on the second weight gradient group, and generate the final trajectory based on the current basis function weight group.

[0128] S14: Obtain the first noise term, generate a second weight gradient group according to the first weight gradient group and the first noise term, then update the current basis function weight group according to the second weight gradient group, and return to execute step S8.

[0129] See also Figure 6 , Figure 6 1 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 200 includes a processor 210, a memory 220 and a communication circuit 230. The processor 210 is coupled to the memory 220 and the communication circuit 230 respectively. The memory 220 stores program data. The processor 210 implements the steps in any of the above-mentioned implementation methods by executing the program data in the memory 220. The detailed steps can be found in the above-mentioned implementations and will not be repeated here.

[0130] The electronic device 200 may be any device with algorithm processing capability, such as a computer, a mobile phone, a robot control cabinet, etc., and is not limited here.

[0131] See also Figure 7 , Figure 7 The computer-readable storage medium 400 stores a computer program 410, which can be executed by a processor to implement the steps in any of the above methods.

[0132] The computer-readable storage medium 400 may specifically be a device that can store the computer program 410, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or may be a server that stores the computer program 410. The server may send the stored computer program 410 to other devices for execution, or may also execute the stored computer program 410 by itself.

[0133] The above description is only an embodiment of the present application and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for implementing robot imitation learning trajectory, characterized in that: The method comprises: Acquire an initial trajectory, and encode the initial trajectory through a dynamic motion primitive algorithm to obtain a basis function weight group corresponding to the initial trajectory, wherein the basis function weight group includes a plurality of basis function weight values; Generate a reward corresponding to the initial trajectory, wherein the reward corresponding to the initial trajectory represents the probability that the robot completes the target task after moving along the initial trajectory; Generate a current basis function weight group according to the basis function weight group corresponding to the initial trajectory and the reward corresponding to the initial trajectory, wherein the current basis function weight group includes a plurality of basis function weight values; Generate a current trajectory according to the current basis function weight group and the dynamic motion primitive algorithm; Generate a reward corresponding to the current trajectory, wherein the reward corresponding to the current trajectory represents the probability that the robot completes the target task after moving along the current trajectory; According to rewards corresponding to the multiple target trajectories that the robot has run, select a plurality of first trajectories from the multiple target trajectories; Generate a first weight gradient group according to the selected basis function weight groups corresponding to the first trajectories, wherein the first weight gradient group includes a plurality of first weight gradient values; Determining whether the first weight gradient group meets preset requirements; In response to the first weight gradient group satisfying the preset requirement, determining the current trajectory as a final trajectory; Otherwise, obtain a first noise term, generate a second weight gradient group according to the first weight gradient group and the first noise term, then update the current basis function weight group of the current trajectory according to the second weight gradient group, and return to execute the step of generating the current trajectory according to the current basis function weight group and the dynamic motion primitive algorithm; The first noise items obtained twice consecutively are different.

2. The method according to claim 1, characterized in that The reward corresponding to the target trajectory is positively correlated with the probability of the robot completing the target task after moving along the target trajectory; The step of selecting a plurality of first trajectories from the plurality of target trajectories according to the rewards corresponding to the plurality of target trajectories that the robot has already run comprises: According to the rewards corresponding to the plurality of target trajectories, a first number of target trajectories corresponding to the largest rewards are selected from the plurality of target trajectories as the plurality of first trajectories.

3. The method according to claim 1, characterized in that The step of generating a current basis function weight group according to the basis function weight group corresponding to the initial trajectory and the reward corresponding to the initial trajectory comprises: Applying a plurality of second noise terms to the basis function weight groups corresponding to the initial trajectory to generate a plurality of noise weight groups; Generate a plurality of test trajectories according to the plurality of noise weight groups and the dynamic motion primitive algorithm respectively; Generate a reward corresponding to each of the test trajectories respectively, wherein the reward corresponding to the test trajectory represents the probability that the robot completes the target task after moving along the test trajectory; According to the reward corresponding to the initial trajectory and the reward corresponding to each of the test trajectories, a plurality of second trajectories are selected from the initial trajectory and the plurality of test trajectories that the robot has run; generating a third weight gradient group according to the selected basis function weight groups corresponding to the plurality of second trajectories; The current basis function weight group is obtained according to the basis function weight group corresponding to the initial trajectory and the third weight gradient group.

4. The method according to claim 3, characterized in that The reward corresponding to the trajectory is positively correlated with the probability that the robot completes the target task after moving along the trajectory; The step of selecting a plurality of second trajectories from the plurality of trajectories that the robot has already run according to the rewards corresponding to the plurality of trajectories that the robot has already run comprises: According to the rewards corresponding to the multiple trajectories that the robot has already run, a second number of trajectories with the largest corresponding rewards are selected from the multiple trajectories that the robot has already run as the multiple second trajectories.

5. The method according to claim 1, characterized in that Before selecting a plurality of first trajectories from the plurality of target trajectories according to the rewards corresponding to the plurality of target trajectories that the robot has run, the method further includes: The plurality of first trajectories screened out during the most recent trajectory screening and the current trajectory are determined as the target trajectory.

6. The method according to claim 1, characterized in that The step of generating a first weight gradient group according to the basis function weight groups corresponding to the selected plurality of first trajectories comprises: The first weight gradient group is generated according to the basis function weight groups corresponding to the selected plurality of first trajectories and an expectation maximization algorithm.

7. The method according to claim 1, characterized in that The step of determining whether the first weight gradient group meets preset requirements includes: Determine whether each of the first weight gradient values ​​in the first weight gradient group is less than the corresponding gradient threshold; If it is less than, it is determined that the first weight gradient group meets the preset requirement; otherwise, it is determined that the first weight gradient group does not meet the preset requirement.

8. The method according to claim 1, characterized in that The step of obtaining the initial trajectory includes: Determining a teaching trajectory generated when the robot is subjected to drag teaching as the initial trajectory; Alternatively, a programmed trajectory generated in a programmable manner is determined as the initial trajectory.

9. The method according to claim 1, characterized in that: Before obtaining the first noise term and generating the second weight gradient group according to the first weight gradient group and the first noise term, the method further includes: Determining whether the plurality of first trajectories currently screened out include the current trajectory; If the current trajectory is not included, performing the step of obtaining the first noise term and generating the second weight gradient group according to the first weight gradient group and the first noise term; Otherwise, the current basis function weight group of the current trajectory is updated according to the first weight gradient group, and the process returns to the step of generating the current trajectory according to the current basis function weight group and the dynamic motion primitive algorithm.

10. An electronic device, characterized in that: The electronic device includes a processor, a memory and a communication circuit, the processor is coupled to the memory and the communication circuit respectively, the memory stores program data, and the processor implements the steps in the method as described in any one of claims 1 to 9 by executing the program data in the memory.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program can be executed by a processor to implement the steps in the method according to any one of claims 1 to 9.

Images