Robot imitation learning method and device based on hybrid perception nonlinear model predictive control

Abstract

The application discloses a robot imitation learning method and device based on a hybrid perception nonlinear model predictive control, and the method comprises the following steps: collecting an external perception picture sequence and an ontology perception information sequence to obtain a first data set; sampling the first data set according to a preset field of view length; performing static kinematics coding and dynamic space-time coding processing on the ontology perception information sequence in the sampled data set to obtain a second data set; constructing a composite loss function; inputting the second data set into a prediction model, outputting a prediction sequence through a nonlinear model predictive control algorithm; according to the prediction sequence, a robot performs an action to complete an imitation learning task; and according to the composite loss function, the prediction model is updated, and the step of collecting the external perception picture sequence and the ontology perception information sequence to obtain the first data set is returned. The application can improve the reasoning speed and the imitation learning task accuracy, and can be widely applied to the technical field of robot imitation learning.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robot imitation learning, and in particular to a robot imitation learning method and device based on hybrid perception nonlinear model predictive control. BACKGROUND

[0002] In the field of machine learning, traditional methods rely heavily on manual programming to endow robots with the ability of autonomous behavior. These methods require complex, hand-crafted rules to determine the behavior of the machine and its dynamic changes in the environment. At the same time, mastering these hand-crafted rule programming methods requires a lot of time and professional coding knowledge. Therefore, imitation learning emerges in the field of machine learning - it adopts a learning-based paradigm to train a policy network to output actions in a specific task context without the need for complex manual coding. There is a self-recurrent modeling paradigm method in current robot imitation learning, which has a simple model structure and high computational efficiency, but performs poorly in long-term action planning. There is also a DP strategy that uses iterative gradient optimization to optimize the action distribution score function to generate relevant action sequences, but the multi-step denoising process indispensable in the DP model introduces a large computational overhead, which damages the real-time response capability and operational efficiency of the entire model. SUMMARY

[0003] Therefore, the embodiments of the present application mainly aim to provide a robot imitation learning method and device based on hybrid perception nonlinear model predictive control, in order to solve at least one of the problems in the prior art, and the present application can improve the reasoning speed and imitation learning task accuracy.

[0004] To achieve the above-mentioned purpose, one aspect of an embodiment of the present application provides a robot imitation learning method based on hybrid perception nonlinear model predictive control, which comprises:

[0005] Collecting external perception picture sequences and proprioceptive information sequences to obtain a first data set;

[0006] According to the pre-set field of view length, sampling the first data set to obtain a sampled data set;

[0007] Performing static kinematics coding and dynamic space-time coding processing on the proprioceptive information sequence in the sampled data set to obtain a second data set;

[0008] Constructing a composite loss function;

[0009] Inputting the second data set into a prediction model to output a prediction sequence through a nonlinear model predictive control algorithm;

[0010] According to the prediction sequence, the robot performs an action to complete the imitation learning task;

[0011] According to the composite loss function, the prediction model is updated, and the acquisition of the external perception picture sequence and the acquisition of the body perception information sequence are returned to obtain the first data set.

[0012] In some embodiments, after the first data set is sampled according to the preset field of view length to obtain a sampling data set, the method further includes:

[0013] According to the field of view length and the sampling data set, a label value is obtained.

[0014] In some embodiments, the body perception information sequence in the sampling data set is processed by static kinematics coding and dynamic space-time coding to obtain a second data set, including the following steps:

[0015] According to the mechanical structure of the robot, a dictionary vector is obtained;

[0016] Each element in the dictionary vector is mapped into a static coding vector;

[0017] The time dimension and the joint dimension of the body perception information sequence are operated by two-dimensional convolution to obtain dynamic coding features;

[0018] According to the sampling data set, the static coding vector and the dynamic coding features, the second data set is obtained.

[0019] In some embodiments, the composite loss function is constructed, including the following steps:

[0020] A state reference value and a control reference value are preset;

[0021] According to a first error of a first state value and the state reference value, a running loss function is constructed;

[0022] According to a second error of a first control value and the control reference value, a control loss function is constructed;

[0023] According to a third error of a final state value and the state reference value, a terminal loss function is constructed;

[0024] According to the running loss function, the control loss function and the terminal loss function, the composite loss function is obtained.

[0025] In some embodiments, the formula used for constructing the composite loss function includes:

[0026]

[0027] In the formula, L runningRepresents the operating cost; N represents the field of view length; x u (k) represents the discrete time t n+k The first state value measured below; x ref (n+k) represents the discrete time t n+k The state reference values ​​are: n = 0, 1, 2, ...; Q represents the first weight matrix.

[0028] In some embodiments, the formula used to construct the composite loss function includes:

[0029]

[0030] In the formula, L control Represents the control cost; N represents the field of view length; u(k) represents the distance at discrete time t. n+k The first control value below; u ref (n+k) represents the discrete time t n+k The control reference values ​​are: n = 0, 1, 2, ...; R represents the second weight matrix.

[0031] In some embodiments, the formula used to construct the composite loss function includes:

[0032]

[0033] In the formula, L terminal Represents terminal cost; N represents field of view length; x u (N) represents the discrete time t n+N The final state value measured below; x ref (n+N) represents the discrete time t n+N The state reference values ​​are: n = 0, 1, 2, ...; S represents the third weight matrix.

[0034] In some embodiments, updating the prediction model according to the composite loss function includes the following steps:

[0035] Based on the composite loss function, the predicted sequence and the label value are reconstructed using loss to obtain the composite loss value;

[0036] The prediction model is updated using the composite loss value.

[0037] To achieve the above objectives, another aspect of this invention proposes a robot imitation learning device based on hybrid sensing nonlinear model predictive control, the device comprising:

[0038] The first module is used to collect external perception image sequences and ontological perception information sequences to obtain the first dataset.

[0039] The second module is configured to sample the first data set according to a preset field of view length to obtain a sampled data set.

[0040] The third module is configured to perform static kinematic coding and dynamic spatio-temporal coding on the body perception information sequence in the sampled data set to obtain a second data set.

[0041] The fourth module is configured to construct a composite loss function.

[0042] The fifth module is configured to input the second data set into a prediction model to output a prediction sequence through a nonlinear model predictive control algorithm.

[0043] The sixth module is configured to perform an action according to the prediction sequence to complete an imitation learning task.

[0044] The seventh module is configured to update the prediction model according to the composite loss function and return to the step of collecting the external perception picture sequence and the body perception information sequence to obtain the first data set.

[0045] To achieve the above object, another aspect of the embodiment of the present application provides an electronic device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned robot imitation learning method based on hybrid perception nonlinear model predictive control when executing the computer program.

[0046] To achieve the above object, another aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program implements the above-mentioned robot imitation learning method based on hybrid perception nonlinear model predictive control when executed by a processor.

[0047] To achieve the above object, another aspect of the embodiment of the present application provides a computer program product or a computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device execute the above-mentioned robot imitation learning method based on hybrid perception nonlinear model predictive control.

[0048] The embodiments of the present application at least have the following beneficial effects: the present application provides a robot imitation learning method and device based on hybrid perception nonlinear model predictive control, which acquires an external perception picture sequence and an ontology perception information sequence to obtain a first data set; samples the first data set according to a preset field of view length to obtain a sampling data set; performs static kinematics coding and dynamic space-time coding processing on the ontology perception information sequence in the sampling data set to obtain a second data set; constructs a composite loss function; inputs the second data set into a prediction model, and outputs a prediction sequence through a nonlinear model predictive control algorithm; according to the prediction sequence, a robot performs an action to complete an imitation learning task; according to the composite loss function, the prediction model is updated, and the step of acquiring an external perception picture sequence and an ontology perception information sequence to obtain a first data set is returned. Through hybrid perception of the external perception picture sequence and the ontology perception information sequence, through the nonlinear model predictive control algorithm, the state sequence and the action sequence are simultaneously predicted in the imitation learning task, so as to improve the reasoning speed and the accuracy of the imitation learning task; and through the composite loss function and the embedded dynamic and static coding processing, it is ensured that the generated prediction action sequence is acceptable. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0050] Figure 1 is a flowchart of the robot imitation learning method based on hybrid perception nonlinear model predictive control provided by the embodiments of the present application;

[0051] Figure 2 is a schematic diagram of the problem of error accumulation that may occur in behavior cloning provided by the embodiments of the present application;

[0052] Figure 3 is a schematic diagram of the overall architecture based on hybrid perception nonlinear model predictive control model provided by the embodiments of the present application;

[0053] Figure 4 is a schematic diagram of the hardware structure of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0054] In order to make the purposes, technical solutions and advantages of the present application clearer, further detailed description will be given below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and are not intended to limit the present application. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as described in the appended claims.

[0055] It should be noted that although the functional modules are divided in the system schematic diagram, the logical sequence is shown in the flowchart, but in some cases, the steps shown or described can be performed in a manner different from the module division in the system or the sequence in the flowchart. The terms "first / S100", "second / S200" in the specification and claims and the above drawings can be used herein to describe various concepts, but unless specifically stated, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information. Depending on the context, the word "if" as used herein can be interpreted as "when" or "upon" or "in response to determining".

[0056] The terms "at least one", "multiple", "each", "any" and the like used in the present application include one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0057] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0058] Imitation learning enables robots to circumvent the need for manual programming or complex reward engineering by imitating expert demonstrations. In recent years, there is a paradigm of autoregressive modeling for robot imitation learning, which has a simple model structure and high computational efficiency, but performs poorly in long-term action planning. There is also a DP policy that generates relevant action sequences by iteratively optimizing the score function of the action distribution. By modeling the probability density, this method can significantly enhance the generation of multi-modal behavior and achieve stable control in high-dimensional action space. However, the multi-step denoising process indispensable in the DP model introduces a large computational overhead, which compromises the real-time responsiveness and operational efficiency of the entire model.

[0059] There are several factions under imitation learning, among which behavior cloning as a basic method directly maps observations (e.g., the state of a robot) to output results (the action the robot is about to perform), which is essentially a supervised learning. Although behavior cloning performs well on a series of real-world robot tasks (e.g., manipulation, autonomous driving, autonomous navigation), it has limitations in handling complex tasks, especially in compound error / covariate shift / error accumulation. Compound error is caused by the accumulation of errors in the sequential execution process, causing the agent to deviate from the training distribution. Behavior cloning has a problem of error accumulation, once a bad decision is made, it will enter a rare state, and the next decision will be worse, entering a vicious cycle. As shown in Figure 2 Figure 1 shows a simple introduction to the problem of error accumulation in behavior cloning for autonomous driving. Assuming that a policy is trained through a dataset and deployed on an autonomous driving system. The policy gives a suitable action output (controls the vehicle to drive normally) when encountering state s0. But in state s1, due to the "not-so-good" action output, the entire system enters a very rare state. "Rare" can be roughly understood as rarely / never appearing in the training dataset. In response to this rare state, the deployed policy does not have "enough experience" to give a reasonable action, so the output control action is worse, causing the entire system to enter a worse state, entering a vicious cycle.

[0060] Early methods to mitigate error accumulation, including interactive methods and synthetic correction generation, often face challenges in high-dimensional visual domains or require costly expert intervention. Recent popular methods focus on enhancing the expressiveness and robustness of behavior cloning: explicit policy modeling, action discretization, and solving multi-modal action distribution problems, but are sensitive to exponential growth of action space dimension and hyperparameters. Energy-based model implicit policy provides greater flexibility, but is unstable when specific optimization is performed. And in current imitation learning, methods that achieve high accuracy in tasks usually sacrifice inference speed, while fast and explicit policy modeling is not as good in precision in fine operations.

[0061] Therefore, as shown in Figure 1 The embodiment of the present application provides a robot imitation learning method based on a hybrid perception nonlinear model predictive control, which can include but is not limited to steps S100 to S700:

[0062] Step S100, collect external perception picture sequences and somatosensory information sequences to obtain a first data set;

[0063] Step S200, according to the pre-set field of view length, sampling the first data set to obtain a sampling data set;

[0064] In step S300, the somatosensory information sequence in the sampling data set is subjected to static kinematic coding and dynamic spatiotemporal coding processing to obtain a second data set.

[0065] In step S400, a composite loss function is constructed.

[0066] In step S500, the second data set is input into a prediction model, and a prediction sequence is output through a nonlinear model predictive control algorithm.

[0067] In step S600, the robot performs an action according to the prediction sequence to complete the imitation learning task.

[0068] In step S700, the prediction model is updated according to the composite loss function, and the steps of collecting the external perception picture sequence and the somatosensory information sequence to obtain the first data set are returned.

[0069] In steps S100 to S700 of some embodiments, through multi-modal perception of collecting the external perception picture sequence and the somatosensory information sequence, combined with nonlinear model predictive control, accurate (referring to high task success rate) and fast (referring to fast model inference speed) robot imitation learning is realized. And by introducing a specific effective loss function, the specific loss function is maximally reduced, and the learnable structure information is embedded through dynamic and static coding to ensure that the generated prediction action is acceptable.

[0070] In step S100 of some embodiments, in order to cope with the composite error in imitation learning, by integrating dual perception: the external perception captures the surrounding environment features, and the somatosensory perception encodes the internal state of the agent, that is, collecting the external perception picture sequence and the somatosensory information sequence, the first data set can be obtained. Through this hybrid method of integrating dual perception, the learned strategy can maintain a dynamic balance between environmental constraints and somatosensory motion, thereby significantly enhancing the operation robustness in the real world.

[0071] In step S200 of some embodiments, the field of view length is pre-set, and in data sampling, a segment of data with a length of the field of view length is taken from the external perception picture sequence and the somatosensory information sequence to obtain a sampling data set. After obtaining the sampling data set, a segment of data with a length of the field of view length after the sampling data set is obtained as a label value.

[0072] In some embodiments, step S300 can include but is not limited to steps S310 to S340:

[0073] In step S310, a dictionary vector is obtained according to the mechanical structure of the robot.

[0074] Step S320, mapping each element in the dictionary vector into a static code vector;

[0075] Step S330, performing convolution operation on the time dimension and joint dimension of the body perception information sequence by two-dimensional convolution to obtain dynamic code features;

[0076] Step S340, obtaining the second data set according to the sampling data set, the static code vector and the dynamic code features.

[0077] In step S310 of some embodiments, the dictionary vector is defined according to the mechanical structure of the robot. For example, according to the structure of the robot arm, a dictionary vector is defined to distinguish different robot arms and joints. For example, for two robot arms with 6 degrees of freedom, a 1x12 dictionary vector [0,0,0,0,0,0,1,1,1,1,1,1] is defined, where the first 6 zeros represent the joints of the first robot arm and the last 6 ones represent the joints of the second robot arm.

[0078] In step S320 of some embodiments, each element in the dictionary vector is mapped into a more complex static code vector. For example, element 0 representing the joints of the first robot arm is mapped to [a1, a2, …, a k ], and element 1 representing the joints of the second robot arm is mapped to [b1, b2, …, b k ]. Where the same element maps to the same vector.

[0079] In step S330 of some embodiments, in dynamic space-time coding, two-dimensional convolution is used on the time and joint trajectory of the body perception information sequence. For example, for two robot arms with 6 degrees of freedom, N time data is collected, and the overall data dimension is Nx12. The convolution kernel can slide in the time dimension and joint dimension, and can adaptively capture the dynamic relationship between joints and the change pattern over time. Through 2D convolution operation, dynamic code feature representation containing time information and joint coordination pattern can be extracted.

[0080] In step S340 of some embodiments, the static code vector can be concatenated with the body perception data in the sampling data set to form a new static code feature. The fusion of static code and dynamic code features can form a complete input feature representation, and the second data set is obtained.

[0081] By introducing a dynamic-static joint encoding scheme, the physical properties in modeling are enhanced. In static kinematic encoding, a fixed vector representation is encoded according to the hardware structure, preserving the structural prior; in dynamic spatio-temporal encoding, 2D convolution is adopted on the time and joint trajectory, which can adaptively capture the coordination patterns between joints. This dynamic-static joint encoding method combines physical prior and learned representation, so that the model can internalize the constraints of the robot form while maintaining the flexibility of unconstrained optimization.

[0082] In some embodiments, step S400 can include but is not limited to steps S410 to S450:

[0083] Step S410, presetting state reference values and control reference values;

[0084] Step S420, constructing a running loss function according to a first error of the first state value and the state reference value;

[0085] Step S430, constructing a control loss function according to a second error of the first control value and the control reference value;

[0086] Step S440, constructing a terminal loss function according to a third error of the final state value and the state reference value;

[0087] Step S450, obtaining the composite loss function according to the running loss function, the control loss function and the terminal loss function.

[0088] In step S410 of some embodiments, control reference values and time-varying state reference values are introduced.

[0089] In some embodiments, it is assumed that a controlled process is obtained, and at discrete time t n The state value x(n) is obtained by measurement, where n=0,1,2,…, and “controlled” means that a control input u(n) can be selected at each time to affect the future behavior of the system state. In tracking control, the task is to determine the control input u(n) so that x(n) follows the given control reference value x ref (n) as well as possible. Wherein, consider The time-varying state reference value x ref (n) is introduced, and u(n)=μ(x(n)) (this is a closed-loop feedback representation, which maps the required control result according to the current state value). In addition, the control reference value u ref is also introduced, so as to measure the error between the control value u and the control reference value. Wherein, X and U are arbitrary metric spaces, and the given nonlinear model predictive control feedback law μ:X→U.

[0090] In step S420 of some embodiments, a running loss function is constructed based on a first error between the first state value and the state reference value, thereby measuring system performance. Penalizing the error can make the state as close to the reference trajectory as possible. For example, in discrete time t... n+k The first state value x was obtained by measurement. u (k), the equivalent formulas for the label are: x u (k)=x u (k,x0) = x(n+k), where x0 = x(n) represents the current state, and the value of k is equivalent to focusing on the k-th predicted value based on the current time n, i.e., the prediction at time n+k. A discrete time t is introduced. n+k The state reference value x ref (n+k), constructing a running loss function to measure the distance between the first state value and the state reference value, we have the following calculation formula:

[0091]

[0092] In the formula, L running Represents the operating cost; N represents the field of view length; x u (k) represents the discrete time t n+k The first state value measured below; x ref (n+k) represents the discrete time t n+k The state reference value is given by n = 0, 1, 2, ...; Q represents the first weight matrix. The first weight matrix Q is determined by the x value at a single time step. u (k) In most cases, it is a vector, and the weighting matrix Q acts as an equivalent to x. u (k) vector and x ref The elements of the vector are weighted, and for convenience, they can actually be set as an identity matrix.

[0093] In step S430 of some embodiments, a control loss function is constructed based on the second error between the first control value and the control reference value. This function penalizes the control input value by controlling the cost to reduce actuator wear and energy consumption. For example, the error at discrete time t is obtained... n+k The first control value u(k) is given, where the value of k is equivalent to focusing on the k-th predicted value (i.e., the prediction at time n+k) based on the current time n. A discrete time t is also introduced. n+k The control reference value u ref (n+k), constructing a control loss function to measure the distance between the first control value and the control reference value, we have the following calculation formula:

[0094]

[0095] In the formula, L controlRepresents the control cost; N represents the field of view length; u(k) represents the distance at discrete time t. n+k The first control value below; u ref (n+k) represents the discrete time t n+k The control reference values ​​are given below; n = 0, 1, 2, ...; R represents the second weight matrix. The second weight matrix R, since u(k) at a single time step is mostly a vector, acts as a weighting matrix between the u(k) vector and u... ref The elements of the vector are weighted, and for convenience, they can actually be set as an identity matrix.

[0096] In step S440 of some embodiments, a terminal loss function is constructed based on the third error between the final state value and the state reference value. This function emphasizes the final state alignment of the execution through terminal cost, further highlighting the trajectory's objective in the short term. For example, in discrete time t... n+N The final state value x is obtained by measurement. u Similarly, the value of N is equivalent to focusing on the Nth predicted value based on the current time n, that is, the prediction at time n+N. And a discrete time t is introduced. n+N The state reference value x ref Given (n+N), construct the terminal loss function, and then we have the following calculation formula:

[0097]

[0098] In the formula, L terminal Represents terminal cost; N represents field of view length; x u (N) represents the discrete time t n+N The final state value measured below; x ref (n+N) represents the discrete time t n+N The state reference value is given by n = 0, 1, 2, ...; S represents the third weight matrix. The third weight matrix S is determined by the x value at a single time step. u (N) In most cases, it is a vector, and the weighting matrix S acts as an equivalent to x. u (N) vectors and x ref The elements of the vector are weighted, and for convenience, they can actually be set as an identity matrix.

[0099] In step S450 of some embodiments, a composite loss function can be obtained based on the running loss function, the control loss function, and the terminal loss function. The formula for calculating the composite loss function L is as follows:

[0100] L=λ1L running +λ2L terminal +L control

[0101] In the formula, λ1 and λ2 are weight parameters, which can be adjusted according to requirements, wherein λ2 should be slightly greater than λ1.

[0102] By designing an adaptive soft constraint construction method, that is, a composite loss function, the physical properties of the robot system and the interaction with the environment are analyzed and constrained, thereby laying a foundation for subsequent acquisition of acceptable actions (feasible region of action) and accessible states.

[0103] In step S500 of some embodiments, as shown in Figure 3 The Transformer-based decoder architecture is used as the backbone network of the prediction model, and the second data set obtained after dynamic and static coding and other processing is input into the prediction model. The prediction model processes a series of mixed perception states and action tokens, thereby outputting state predictions and corresponding action sequences generated in N steps. In each prediction cycle, a number of initial operation values (which can be artificially set) are used as control inputs for execution, realizing a closed-loop control paradigm similar to the principle of nonlinear model predictive control (NMPC).

[0104] In some embodiments, according to the rolling optimization of the NMPC algorithm, a multi-step “state-action” prediction is adopted: due to the requirement of time coherence optimization of NMPC, NMPC requires a field of view length of at least 2, so the minimum prediction range length is set to 2. The extended field of view promotes global sequence optimization, reduces control input oscillation and enhances the regularity of motion response. Since a single-step optimization strategy may cause excessive control / action value adjustment, resulting in discontinuous trajectories. Therefore, considering the inertia and state propagation characteristics of the physical system, multi-step prediction ensures the temporal consistency of the generated execution commands, thereby improving control accuracy and reducing energy consumption.

[0105] In the NMPC algorithm, the predicted state and action values are tightly coupled because the rolling horizon optimization process of NMPC is based on the process mathematical model (x + = f(x, u), which represents the state x + at the next time point, which depends on the state value x and the control value u at the current time point), that is, the tight connection between the state and control values in the process mathematical model, so that they are optimized simultaneously to produce reliable control output. The embodiments of the present application use a unified prediction framework, that is, simultaneous prediction of state and action. Through the prediction model based on the backbone network of transformer, the connection between state and action values is established, and the composite loss function constraint is cooperated to generate a coherent estimate of the state and action sequence, thereby ensuring more accurate and stable final control (action) values.

[0106] In step S600 of some embodiments, the imitation learning task is completed according to the output predicted state and action sequence, and acting on the robot to drive the robot to perform the corresponding action.

[0107] In some embodiments, step S700 can include but is not limited to steps S710-S720:

[0108] Step S710, according to the composite loss function, loss reconstruction is performed on the predicted sequence and the label value to obtain a composite loss value;

[0109] Step S720, update the prediction model through the composite loss value.

[0110] In steps S710-S720 of some embodiments, the physical properties of the robot system and the interaction with the environment are strictly analyzed and constrained through the adaptive soft constraint of the composite loss function. The composite loss function can be used to model the predicted sequence more accurately, so that the model parameters are optimized by loss reconstruction of the predicted sequence and the label value through composite loss function calculation, and updating the prediction model parameters through back propagation.

[0111] As shown in Figure 3 Taking a set of system equipment (including an externally erected RGB-D camera, a six-degree-of-freedom robot arm, and a one-degree-of-freedom end effector (gripper)) as an example, the robot imitation learning process based on the hybrid perception nonlinear model predictive control includes:

[0112] Step 1: Collect the data set required for imitation learning, which includes 50 episodes, each episode has a length of 400 frames, and each frame contains the observed RGBD image at the current time (state 1), the body state of the robot arm and the gripper at the current time (state 2), wherein the state value can also be directly used as a control value to control the motion of the agent.

[0113] Step 2: Set the field of view length to N, in a certain data sampling, take a length of N input data in an episode as the input of the model (equivalent to input the external perception picture sequence and the body perception information sequence), and obtain the data with a length of N after the data as the label value (the picture sequence in the label data is regarded as the future state 1, and the body state sequence of the robot arm and the gripper is regarded as the future state 2 and the future control value).

[0114] Step 3: Input the length N input data into the model to obtain an output length N predicted sequence, and then perform composite loss function calculation on the predicted sequence and the label value, and optimize the model parameters through back propagation.

[0115] The embodiment of the present application also provides a robot imitation learning device based on a hybrid perception nonlinear model predictive control, which can realize the robot imitation learning method based on the hybrid perception nonlinear model predictive control.

[0116] The first module is configured to collect an external perception picture sequence and an ontology perception information sequence to obtain a first data set;

[0117] The second module is configured to sample the first data set according to a preset field of view length to obtain a sampling data set;

[0118] The third module is configured to perform static kinematics coding and dynamic space-time coding processing on the ontology perception information sequence in the sampling data set to obtain a second data set;

[0119] The fourth module is configured to construct a composite loss function;

[0120] The fifth module is configured to input the second data set into a prediction model, and output a prediction sequence through a nonlinear model predictive control algorithm;

[0121] The sixth module is configured to make a robot perform an action according to the prediction sequence to complete an imitation learning task;

[0122] The seventh module is configured to update the prediction model according to the composite loss function, and return to the step of collecting the external perception picture sequence and the ontology perception information sequence to obtain the first data set.

[0123] It can be understood that the contents in the above method embodiments are all applicable to the present device embodiment, the present device embodiment specifically realizes the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0124] The embodiment of the present application also provides an electronic device, which comprises a processor and a memory, the memory stores a computer program, and the processor realizes the robot imitation learning method based on the hybrid perception nonlinear model predictive control when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer and the like.

[0125] It can be understood that the contents in the above method embodiments are all applicable to the present device embodiment, the present device embodiment specifically realizes the same functions as the above method embodiments, and achieves the same beneficial effects as the above method embodiments.

[0126] Reference Figure 4 , Figure 4 The hardware structure of the electronic device of another embodiment is illustrated, and the electronic device comprises:

[0127] The processor 801 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0128] The memory 802 can be implemented by a ROM (Read Only Memory), a static storage device, a dynamic storage device, or a RAM (Random Access Memory), and the like. The memory 802 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 802 and are called and executed by the processor 801 to implement the robot imitation learning method based on the hybrid perception nonlinear model predictive control according to the embodiments of the present application.

[0129] The input / output interface 803 is configured to realize information input and output.

[0130] The communication interface 804 is configured to realize the communication interaction between the device and other devices. The communication can be realized by a wired manner (for example, a USB, a network cable, or the like) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, or the like).

[0131] The bus 805 is configured to transmit information between the components (for example, the processor 801, the memory 802, the input / output interface 803, and the communication interface 804) of the device.

[0132] The processor 801, the memory 802, the input / output interface 803, and the communication interface 804 are connected to each other through the bus 805 to realize the communication connection between the components in the device.

[0133] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the robot imitation learning method based on the hybrid perception nonlinear model predictive control.

[0134] It can be understood that the contents in the above method embodiments are all applicable to the present storage medium embodiments. The present storage medium embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0135] The embodiment of the present application further provides a computer program product or computer program, which comprises computer instructions stored in a computer readable storage medium. A processor of a computer device can read the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the robot imitation learning method based on the hybrid perception nonlinear model predictive control.

[0136] In summary, the robot imitation learning method and device based on the hybrid perception nonlinear model predictive control have the following advantages:

[0137] 1. The embodiment of the present application is used for robot imitation learning, and the rolling optimization idea in the NMPC is fused and implemented, so that the state sequence and the action sequence are simultaneously predicted (unified prediction framework) in the imitation learning task, and efficient and high-performance agent operation is realized.

[0138] 2. The embodiment of the present application proposes a soft constraint method with strong generalization: without directly describing the complex situation of the physical constraint in the real world, the specific loss function (essentially a penalty function method, a specific effective loss function is introduced) is reduced to the greatest extent, and the learnable structure information (dynamic and static coding) is embedded to ensure that the generated predicted action is acceptable.

[0139] In some alternative embodiments, the functions / operations mentioned in the block diagram can not occur in the order mentioned in the operation diagram. For example, depending on the functions / operations involved, two blocks shown in succession can actually be executed substantially simultaneously or the blocks can sometimes be executed in reverse order. In addition, the embodiments presented and described in the flowcharts of the present application are provided by way of example, and the purpose is to provide a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logical flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and in which sub-operations described as part of larger operations are independently executed.

[0140] Furthermore, although the present application is described in the context of functional modules, it is to be understood that one or more of the described functions and / or features can be integrated in a single physical device and / or software module, or one or more functions and / or features can be implemented in separate physical devices or software modules. It will also be appreciated that detailed discussion of the actual implementation of each module is not necessary to an understanding of the application. Rather, the actual implementation of the modules, in combination with their attributes, functions, and internal relationships, are to be understood within the context of the devices disclosed herein. Thus, those skilled in the art with access to patents, scientific journals, and other public sources known by those skilled in the art will be able, using ordinary skill, to practice the application as set forth in the claims without undue experimentation. It is also to be understood that the specific concepts disclosed are merely illustrative and that the scope of the present application is to be determined by the entire scope of the claims, along with all equivalents of the claims and their equivalents.

[0141] If the functions are implemented in software, the functions can be stored in or implemented as one or more computer program products, which can be incorporated into a computer- readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this document, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. The computer-readable medium can be, for example but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium.

[0142] The logic and / or steps represented in the flowcharts and / or described herein, for example, can be considered as a sequence of instructions to implement logic functions, and can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor- containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. In the context of this specification, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0143] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection (electronic) having one or more wires, a portable computer diskette (magnetic), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can also be paper or another suitable medium upon which the program is printed, as the program can be electronically captured, for example, via optical scanning of the paper or other medium, then compiled, interpreted, or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

[0144] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof. In the embodiments described above, various steps or methods can be implemented, for example, by software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, and in another embodiment, any of the following techniques can be used to implement the hardware used in the described embodiments: discrete logic circuitry having logic gates for implementing logic functions upon data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and so forth.

[0145] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" and the like means that the specific feature, structure, material or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the application. Illustrative expressions of the above terms in the specification do not necessarily refer to the same embodiment or example. Also, the specific feature, structure, material or characteristic described can be combined in any appropriate manner in one or more embodiments or examples.

[0146] Although embodiments of the application have been shown and described, it would be recognized by those of ordinary skill in the art that various changes, modifications, substitutions, and alterations can be made hereto without departing from the spirit and scope of the application. The scope of the application is limited only by the claims and the equivalents thereof.

[0147] The above is a specific description of the preferred embodiments of the present application, but the present application is not limited to the described embodiments, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present application, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A robot imitation learning method based on a hybrid perception nonlinear model predictive control, characterized in that, The method comprises the following steps: Collecting an external perception picture sequence and an ontology perception information sequence to obtain a first data set; Sampling the first data set according to a preset field of view length to obtain a sampling data set; Performing static kinematics coding and dynamic space-time coding processing on the ontology perception information sequence in the sampling data set to obtain a second data set; Constructing a composite loss function; Inputting the second data set into a prediction model to output a prediction sequence through a nonlinear model predictive control algorithm; According to the prediction sequence, the robot performs an action to complete an imitation learning task; According to the composite loss function, the prediction model is updated, and the step of collecting an external perception picture sequence and an ontology perception information sequence to obtain a first data set is returned.

2. The robot imitation learning method based on the hybrid perception nonlinear model predictive control according to claim 1, wherein, After the step of sampling the first data set according to a preset field of view length to obtain a sampling data set, the method further comprises the following steps: According to the field of view length and the sampling data set, a label value is obtained.

3. The robot imitation learning method based on hybrid perception nonlinear model predictive control according to claim 1, wherein, The step of performing static kinematics coding and dynamic space-time coding processing on the ontology perception information sequence in the sampling data set to obtain a second data set comprises the following steps: Obtaining a dictionary vector according to the mechanical structure of the robot; Mapping each element in the dictionary vector into a static coding vector; Performing a convolution operation on the time dimension and joint dimension of the ontology perception information sequence through two-dimensional convolution to obtain a dynamic coding feature; According to the sampling data set, the static coding vector and the dynamic coding feature, the second data set is obtained.

4. The robot imitation learning method based on hybrid perception nonlinear model predictive control according to claim 1, wherein, The step of constructing a composite loss function comprises the following steps: Pre-setting a state reference value and a control reference value; According to a first error of a first state value and the state reference value, an operation loss function is constructed; According to a second error of a first control value and the control reference value, a control loss function is constructed; According to a third error of a final state value and the state reference value, a terminal loss function is constructed; According to the operation loss function, the control loss function and the terminal loss function, the composite loss function is obtained.

5. The robot imitation learning method based on hybrid perception nonlinear model predictive control according to claim 1, wherein, The formula used in the step of constructing a composite loss function comprises: where L running represents the running cost; N represents the horizon length; x u (k) represents the first state value measured at discrete time t n+k (k); x ref (n+k) represents the state reference value at discrete time t n+k (n+k); n = 0, 1, 2,...; Q represents the first weight matrix.

6. The robot imitation learning method based on hybrid perception nonlinear model predictive control according to claim 1, wherein, The formula used in the step of constructing a composite loss function comprises: where L control represents the control cost; N represents the horizon length; u(k) represents the first control value at discrete time t n+k k; u ref (n+k) represents the control reference value at discrete time t n+k n = 0, 1, 2,...; and R represents the second weight matrix.

7. The robot imitation learning method based on hybrid perception nonlinear model predictive control according to claim 1, wherein, The formula used in the step of constructing a composite loss function comprises: In the formula, L terminal Represents terminal cost; N represents field of view length; x u (N) represents the discrete time t n+N The final state value measured below; x ref (n+N) represents the discrete time t n+N The state reference values ​​are: n = 0, 1, 2, ...; S represents the third weight matrix.

8. The robot imitation learning method based on hybrid perception nonlinear model predictive control according to claim 2, wherein, The step of updating the prediction model according to the composite loss function comprises the following steps: According to the composite loss function, a composite loss value is obtained by loss reconstruction of the prediction sequence and the label value; The prediction model is updated through the composite loss value.

9. A robot imitation learning device based on a hybrid perception nonlinear model predictive control, characterized by, The method comprises the following steps: A first module is configured to collect an external perception picture sequence and an ontology perception information sequence to obtain a first data set; A second module is configured to sample the first data set according to a preset field of view length to obtain a sampling data set; A third module is configured to perform static kinematics coding and dynamic space-time coding processing on the ontology perception information sequence in the sampling data set to obtain a second data set; A fourth module is configured to construct a composite loss function; A fifth module for inputting the second data set into a prediction model, outputting a prediction sequence through a nonlinear model predictive control algorithm; A sixth module for performing an action by a robot according to the prediction sequence to complete an imitation learning task; A seventh module for updating the prediction model according to the composite loss function and returning the steps of collecting the external perception picture sequence and the proprioceptive information sequence to obtain the first data set.

10. An electronic device, comprising: comprise a processor and a memory; The memory is used to store a program; The processor executes the program to implement the method of any one of claims 1 to 8.

Images