Robot control model training method, robot control method and device

Abstract

This invention discloses a training method for a robot control model, a robot control method, and an apparatus. The method includes: acquiring a demonstration trajectory dataset of a robot, the demonstration trajectory data including a demonstration action sequence and a visual state sequence; identifying key action segments and non-key action segments from the demonstration action sequence; reconstructing the supervised action ground truth mapped to the visual state at each time step in each non-key action segment as the demonstration action corresponding to the starting time step of the next temporally adjacent key action segment; inputting the visual state sequence into a prediction model and outputting the corresponding predicted action sequence, remapping the predicted action corresponding to the time step of the non-key action segment as the predicted action corresponding to the starting time step of the next temporally adjacent key action segment; and training the prediction model by imitation learning based on the supervised action ground truth mapped to the visual state at each time step in the reconstructed visual state sequence and the predicted action sequence to obtain a robot control model.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and specifically to a method for training a robot control model, a robot control method, a device, an electronic device, and a storage medium. Background Technology

[0002] In recent years, visual imitation learning and vision-language-action (VLA) models have made significant progress in the field of robot operation. End-to-end robot strategies typically employ dense control interfaces, outputting new low-level action instructions at each environmental control step.

[0003] In related technologies, robots often need to go through a large number of free space smooth motion stages in long-term tasks. These stages are not sensitive to high-frequency control, but related technical strategies still need to execute a large number of low-information actions step by step, resulting in redundant execution steps and low efficiency. Moreover, dense behavior cloning has a covariate offset problem, and small local prediction errors will accumulate continuously during long-term rolling execution, reducing the success rate and robustness of the task. Although some methods introduce keyframes or action anchor point structures, dense connection trajectories still need to be generated and executed between key segments, which cannot truly reduce the redundant execution caused by low-information motion. Summary of the Invention

[0004] This invention provides a method for training a robot control model, a robot control method, a device, an electronic device, and a storage medium, aiming to improve the problems of low robot control execution efficiency and low task success rate in related technologies.

[0005] In a first aspect, the present invention provides a method for training a robot control model, the method comprising: Obtain a demonstration trajectory dataset of the robot. Each demonstration trajectory data in the demonstration trajectory dataset includes a demonstration action sequence and a visual state sequence. The demonstration action sequence includes the demonstration action of the robot at each time step, and the visual state sequence includes the visual state of the robot at each time step. Identify key and non-key action segments from the demonstrated action sequence; The supervised action ground truth mapped to the visual state at each time step in each non-critical action segment is reconstructed into the demonstration action corresponding to the starting time step in the next temporally adjacent critical action segment, while keeping the supervised action ground truth mapped to the visual state at each time step in each critical action segment unchanged. The visual state sequence in each demonstration trajectory data is input into the prediction model, and the corresponding predicted action sequence is output. The predicted action corresponding to the time step of the visual state sequence belonging to the key action segment is the predicted action corresponding to the time step output by the prediction model. The predicted action corresponding to the time step of the non-key action segment is remapped to the predicted action corresponding to the starting time step of the next key action segment that is temporally adjacent. The prediction model is trained by imitation learning based on the supervised action ground truth and predicted action sequence mapped by the visual state at each time step after reconstruction in the visual state sequence, to obtain the robot control model.

[0006] In some embodiments, determining key and non-key action segments from the demonstration action sequence includes: A sliding window of fixed time length is used to slide over the demonstration action sequence. A short-time discrete cosine transform is performed on the current sliding window in the demonstration action sequence to obtain the transform coefficients of the current sliding window. The spectral energy corresponding to each sliding window is determined based on the transformation coefficients corresponding to each sliding window in the demonstration action sequence. The high-frequency energy ratio of each sliding window is calculated based on the spectral energy corresponding to each sliding window in the demonstration action sequence. The high-frequency energy ratio is the ratio of the sum of the spectral energies of the corresponding frequency points in the sliding window that are greater than the preset frequency point value to the sum of the spectral energies of each frequency point in the sliding window. For each sliding window in the demonstration action sequence, if the proportion of high-frequency energy in the sliding window is less than a preset ratio, the sliding window is determined as a non-critical action segment. If the proportion of high-frequency energy in the sliding window is greater than or equal to a preset proportion, the sliding window will be divided into key action segments.

[0007] In some embodiments, determining key and non-key action segments from the demonstration action sequence includes: A sliding window of fixed duration is used to slide over the demonstration action sequence; The straight line connecting the trajectory point corresponding to the first time step and the trajectory point corresponding to the last time step in the current sliding window is used as the reference trajectory. Calculate the vertical distance from the trajectory point corresponding to each time step within the current sliding window to the reference trajectory; Normalize the vertical distances of the trajectory points at each time step within the current sliding window to obtain the normalized distances of the trajectory points at each time step within the current sliding window. Calculate the normalized average distance of the trajectory points corresponding to each time step within the current sliding window, and use it as the trajectory curvature of the current sliding window; Sort the trajectory curvature of each sliding window in descending order, and divide the demonstration action subsequences corresponding to the first preset number of sliding windows into key action segments. The remaining sliding windows are divided into non-critical action segments, corresponding to the demonstration action subsequences.

[0008] In some embodiments, inputting the visual state sequence from each demonstration trajectory data into the prediction model and outputting the corresponding predicted action sequence includes: For each demonstration trajectory data visual state sequence, the visual state at each time step in the visual state sequence is sequentially input into the prediction model for action prediction; If the current time step is located in a predetermined key action segment, the predicted action currently output by the prediction model will be used as the predicted action corresponding to the current time step. If the current time step is located in a predetermined non-critical action segment, the predicted action currently output by the prediction model is remapped to the predicted action corresponding to the starting time step of the next critical action segment that is temporally adjacent to the non-critical action segment. Finally, the predicted action sequence corresponding to the visual state sequence is obtained.

[0009] In some embodiments, the visual state corresponding to each time step includes the robot's visual image and the robot's physical state.

[0010] In some embodiments, the demonstration action corresponding to each time step includes the robot end-effector pose, end-effector speed, and gripper state.

[0011] Secondly, the present invention provides a robot control method based on a robot control model, wherein the robot control model is trained using the aforementioned robot control model training method, and the robot control method includes: Obtain the robot's current visual state at the current time step; The current visual state is input into the trained robot control model, and the robot control model predicts and outputs the robot's next time step action. Control the robot to execute the action at the next time step.

[0012] Thirdly, the present invention provides a training device for a robot control model, the device comprising: The acquisition module is used to acquire a robot demonstration trajectory dataset. Each demonstration trajectory data in the demonstration trajectory dataset includes a demonstration action sequence and a visual state sequence. The demonstration action sequence includes the robot's demonstration action at each time step, and the visual state sequence includes the robot's visual state at each time step. An action recognition module is used to identify key action segments and non-key action segments from the demonstration action sequence; The supervision target reconstruction module is used to reconstruct the supervision action truth value mapped by the visual state of each time step in each non-critical action segment into the demonstration action corresponding to the starting time step in the next temporally adjacent critical action segment, while keeping the supervision action truth value mapped by the visual state of each time step in each critical action segment unchanged. The prediction module is used to input the visual state sequence in each demonstration trajectory data into the prediction model and output the corresponding predicted action sequence; wherein, the predicted action corresponding to the time step belonging to the key action segment in the visual state sequence is the predicted action corresponding to the time step output by the prediction model, and the predicted action corresponding to the time step belonging to the non-key action segment is remapped to the predicted action corresponding to the starting time step of the next temporally adjacent key action segment. The training module is used to perform imitation learning training on the prediction model based on the supervised action ground value and predicted action sequence mapped by the visual state at each time step after reconstruction in the visual state sequence, so as to obtain the robot control model.

[0013] Fourthly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform a training method for a robot control model according to any embodiment of the first aspect, or to perform a robot control method based on a robot control model according to any embodiment of the second aspect.

[0014] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a computer to execute a training method for a robot control model according to any embodiment of the first aspect above, or to execute a robot control method based on a robot control model according to any embodiment of the second aspect above.

[0015] According to the robot control model training method of the present invention, before motion prediction training, key motion segments and non-key motion segments are identified in the demonstration motion sequence, and the supervised motion ground truth of the time step in the non-key motion segment is reconstructed into the demonstration motion of the starting time step in the adjacent key motion segment. During motion prediction training, the predicted motion corresponding to the time step of the non-key motion segment is remapped to the predicted motion corresponding to the starting time step of the adjacent key motion segment. This allows the model to learn to predict and output fine control actions in key motion segments, and in non-key motion segments, instead of outputting fragmented control actions step by step, it directly outputs the control action pointing to the starting time step of the next key motion segment. Thus, in practical robot control applications, the robot can be controlled to skip low-information motions, concentrate the control budget on key interaction stages such as contact, grasping, and alignment, effectively reduce redundant execution steps, improve the robot's task execution efficiency, effectively reduce the risk of error accumulation, and improve the task success rate. Attached Figure Description

[0016] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating a method for training a robot control model according to an embodiment of the present invention; Figure 2 This is a flowchart illustrating an implementation method for determining key and non-key action segments from a sequence of exemplary actions in an embodiment of the present invention. Figure 3 This is a flowchart illustrating another method for determining key and non-key action segments from a demonstration action sequence in an embodiment of the present invention. Figure 4 A block diagram illustrating the composition of a robot control model training device provided in an embodiment of the present invention; Figure 5 A flowchart illustrating a robot control method based on a robot control model provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] In related technologies, robots often need to go through a large number of free space smooth motion stages in long-term tasks. These stages are not sensitive to high-frequency control, but related technical strategies still need to execute a large number of low-information actions step by step, resulting in redundant execution steps and low efficiency. Moreover, dense behavior cloning has a covariate offset problem, and small local prediction errors will accumulate continuously during long-term rolling execution, reducing the success rate and robustness of the task. Although some methods introduce keyframes or action anchor point structures, dense connection trajectories still need to be generated and executed between key segments, which cannot truly reduce the redundant execution caused by low-information motion.

[0020] In view of this, embodiments of the present invention provide a method for training a robot control model, a robot control method, a device, an electronic device, and a storage medium, which aim to effectively improve the problems of low robot control execution efficiency and low task success rate in related technologies.

[0021] According to an embodiment of the present invention, a method for training a robot control model is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0022] This invention provides a method for training a robot control model, applied to a server. Figure 1 This is a flowchart illustrating a method for training a robot control model according to an embodiment of the present invention, as shown below. Figure 1 As shown, the training method for the robot control model includes the following steps S101 to S105.

[0023] Step S101: Obtain the robot's demonstration trajectory dataset. Each demonstration trajectory data in the demonstration trajectory dataset includes a demonstration action sequence and a visual state sequence. The demonstration action sequence includes the robot's demonstration action at each time step, and the visual state sequence includes the robot's visual state at each time step.

[0024] A demonstration trajectory dataset refers to one or more sets of complete operation records of a target task performed by a human or expert robot. It serves as a source of training samples for robot imitation learning. The demonstration trajectory dataset includes multiple demonstration trajectory data, each of which corresponds to a complete task execution process (e.g., a grasping, an assembly, or a placement).

[0025] Demonstration trajectory data refers to all time-series data from the start to the end of a single complete teaching process, including two pairs of sequences: demonstration action sequence and visual state sequence, which correspond one-to-one in time and are aligned step by step.

[0026] A demonstration sequence of actions is an ordered sequence of actions that a robot performs sequentially over time during the teaching process. The term "sequence" indicates that the actions are arranged in chronological order.

[0027] A time step is a point in time after the continuous teaching process is discretized at fixed time intervals. For example, sampling is performed every 20ms, and each sampling is a time step. The entire task is composed of a series of time steps.

[0028] The demonstration action refers to the true value of the robot's actual action given by the expert at a certain time step. It is a supervisory signal for imitation learning and is also referred to as the supervisory action true value in this embodiment of the invention.

[0029] A visual state sequence is an ordered sequence of the environment and its own state observed by the robot over time during the teaching process, and it is also arranged in chronological order.

[0030] Visual state refers to all the state information that a robot can observe at a certain time step. It usually includes two parts: external visual information: scene images, depth maps, target object positions, poses, etc. captured by the camera; and body state information: the robot's own position, joint angles, end effector pose, motion speed, and other self-states.

[0031] Visual states serve as input observations to the policy network of the prediction model, enabling the robot to make action decisions based on "what it sees".

[0032] Step S102: Identify key and non-key action segments from the demonstration action sequence.

[0033] By analyzing the demonstration action sequence, key action segments and non-key action segments are identified. Each key action segment consists of a demonstration action at one time step or multiple consecutive time steps in the demonstration action sequence, and each non-key action segment consists of a demonstration action at one time step or multiple consecutive time steps in the demonstration action sequence. Each time step in the demonstration action sequence is labeled as a key action segment or a non-key action segment, thus completing the division of key action segments and non-key action segments in the demonstration action sequence.

[0034] In this embodiment of the invention, non-critical action segments are low-information smooth motion regions in the demonstration action sequence, while critical action segments are interactively dense critical action regions in the demonstration action sequence.

[0035] Step S103: Reconstruct the supervised action ground truth value mapped to the visual state of each time step in each non-critical action segment into the demonstration action corresponding to the starting time step in the next temporally adjacent critical action segment, while keeping the supervised action ground truth value mapped to the visual state of each time step in each critical action segment unchanged.

[0036] Understandably, the key action segments and non-key action segments obtained from the demonstration action sequence are also arranged in chronological order. For each non-key action segment in the demonstration action sequence, the supervised action ground truth value mapped by the visual state of each time step in the non-key action segment is reconstructed into the demonstration action corresponding to the starting time step in the next temporally adjacent key action segment.

[0037] For example, if the sequence of demonstration actions is divided into {non-critical action segment 1, critical action segment 1, non-critical action segment 2, critical action segment 2, ..., non-critical action segment n, ..., critical action segment n}, then the supervised action ground truth (demonstration action) corresponding to the visual state of each time step in non-critical action segment 1 is reconstructed into the demonstration action corresponding to the starting time step of the next critical action segment that is temporally adjacent to non-critical action segment 1. The next critical action segment that is temporally adjacent to non-critical action segment 1 is the critical action segment 1.

[0038] For each key action segment in the demonstration action sequence, the supervised action ground value (demonstration action) mapped to the visual state at each time step in each key action segment remains unchanged.

[0039] Step S104: Input the visual state sequence in each demonstration trajectory data into the prediction model and output the corresponding predicted action sequence. The predicted action corresponding to the time step of the visual state sequence belonging to the key action segment is the predicted action corresponding to the time step output by the prediction model. The predicted action corresponding to the time step of the non-key action segment is remapped to the predicted action corresponding to the starting time step of the next temporally adjacent key action segment.

[0040] In this embodiment of the invention, the prediction model refers to the policy network of robot imitation learning, which is used to infer the action to be performed based on the visual state observed at each time step. For example, the prediction model is a model based on visual imitation learning, or a vision-language-action (VLA) model.

[0041] For each visual state sequence of the demonstration trajectory data, the visual state at each time step in the visual state sequence is sequentially input into the prediction model for action prediction. The prediction model performs forward reasoning step by step based on the input visual state sequence and outputs a set of predicted action sequences that are the same length as the input visual state sequence and correspond one-to-one in time.

[0042] If the current time step belongs to the time step in the aforementioned key action segment, then the predicted action corresponding to the current time step remains unchanged as the predicted action output by the prediction model for the current time step.

[0043] If the current time step belongs to the time step of the aforementioned non-critical action segment, then the predicted action corresponding to the current time step output by the prediction model will be relabeled as the predicted action corresponding to the starting time step of the next critical action segment that is temporally adjacent to the non-critical action segment.

[0044] After relabeling and remapping the predicted actions at each time step within the non-critical action segment, the final predicted action sequence is output, which includes the predicted action corresponding to each time step.

[0045] Step S105: Based on the supervised action ground truth and predicted action sequence mapped by the visual state at each time step in the visual state sequence, the prediction model is trained by imitation learning to obtain the robot control model.

[0046] Using the real values ​​of the supervised actions at each time step after remapping as the learning target, and the predicted action sequence output by the model as the network output, the prediction model is optimized through error backpropagation, and finally a robot control model that can be directly used for actual robot control is obtained.

[0047] Specifically, a training loss function is constructed with the ground truth of the supervised action at each time step as the objective and the predicted action at each time step as the output. The difference between the predicted action and the corresponding ground truth of the supervised action is calculated step by step to form the training error. Based on the training error, the gradient is backpropagated through the optimizer to update the model parameters, so that the predicted action continuously approaches the ground truth of the supervised action. When the model converges and the error between the predicted action sequence and the ground truth of the supervised action reaches a preset threshold, the training is completed, and the robot control model is obtained. The trained robot control model can be deployed to a physical robot and directly output stable and task-focused control actions based on the real-time visual state.

[0048] According to the robot control model training method of the present invention, before motion prediction training, key motion segments and non-key motion segments are identified in the demonstration motion sequence, and the supervised motion ground truth of the time step in the non-key motion segment is reconstructed into the demonstration motion of the starting time step in the adjacent key motion segment. During motion prediction training, the predicted motion corresponding to the time step of the non-key motion segment is remapped to the predicted motion corresponding to the starting time step of the adjacent key motion segment. This allows the model to learn to predict and output fine control actions in key motion segments, and in non-key motion segments, instead of outputting fragmented control actions step by step, it directly outputs the control action pointing to the starting time step of the next key motion segment. Thus, in practical robot control applications, the robot can be controlled to skip low-information motions, concentrate the control budget on key interaction stages such as contact, grasping, and alignment, effectively reduce redundant execution steps, improve the robot's task execution efficiency, effectively reduce the risk of error accumulation, and improve the task success rate.

[0049] Figure 2 This is a flowchart illustrating an implementation method for determining key and non-key action segments from a demonstration action sequence according to an embodiment of the present invention. In some embodiments, frequency domain energy analysis is performed on the demonstration action sequence based on the Short-Time Discrete Cosine Transform (DCT) to identify densely interacting key action segments and low-information non-key action segments in the demonstration action sequence, such as... Figure 2 As shown, in step S102 above, determining key and non-key action segments from the demonstration action sequence includes: Step S201: A sliding window of fixed time length is slid across the demonstration action sequence, and a short-time discrete cosine transform is performed on the current sliding window in the demonstration action sequence to obtain the transform coefficients of the current sliding window.

[0050] Define a sliding window with a fixed time length of W, which represents the number of time steps contained in the sliding window. For example, W=8 means that the sliding window contains 8 time steps.

[0051] The demonstration action sequence contains the end action velocity of each time step. The end action velocity corresponding to each time step is differiated from the end action velocity of the previous time step to obtain the end action velocity difference corresponding to each time step. The end action velocity difference corresponding to the first time step is 0. The sliding window short-time discrete cosine transform is performed on the end action velocity difference sequence.

[0052] Specifically, for the current sliding window t, a short-time discrete cosine transform is performed on the end-effector velocity differences corresponding to each time step within the current sliding window t to obtain the transform coefficients C of the current sliding window t. t，k .

[0053] According to the short-time discrete cosine transform, the transform coefficients C t，k It can be represented as:

[0054] Where k represents the DCT frequency of the current sliding window t. For a current sliding window t of length W, the DCT transform will produce W transform coefficients. Therefore, the value of k ranges from 0, 1, 2, ..., W. 1. The larger k is, the higher the corresponding frequency. k=0 represents the DC component, and k=1 to k=W... 1 corresponds to frequency components from low to high, k=W 1 represents the highest frequency that can be represented within window t; n represents the nth time step within the current sliding window t; Let be the normalization constant corresponding to the k-th frequency point of the discrete cosine transform. This represents the end-action velocity difference corresponding to the nth time step within the current sliding window t.

[0055] Each transformation coefficient C of the current sliding window t t，k This indicates the intensity of the local action within the current sliding window t at the k-th DCT frequency.

[0056] Step S202: Determine the spectral energy corresponding to each sliding window based on the transformation coefficients corresponding to each sliding window in the demonstration action sequence.

[0057] For each sliding window, the second-order norm squared process is applied to the change coefficient corresponding to the current sliding window t to obtain the spectral energy corresponding to the current sliding window t.

[0058] The spectral energy corresponding to the current sliding window t can be expressed as:

[0059] This represents the spectral energy of the previous sliding window t at the k-th DCT frequency point.

[0060] Step S203: Calculate the high-frequency energy ratio of each sliding window based on the spectral energy corresponding to each sliding window in the demonstration action sequence. The high-frequency energy ratio is the ratio of the sum of spectral energies at corresponding frequency points within the sliding window that are greater than a preset frequency point value to the sum of spectral energies at each frequency point within the sliding window.

[0061] High-frequency energy percentage r t It can be represented as:

[0062] Where k∈H represents the frequency point corresponding to a high frequency, which is greater than the preset frequency point value. For example, if the sliding window length W=8, then k=0~3 can be divided into the frequency points corresponding to low frequencies, and k=4~7 can be divided into the frequency points corresponding to high frequencies, with the preset frequency point value=3.

[0063] Step S204: For each sliding window in the demonstration action sequence, if the proportion of high-frequency energy in the sliding window is less than a preset proportion, the sliding window is determined as a non-critical action segment.

[0064] Step S205: If the proportion of high-frequency energy in the sliding window is greater than or equal to a preset proportion, the sliding window is divided into key action segments.

[0065] When the high-frequency energy ratio of the sliding window is r t A larger value indicates that the sliding window is experiencing rapid changes, typically corresponding to critical, interaction-intensive actions such as contact, grabbing, and target alignment. Conversely, a smaller value indicates a lower value. t When the value is small, it indicates that the sliding window has low-information smooth motion.

[0066] By using a preset ratio, sliding windows exhibiting high-frequency changes are selected as critical motion segments, while sliding windows exhibiting smooth, low-information motion are selected as non-critical motion segments. The preset ratio can be set according to actual needs; for example, it can be set to 0.75.

[0067] Figure 3 This is a flowchart illustrating another implementation method for determining key and non-key action segments from a demonstration action sequence in an embodiment of the present invention. In some embodiments, geometric trajectory curvature complexity analysis is performed on the demonstration action sequence to identify densely interacting key action segments and low-information non-key action segments, such as... Figure 3 As shown, in step S102 above, determining key and non-key action segments from the demonstration action sequence includes: Step S301: A sliding window of fixed time length is used to slide over the demonstration action sequence.

[0068] The sliding window uses a window with the same length as the sliding window mentioned above.

[0069] Step S302: The straight line connecting the trajectory point corresponding to the first time step and the trajectory point corresponding to the last time step within the current sliding window is used as the reference trajectory.

[0070] In the robot's demonstration action sequence, the demonstration action corresponding to each time step can be regarded as a trajectory point in space or motion timing.

[0071] Step S303: Calculate the vertical distance from the trajectory point corresponding to each time step within the current sliding window to the reference trajectory.

[0072] The trajectory point corresponding to the nth time step within the current sliding window t is denoted as p. t，n trajectory point p t，n The vertical projection point to the reference trajectory is denoted as l. t，n Then the trajectory point p corresponding to the nth time step within the current sliding window t t，n The perpendicular distance to the reference trajectory can be expressed as ||p t，n -l t，n ||.

[0073] Step S304: Normalize the vertical distances of the trajectory points corresponding to each time step within the current sliding window to obtain the normalized distances of the trajectory points corresponding to each time step within the current sliding window.

[0074] The normalization factor S corresponding to the current sliding window t is preset. t The vertical distances of the trajectory points at each time step within the current sliding window are normalized to obtain the normalized distances of the trajectory points at each time step within the current sliding window.

[0075] The normalized distance can be expressed as: .

[0076] In some embodiments, the length of the reference trajectory corresponding to the current sliding window t (the straight-line distance between the trajectory point corresponding to the first time step and the trajectory point corresponding to the last time step) can be used as the normalization factor S corresponding to the current sliding window t. t .

[0077] In some embodiments, the average step length between trajectory points at adjacent time steps within the current sliding window t can be used as the normalization factor S corresponding to the current sliding window t. t .

[0078] Step S305: Calculate the normalized average distance of the trajectory points corresponding to each time step within the current sliding window, and use it as the trajectory curvature of the current sliding window.

[0079] The normalized distance average b of the trajectory points at each time step within the current sliding window t. t It can be represented as:

[0080] Where W represents the window length of the current sliding window t, and n represents the nth time step within the current sliding window t.

[0081] Step S306: Sort the trajectory curvature of each sliding window in descending order, and divide the first preset number of sliding windows into key action segments.

[0082] Step S307: Divide the remaining sliding windows into non-critical action segments.

[0083] When the trajectory curvature of the sliding window is large, it indicates that the sliding window is undergoing rapid changes, which usually corresponds to key interactive actions such as contact, grabbing, and target alignment. Conversely, when the trajectory curvature of the sliding window is small, it indicates that the sliding window is undergoing smooth motion with low information.

[0084] Therefore, the trajectory curvature corresponding to each sliding window is sorted in descending order, and a preset number of sliding windows at the top of the sort are classified as key action segments, while the remaining sliding windows are classified as non-key action segments.

[0085] The ratio of the preset number to the total number of sliding windows is the preset ratio, which can be set according to actual needs. For example, the preset ratio can be set to 0.75.

[0086] In some embodiments, key and non-key action segments obtained based on short-time discrete cosine transform analysis can be merged with key and non-key action segments obtained based on geometric trajectory curvature analysis. Duplicate key and non-key action segments are removed, while different key and non-key action segments are retained, ultimately resulting in the division of key and non-key action segments of the demonstration action sequence.

[0087] In some embodiments, in step S104 above, the visual state sequence in each demonstration trajectory data is input into the prediction model, and the corresponding predicted action sequence is output. This includes: for the visual state sequence of each demonstration trajectory data, the visual state of each time step in the visual state sequence is sequentially input into the prediction model for action prediction; if the current time step is located in a predetermined key action segment, the predicted action currently output by the prediction model is used as the predicted action corresponding to the current time step; if the current time step is located in a predetermined non-key action segment, the predicted action currently output by the prediction model is remapped to the predicted action corresponding to the starting time step of the next key action segment that is temporally adjacent to the non-key action segment; and finally, the predicted action sequence corresponding to the visual state sequence is predicted.

[0088] During model training, define the target starting index:

[0089] Where T represents the T-th time step of the model input, y T For a binary indicator, y T =1 indicates that the T-th time step belongs to the key action segment, y T =0 indicates that the T-th time step belongs to a non-critical action segment, T start (T) indicates from which future time step the model should start predicting the action to be performed when the model is given the visual state at the current time step T.

[0090] When y T When =1, T start (T)=T+1 indicates that the current time step T is within a critical action segment, and the model needs to use the Refine mode to intensively predict the action of the next time step.

[0091] When y T =0 and there is a critical action segment after the current time step T. start (T)=T + (T)=s_i, where s_i represents the starting time step of the next key action segment adjacent to the current time step T. At this time, the model adopts the Skip mode, directly jumping to the starting time step of the next key action segment to predict the action, ignoring the low-information motion in between.

[0092] When y T When the value of the current time step T is 0 and there are no critical action segments after the current time step T (i.e., it is at the end of the trajectory), the model backtracks to execute the action prediction of the next time step T+1 to ensure stability.

[0093] In some embodiments, the visual state corresponding to each time step includes, but is not limited to, the robot's visual image and the robot's body state.

[0094] Among them, the robot's visual images refer to the scene image information collected by the robot's visual acquisition devices (such as monocular cameras, binocular cameras, depth cameras, etc.) at the current time step. These images may include RGB images, depth images, grayscale images, feature maps, etc., and are used to perceive the external environment, the position, posture, shape of target objects, and scene layout.

[0095] The robot's body state refers to the robot's motion and posture-related state information at the current time step, including but not limited to the robot's joint angles, end effector pose, joint velocity, acceleration, gripper opening and closing state, robot base position and posture, etc., which are used to reflect the robot's current motion state.

[0096] In some embodiments, the demonstration action corresponding to each time step includes, but is not limited to, robot end-effector pose, end-effector speed, and gripper state.

[0097] Among them, robot end-effector pose refers to the spatial position and orientation of the robot end effector (such as gripper, robotic arm end gripper, etc.) in the current time step; end-effector motion speed refers to the motion speed of the robot end effector in the current time step, including linear velocity and angular velocity, which is used to control the speed and stability of the end effector's motion; gripper state refers to the working state of the robot gripper (or the gripping component of the end effector) in the current time step, mainly including parameters such as gripping state (clamping, releasing), gripping force, opening and closing angle.

[0098] In some embodiments, depending on the specific task requirements, the demonstration action corresponding to each time step may also include other motion parameters such as robot joint angle, joint speed, acceleration, and actuator driving force.

[0099] This invention also provides a robot control model training device, which is used to implement the above-described robot control model training method embodiments and preferred embodiments, and will not be repeated hereafter. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0100] Figure 4 This is a block diagram illustrating the composition of a robot control model training device according to an embodiment of the present invention. The embodiment of the present invention provides a robot control model training device, such as... Figure 4 As shown, the training device for the robot control model includes: The acquisition module 401 is used to acquire the robot demonstration trajectory dataset. Each demonstration trajectory data in the demonstration trajectory dataset includes a demonstration action sequence and a visual state sequence. The demonstration action sequence includes the robot's demonstration action at each time step, and the visual state sequence includes the robot's visual state at each time step. Action recognition module 402 is used to determine key action segments and non-key action segments from the demonstration action sequence; The supervision target reconstruction module 403 is used to reconstruct the supervision action truth value mapped by the visual state of each time step in each non-critical action segment into the demonstration action corresponding to the starting time step in the next temporally adjacent critical action segment, while keeping the supervision action truth value mapped by the visual state of each time step in each critical action segment unchanged. The prediction module 404 is used to input the visual state sequence in each demonstration trajectory data into the prediction model and output the corresponding prediction action sequence. The predicted action corresponding to the time step of the visual state sequence belonging to the key action segment is the predicted action corresponding to the time step output by the prediction model. The predicted action corresponding to the time step of the non-key action segment is remapped to the predicted action corresponding to the starting time step of the next key action segment that is temporally adjacent. The training module 405 is used to perform imitation learning training on the prediction model based on the supervised action ground value and predicted action sequence mapped by the visual state at each time step after reconstruction in the visual state sequence, so as to obtain the robot control model.

[0101] The robot control model training device provided in this embodiment of the invention can execute the robot control model training method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution. Further functional descriptions of the various modules and units are the same as in the corresponding embodiments described above, and will not be repeated here.

[0102] Figure 5 This is a flowchart illustrating a robot control method based on a robot control model provided in an embodiment of the present invention, as shown below. Figure 5 As shown, the robot control model is trained using the above-described training method, which includes: Step S501: Obtain the robot's current visual state at the current time step.

[0103] The current visual state corresponding to the current time step refers to the observation information acquired by the robot at the current time step, including but not limited to the visual images acquired by the robot's onboard visual acquisition devices (such as monocular cameras, binocular cameras, depth cameras, etc.) at the current time step, as well as the robot's own body state related to its motion and posture at the current time step.

[0104] Step S502: Input the current visual state into the trained robot control model, and predict and output the control action corresponding to the current time step through the robot control model.

[0105] Step S503: Control the robot to execute the control action corresponding to the current time step.

[0106] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The following details the structure of the device. Figure 6 It shows a schematic diagram of a structure suitable for implementing an electronic device in the embodiments of the present invention.

[0107] like Figure 6 As shown, the electronic device may include a processor (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 602 or a program loaded from memory 608 into random access memory (RAM) 603. RAM 603 also stores various programs and data required for the operation of the electronic device. The processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0108] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 608 including, for example, magnetic tapes, hard disks, etc.; and communication devices 609. Communication device 609 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 6 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.

[0109] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 609, or installed from a memory 608, or installed from a ROM 602. When the computer program is executed by the processor 601, it performs the functions defined in the training method or robot control method of the embodiments of the present invention.

[0110] Figure 6The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0111] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the training method or robot control method shown in the above embodiments is implemented.

[0112] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0113] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and all such modifications and variations fall within the scope defined by the appended claims.

Claims

1. A method for training a robot control model, characterized in that, The method includes: Obtain a demonstration trajectory dataset of the robot. Each demonstration trajectory data in the demonstration trajectory dataset includes a demonstration action sequence and a visual state sequence. The demonstration action sequence includes the demonstration action of the robot at each time step, and the visual state sequence includes the visual state of the robot at each time step. Identify key and non-key action segments from the demonstrated action sequence; The supervised action ground truth mapped to the visual state at each time step in each non-critical action segment is reconstructed into the demonstration action corresponding to the starting time step in the next temporally adjacent critical action segment, while keeping the supervised action ground truth mapped to the visual state at each time step in each critical action segment unchanged. The visual state sequence in each demonstration trajectory data is input into the prediction model, and the corresponding predicted action sequence is output. The predicted action corresponding to the time step of the visual state sequence belonging to the key action segment is the predicted action corresponding to the time step output by the prediction model. The predicted action corresponding to the time step of the non-key action segment is remapped to the predicted action corresponding to the starting time step of the next key action segment that is temporally adjacent. The prediction model is trained by imitation learning based on the supervised action ground truth and predicted action sequence mapped by the visual state at each time step after reconstruction in the visual state sequence, to obtain the robot control model.

2. The method according to claim 1, characterized in that, The step of determining key and non-key action segments from the demonstrated action sequence includes: A sliding window of fixed time length is used to slide over the demonstration action sequence. A short-time discrete cosine transform is performed on the current sliding window in the demonstration action sequence to obtain the transform coefficients of the current sliding window. The spectral energy corresponding to each sliding window is determined based on the transformation coefficients corresponding to each sliding window in the demonstration action sequence. The high-frequency energy ratio of each sliding window is calculated based on the spectral energy corresponding to each sliding window in the demonstration action sequence. The high-frequency energy ratio is the ratio of the sum of the spectral energies of the corresponding frequency points in the sliding window that are greater than the preset frequency point value to the sum of the spectral energies of each frequency point in the sliding window. For each sliding window in the demonstration action sequence, if the proportion of high-frequency energy in the sliding window is less than a preset ratio, the sliding window is determined as a non-critical action segment. If the proportion of high-frequency energy in the sliding window is greater than or equal to a preset proportion, the sliding window is divided into key action segments.

3. The method according to claim 1, characterized in that, The step of determining key and non-key action segments from the demonstrated action sequence includes: A sliding window of fixed duration is used to slide over the demonstration action sequence; The straight line connecting the trajectory point corresponding to the first time step and the trajectory point corresponding to the last time step within the current sliding window is used as the reference trajectory. Calculate the vertical distance from the trajectory point corresponding to each time step within the current sliding window to the reference trajectory; Normalize the vertical distances of the trajectory points at each time step within the current sliding window to obtain the normalized distances of the trajectory points at each time step within the current sliding window. Calculate the normalized average distance of the trajectory points corresponding to each time step within the current sliding window, and use it as the trajectory curvature of the current sliding window; Sort the trajectory curvature of each sliding window in descending order, and divide the demonstration action subsequences corresponding to the first preset number of sliding windows into key action segments. The remaining sliding windows are divided into non-critical action segments, corresponding to the demonstration action subsequences.

4. The method according to claim 1, characterized in that, The step of inputting the visual state sequence from each demonstration trajectory data into the prediction model and outputting the corresponding predicted action sequence includes: For each demonstration trajectory data visual state sequence, the visual state at each time step in the visual state sequence is sequentially input into the prediction model for action prediction; If the current time step is located in a predetermined key action segment, the predicted action currently output by the prediction model will be used as the predicted action corresponding to the current time step. If the current time step is located in a predetermined non-critical action segment, the predicted action currently output by the prediction model is remapped to the predicted action corresponding to the starting time step of the next critical action segment that is temporally adjacent to the non-critical action segment. Finally, the predicted action sequence corresponding to the visual state sequence is obtained.

5. The method according to claim 1, characterized in that, The visual state corresponding to each time step includes the robot's visual image and the robot's physical state.

6. The method according to claim 1, characterized in that, The demonstration actions corresponding to each time step include the robot's end-effector pose, end-effector speed, and gripper state.

7. A robot control method based on a robot control model, characterized in that, The robot control model is trained using the training method described in any one of claims 1 to 6, and the robot control method includes: Obtain the robot's current visual state at the current time step; The current visual state is input into the trained robot control model, and the robot control model predicts and outputs the robot's next time step action. Control the robot to execute the action at the next time step.

8. A training device for a robot control model, characterized in that, The device includes: The acquisition module is used to acquire a robot demonstration trajectory dataset. Each demonstration trajectory data in the demonstration trajectory dataset includes a demonstration action sequence and a visual state sequence. The demonstration action sequence includes the robot's demonstration action at each time step, and the visual state sequence includes the robot's visual state at each time step. An action recognition module is used to identify key action segments and non-key action segments from the demonstration action sequence; The supervision target reconstruction module is used to reconstruct the supervision action truth value mapped by the visual state of each time step in each non-critical action segment into the demonstration action corresponding to the starting time step in the next temporally adjacent critical action segment, while keeping the supervision action truth value mapped by the visual state of each time step in each critical action segment unchanged. The prediction module is used to input the visual state sequence in each demonstration trajectory data into the prediction model and output the corresponding predicted action sequence; wherein, the predicted action corresponding to the time step belonging to the key action segment in the visual state sequence is the predicted action corresponding to the time step output by the prediction model, and the predicted action corresponding to the time step belonging to the non-key action segment is remapped to the predicted action corresponding to the starting time step of the next temporally adjacent key action segment. The training module is used to perform imitation learning training on the prediction model based on the supervised action ground value and predicted action sequence mapped by the visual state at each time step after reconstruction in the visual state sequence, so as to obtain the robot control model.

9. An electronic device, characterized in that, include: The system includes a memory and a processor, which are interconnected. The memory stores computer instructions, and the processor executes the computer instructions to perform the training method for the robot control model according to any one of claims 1 to 6, or to perform the robot control method according to claim 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to execute the training method of the robot control model according to any one of claims 1 to 6, or to execute the robot control method according to claim 7.

Images