Action prediction model training method and action prediction method

Abstract

The invention provides a training method of an action prediction model and an action prediction method. According to the main technical scheme, the method comprises the steps of obtaining training data comprising a plurality of training samples; the training sample at least comprises an instruction received by the robot at a first time point, a collected image and an action state, and an action executed at a second time point; identifying whether a preset type of key action exists between the first time point and the second time point; training an action prediction model based on the training data; the training comprises the steps of inputting an instruction received at a first time point, a collected image and an action state into an action prediction model, if a key action of a preset type is identified, taking the key action of the preset type as a prediction target of the action prediction model, otherwise, taking the key action of the preset type as a prediction target of the action prediction model; and taking the action executed at the second time point as a prediction target of the action prediction model. The execution speed and accuracy of the robot action can be improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a training method for an action prediction model and an action prediction method. Background Art

[0002] The robot world model is a key tool for the robot to understand and predict the surrounding environment, and plays a central role in guiding the robot's decision-making and planning when performing tasks.

[0003] Currently, autoregressive models are widely used to directly output the robot's motion plan. However, this approach has an inherent contradiction between execution speed and execution accuracy. Specifically, the efficiency of the autoregressive model is limited due to the serial output of action tokens, resulting in the mainstream design scheme being able to output motion plans at a single moment. This results in the planned action being close to the current time point, although it can maintain a high accuracy, but the execution speed is slow; and when the planned action is far away from the current time point, the execution speed is fast, but the accuracy is reduced. This problem is particularly prominent in actions that require precise coordination between the robot's mechanical arm and gripper. Summary of the invention

[0004] In view of this, the present application provides a training method for an action prediction model and an action prediction method to optimize the execution speed and execution accuracy of robot actions.

[0005] This application provides the following solutions:

[0006] In a first aspect, a method for training an action prediction model is provided, comprising:

[0007] Acquire training data including a plurality of training samples; the training samples at least include: instructions received by the robot at a first time point, images collected and action states, and actions performed at a second time point, the second time point being a future time point after a preset time interval from the first time point;

[0008] Identify whether there is a key action of a preset type between the first time point and the second time point;

[0009] Based on the training data, an action prediction model is trained; the training includes: inputting the instructions, collected images and action states received at the first time point into the action prediction model; if the recognition obtains a key action of a preset type, the key action of the preset type is used as the prediction target of the action prediction model; otherwise, the action performed at the second time point is used as the prediction target of the action prediction model.

[0010] Optionally, the action state includes: the action state of the robot's mechanical arm and the opening and closing state of the gripper;

[0011] The key actions of the preset type include: a clamp opening action or a clamp closing action.

[0012] Optionally, identifying whether there is a key action of a preset type between the first time point and the second time point includes:

[0013] By using the degree of opening and closing of the robot's gripper jaws between the first time point and the second time point, it is identified whether there is a gripper jaw opening action or a gripper jaw closing action between the first time point and the second time point.

[0014] Optionally, the using the degree of opening and closing of the gripper of the robot between the first time point and the second time point to identify whether there is a gripper opening action or a gripper closing action between the first time point and the second time point includes:

[0015] Preprocessing the training data to extract the gripper action value at the first time point and the gripper action value at the second time point;

[0016] When the gripper action value at the first time point is less than a closing threshold, and the gripper action value at the second time point is greater than the closing threshold, it is determined that there is a gripper closing action between the first time point and the second time point;

[0017] When the gripper action value at the first time point is greater than the opening threshold, and the gripper action value at the second time point is less than the opening threshold, it is determined that there is a gripper opening action between the first time point and the second time point;

[0018] Otherwise, it is determined that there is no jaw opening action or jaw closing action between the first time point and the second time point.

[0019] Optionally, the action prediction model is implemented based on a multimodal autoregressive model.

[0020] In a second aspect, an action prediction method is provided, comprising:

[0021] Acquiring input data, wherein the input data includes instructions received by the robot at a first time point, collected images, and action states;

[0022] Inputting the input data into an action prediction model to obtain an action predicted by the action prediction model according to the input data;

[0023] The action prediction model is pre-trained using a method as described in any one of the first aspects above.

[0024] Optionally, the method further comprises:

[0025] According to the predicted action, the robot is controlled to perform a corresponding action.

[0026] In a third aspect, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the steps of the method described in any one of the first aspect or the second aspect are implemented.

[0027] In a fourth aspect, an electronic device is provided, including:

[0028] one or more processors; and

[0029] A memory associated with the one or more processors, the memory being used to store program instructions, wherein when the program instructions are read and executed by the one or more processors, the steps of the method described in any one of the first aspect or the second aspect are executed.

[0030] In a fifth aspect, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the steps of the method described in any one of the first aspect or the second aspect.

[0031] The embodiment of the present application realizes the intelligent switching of prediction targets during the training process of the action prediction model by identifying whether there is a key action between the first time point and the second time point. When there is a key action, the action prediction model uses the key action as the prediction target, and when there is no key action, the action at the second time point is used as the prediction target. This method optimizes the training process of the action prediction model, effectively improves the execution speed and accuracy of the robot's actions, and enhances the coordination and efficiency of the robot in complex tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 A flowchart of a method for training an action prediction model provided in an embodiment of the present application;

[0034] Figure 2 A schematic diagram of the opening and closing time curve of the clamping jaws provided in an embodiment of the present application;

[0035] Figure 3 It is a schematic diagram of the multimodal autoregressive model;

[0036] Figure 4A flowchart of the action prediction method provided in an embodiment of the present application;

[0037] Figure 5 A schematic block diagram of a training device for an action prediction model provided in an embodiment of the present application;

[0038] Figure 6 A schematic block diagram of an action prediction device provided in an embodiment of the present application

[0039] Figure 7 A schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0041] The terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. The singular forms "a", "said" and "the" used in the embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0042] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0043] There are already some technologies that use autoregressive models to predict the robot's motion planning. When the model predicts a short-term motion close to the current time point, although it can achieve accurate coordination between the robot arm and the gripper, the execution speed is limited. On the contrary, when the model predicts a long-term motion far away from the current time point, although it can improve the execution speed, it often leads to uncoordinated motion of the robot arm and the gripper, affecting task execution, such as failure to grasp an object.

[0044] In view of this, the present application provides a new idea and a new action prediction model. Figure 1 The flowchart of the training method of the action prediction model provided in the embodiment of the present application, the execution subject of the method can be a model training device, which can be set in any computer device with data storage and processing capabilities, can be set on the server side, and can also be set in a computer terminal with strong data processing capabilities. The method may include the following steps:

[0045] Step 101: Acquire training data including a plurality of training samples, where the training samples at least include instructions received by the robot at a first time point, images collected and action states, and actions performed at a second time point, where the second time point is a future time point after a preset time interval from the first time point;

[0046] Step 102: Identify whether there is a key action of a preset type between the first time point and the second time point;

[0047] Step 103: Based on the training data, train the action prediction model; wherein the training includes: inputting the instructions received at the first time point, the collected images and the action state into the action prediction model; if a key action of a preset type is identified, the key action of the preset type is used as the prediction target of the action prediction model; otherwise, the action performed at the second time point is used as the prediction target of the action prediction model.

[0048] It can be seen that this application realizes the intelligent switching of prediction targets during the training of the action prediction model by identifying whether there is a key action between the first time point and the second time point. When a key action is identified, the action prediction model uses the key action as the prediction target. When the key action is not identified, the action at the second time point is used as the prediction target. This method optimizes the training process of the action prediction model, effectively improves the execution speed and accuracy of the robot's actions, and enhances the coordination and efficiency of the robot in complex tasks.

[0049] The following is a detailed description of each step in the above process and the effects that can be further produced in conjunction with the embodiments. It should be noted that the limitations such as "first" and "second" involved in the present disclosure do not have limitations in terms of size, order, and quantity, and are only used to distinguish them in name. For example, "first time point" and "second time point" are only used to distinguish two time points in name.

[0050] First, the above step 101, namely “obtaining training data including a plurality of training samples” is described in detail in conjunction with the embodiment.

[0051] In an embodiment of the present application, each training sample includes at least: instructions received by the robot at a first time point, captured images and action states, and actions performed at a second time point, wherein the second time point is a future time point after a preset time interval from the first time point.

[0052] The instruction received by the robot at the first time point refers to the operation instruction used to instruct the robot to perform the grasping task. The image collected by the robot at the first time point refers to the environmental image collected by the robot through the sensor or camera carried by the robot at the first time point, which provides the robot with the visual information required to perform the grasping task, so that it can identify the target object and the position of the target object in the grasping task. The action state of the robot at the first time point includes but is not limited to the action state of the robot arm and the opening and closing state of the gripper. The action state of the robot arm may include parameters such as the angle, speed, acceleration, etc. of each joint, and the opening and closing state of the gripper may include the degree of opening and closing of the gripper.

[0053] The action performed at the second time point refers to the action actually performed by the robot after a preset time interval based on the first time point. The second time point refers to a future time point after a preset time interval from the first time point, and the preset time interval is predefined and can be adjusted according to actual needs.

[0054] The instructions received by the robot at the first time point, the collected images and action states, and the actions performed at the second time point are combined to form a training sample. A plurality of training samples constitute training data.

[0055] The above step 102, namely "identifying whether there is a key action of a preset type between the first time point and the second time point", is described in detail below in conjunction with an embodiment.

[0056] Since the most common task in the robot's execution process is grasping, in the embodiment of the present application, the key action of the preset type may include a gripper opening action or a gripper closing action. If it is another type of task, another type of key action may be used.

[0057] Taking the robot's task of grasping a target object as an example, the gripper opening action refers to the process of the robot's gripper changing from a closed state to an open state. This action is a key step in releasing or placing the target object in the grasping task. This process allows the robot to release and put down the target object, ensuring that the target object is correctly placed or released at the specified location; and the gripper closing action refers to the process of the robot's gripper changing from an open state to a closed state. This action is a key step in achieving the grasping of the target object. This process allows the robot's gripper to gradually close to surround and firmly grasp the target object.

[0058] In order to ensure the coordination of the robot's mechanical arm and gripper movements, the present application embodiment proposes a method for identifying whether a gripper opening movement or a gripper closing movement will occur between a first time point and a second time point. Specifically, the degree of gripper opening and closing of the robot between the first time point and the second time point can be used for identification.

[0059] As one of the possible ways, the identification process is as follows:

[0060] First, the training data is preprocessed to extract the jaw action value at the first time point and the jaw action value at the second time point; the jaw action value here is used to characterize the degree of opening and closing of the jaw.

[0061] In an embodiment of the present application, the opening and closing states of the jaws at the first time point and the opening and closing states of the jaws at the second time point contained in the training data are normalized to obtain the jaw action value at the first time point and the jaw action value at the second time point, wherein the action value obtained after normalization ranges from 0 to 1, 0 represents that the jaws are fully closed, and 1 represents that the jaws are fully open.

[0062] Secondly, according to the gripper action value at the first time point and the gripper action value at the second time point, as well as the preset opening threshold and closing threshold, it is determined whether there is a gripper opening action or a gripper closing action between the first time point and the second time point.

[0063] As a feasible method, when the jaw action value at the first time point is less than the closing threshold, and the jaw action value at the second time point is greater than the closing threshold, it is determined that there is a jaw closing action between the first time point and the second time point; when the jaw action value at the first time point is greater than the opening threshold, and the jaw action value at the second time point is less than the opening threshold, it is determined that there is a jaw opening action between the first time point and the second time point; otherwise, it is determined that there is no jaw opening action or jaw closing action between the first time point and the second time point.

[0064] like Figure 2 As shown, it is a schematic diagram of the opening and closing time curve of the clamp provided in the embodiment of the present application. The curve shows the relationship between the opening and closing degree of the clamp and time, wherein the horizontal axis represents time and the vertical axis represents the opening and closing degree of the clamp. In the opening and closing time curve, the closing point and the opening point of the clamp are set, wherein the closing threshold corresponding to the closing point is 0.2, and the opening threshold corresponding to the opening point is 0.8. After determining the clamp action value of the first time point and the clamp action value of the second time point, the closing threshold and the opening threshold on the two clamp action values ​​and the clamp opening and closing time curve are compared. If the clamp action value of the first time point is less than 0.2, and the clamp action value of the second time point is greater than 0.2, it is determined that there is a clamp closing action between the first time point and the second time point; if the clamp action value of the first time point is greater than 0.8, and the clamp action value of the second time point is less than 0.8, it is determined that there is a clamp opening action between the first time point and the second time point; otherwise, it is determined that there is no clamp opening action or clamp closing action between the first time point and the second time point.

[0065] It should be noted that the opening threshold and closing threshold here can be set according to actual needs, and the embodiments of the present application are not limited thereto.

[0066] The above step 103, namely “training the action prediction model based on training data”, is described in detail below in conjunction with an embodiment.

[0067] The training process may include: inputting the instructions, captured images and action states received at the first time point into the action prediction model; if a preset type of key action is identified, the preset type of key action is used as the prediction target of the action prediction model; otherwise, the action performed at the second time point is used as the prediction target of the action prediction model.

[0068] The action prediction model in the embodiment of the present application can be implemented based on a multimodal autoregressive model.

[0069] like Figure 3 As shown in the figure, it is a schematic diagram of a multimodal autoregressive model. The input data of the model are the instructions received at time t, the collected images and the robot action state. The output data of the model, time t1 represents the predicted action close to the current time t, which can generally be set between 0.1s and 0.5s in practical applications. Time t2 has a key action, such as the gripper opening action or the gripper closing action. Time t3 represents the predicted action far from the current time t, which can generally be set between 1s and 2s in practical applications.

[0070] In the prior art, the data at time t1 is generally input and the action at time t1 is output. In this way, the robot's action execution amplitude is small, and the action coordination of the robot arm and the gripper is better, but the action execution is slow.

[0071] In the embodiment of the present application, the initial prediction target of the model is to predict the action at time t3. If it is identified that there is a jaw opening action or a jaw closing action between time t and time t3, that is, there is a key action at time t2, the prediction target is modified to predict the action at time t2.

[0072] In the embodiment of the present application, by identifying whether there is a key action between the first time point and the second time point, the intelligent switching of the prediction target during the training of the action prediction model is realized. When the key action is identified, the action prediction model uses the key action as the prediction target. When the key action is not identified, the action at the second time point is used as the prediction target. This method optimizes the training process of the action prediction model, effectively improves the execution speed and accuracy of the robot's actions, and enhances the coordination and efficiency of the robot in complex tasks.

[0073] The autoregressive model is a model for time series analysis and prediction, which can predict future values ​​based on historical values. In the training process, the relationship between the observed values ​​at the current moment (first time point) and the future moment (second time point) is mainly modeled. In an embodiment of the present application, if a key action is identified at the first time point and the second time point, the key action is used as the prediction target, that is, the key action is used as the true value of the prediction result of the second time point used when training the action prediction model (i.e., the above-mentioned observation value); if the key action is not identified at the first time point and the second time point, the action at the second time point is used as the prediction target, that is, the action at the second time point is used as the true value of the prediction result of the second time point used by the action prediction model (i.e., the above-mentioned observation value). The training objectives include: minimizing the difference between the prediction result of the action prediction model and the corresponding true value.

[0074] In the embodiments of the present specification, a loss function can be constructed according to the above training objectives, and the value of the loss function can be used in each round of iteration to update the model parameters by methods such as gradient descent until the preset training end condition is met. The training end condition may include, for example, the value of the loss function is less than or equal to the preset loss function threshold, the number of iterations reaches the preset number threshold, etc.

[0075] The loss function used may be, for example, mean square error, mean absolute error, etc.

[0076] Based on the above-mentioned action prediction model obtained through training, the present application embodiment also provides an action prediction method, such as Figure 4 As shown, the following steps are included:

[0077] Step 401: obtaining input data, where the input data includes instructions received by the robot at a first time point, collected images, and motion states;

[0078] Step 402: inputting input data into the action prediction model, and obtaining the action predicted by the action prediction model according to the input data;

[0079] Among them, the action prediction model is pre-trained using the above-mentioned model training method.

[0080] For the specific principles of the action prediction model, please refer to the relevant records in the previous embodiments, which will not be repeated here.

[0081] In step 401, the instructions, images and motion states received by the robot at the first time point are first obtained as input data, where the instructions may include text commands, voice commands or other forms of control information. At the same time, the robot's camera or other visual sensors collect image data of the current environment, and the robot's motion state at the first time point is also recorded, including the joint angle, speed, acceleration, etc. of the robot arm, as well as the opening and closing state of the gripper.

[0082] In step 402, the acquired instructions, images and action states are first preprocessed, including data cleaning, feature extraction and normalization, and then input into the action prediction model. The action prediction model predicts the action of the robot at the next time point based on the input multimodal data.

[0083] Optionally, the embodiment of the present application may further include:

[0084] Step 403: According to the predicted action, control the robot to perform the corresponding action.

[0085] Through the above method, the robot's actions can be effectively predicted and controlled, improving the robot's intelligence level and operational efficiency.

[0086] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0087] According to an embodiment of another aspect, a training device for an action prediction model is provided. Figure 5 A schematic block diagram of a training device for the action prediction model according to an embodiment is shown, and the device is arranged in any computer device with data storage and processing capabilities. Figure 5 As shown, the device 500 includes: a sample acquisition unit 501, a key action recognition unit 502 and a model training unit 503. The main functions of each component unit are as follows:

[0088] The sample acquisition unit 501 is configured to acquire training data including a plurality of training samples; the training samples at least include: instructions received by the robot at a first time point, images collected and action states, and actions performed at a second time point, where the second time point is a future time point after a preset time interval from the first time point;

[0089] A key action identification unit 502 is configured to identify whether there is a key action of a preset type between the first time point and the second time point;

[0090] The model training unit 503 is configured to train the action prediction model based on the training data; the training includes: inputting the instructions, collected images and action states received at the first time point into the action prediction model, and if the recognition obtains a key action of a preset type, then the key action of the preset type is used as the prediction target of the action prediction model; otherwise, the action performed at the second time point is used as the prediction target of the action prediction model.

[0091] Optionally, the action state includes: the action state of the robot's mechanical arm and the opening and closing state of the gripper;

[0092] The key actions of the preset type include: a clamp opening action or a clamp closing action.

[0093] Optionally, the key action recognition unit 502 is configured to:

[0094] By using the degree of opening and closing of the robot's gripper jaws between the first time point and the second time point, it is identified whether there is a gripper jaw opening action or a gripper jaw closing action between the first time point and the second time point.

[0095] Optionally, the key action recognition unit 502 is specifically configured as follows:

[0096] Preprocessing the training data to extract the gripper action value at the first time point and the gripper action value at the second time point;

[0097] When the gripper action value at the first time point is less than a closing threshold, and the gripper action value at the second time point is greater than the closing threshold, it is determined that there is a gripper closing action between the first time point and the second time point;

[0098] When the gripper action value at the first time point is greater than the opening threshold, and the gripper action value at the second time point is less than the opening threshold, it is determined that there is a gripper opening action between the first time point and the second time point;

[0099] Otherwise, it is determined that there is no jaw opening action or jaw closing action between the first time point and the second time point.

[0100] Optionally, the action prediction model is implemented based on a multimodal autoregressive model.

[0101] According to an embodiment of another aspect, an action prediction device is provided. Figure 6A schematic block diagram of the action prediction device according to one embodiment is shown, and the device is arranged in any computer device with data storage and processing capabilities. Figure 6 As shown, the device 600 includes: a data acquisition unit 601, an action prediction unit 602 and an action control unit 603. The main functions of each component unit are as follows:

[0102] The data acquisition unit 601 is configured to acquire input data, wherein the input data includes instructions, collected images and action states received by the robot at a first time point;

[0103] The action prediction unit 602 is configured to input the input data into the action prediction model, and obtain the action predicted by the action prediction model according to the input data;

[0104] The action prediction model is pre-trained using the action prediction model training method described above.

[0105] The action control unit 603 is configured to control the robot to perform corresponding actions according to the predicted actions.

[0106] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system or device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The system or device embodiment described above is merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0107] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0108] In addition, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods in the aforementioned method embodiments are implemented.

[0109] And an electronic device, comprising:

[0110] one or more processors; and

[0111] A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of the aforementioned method embodiments.

[0112] The present application also provides a computer program product, including a computer program, which implements the steps of any one of the methods in the aforementioned method embodiments when executed by a processor.

[0113] in, Figure 7 The architecture of the electronic device is shown as an example, which may include a processor 710, a video display adapter 711, a disk drive 712, an input / output interface 713, a network interface 714, and a memory 720. The processor 710, the video display adapter 711, the disk drive 712, the input / output interface 713, the network interface 714, and the memory 720 may be communicatively connected via a communication bus 730.

[0114] Among them, the processor 710 can be implemented by a general-purpose CPU, a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, etc., to execute relevant programs to implement the technical solutions provided in this application.

[0115] The memory 720 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 720 can store an operating system 721 for controlling the operation of the electronic device 700, and a basic input and output system (BIOS) 722 for controlling the low-level operation of the electronic device 700. In addition, a web browser 723, a data storage management system 724, and a training device 400 or a motion prediction device 500 for an action prediction model, etc. can also be stored. The training device 400 or the motion prediction device 500 for the above-mentioned action prediction model can be an application program that specifically implements the operations of the aforementioned steps in the embodiment of the present application. In short, when the technical solution provided in the present application is implemented by software or firmware, the relevant program code is stored in the memory 720 and is called and executed by the processor 710.

[0116] The input / output interface 713 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0117] The network interface 714 is used to connect to a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through a wired mode (such as USB, network cable, etc.) or a wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0118] The bus 730 comprises a pathway for transmitting information between the various components of the device (eg, the processor 710, the video display adapter 711, the disk drive 712, the input / output interface 713, the network interface 714, and the memory 720).

[0119] It should be noted that, although the above device only shows a processor 710, a video display adapter 711, a disk drive 712, an input / output interface 713, a network interface 714, a memory 720, a bus 730, etc., in the specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the solution of the present application, and does not necessarily include all the components shown in the figure.

[0120] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present application can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly contributed to the prior art in the form of a computer program product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0121] The technical solution provided by the present application is described in detail above. The principle and implementation method of the present application are described in detail using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A method for training an action prediction model, characterized in that: include: Acquire training data including a plurality of training samples; The training samples include at least: instructions received by the robot at a first time point, images collected and action states, and actions performed at a second time point, where the second time point is a future time point after a preset time interval from the first time point; Identify whether there is a key action of a preset type between the first time point and the second time point; Based on the training data, an action prediction model is trained; the training includes: inputting the instructions, collected images and action states received at the first time point into the action prediction model; if the recognition obtains a key action of a preset type, the key action of the preset type is used as the prediction target of the action prediction model; otherwise, the action performed at the second time point is used as the prediction target of the action prediction model.

2. The method according to claim 1, characterized in that The action state includes: the action state of the robot's mechanical arm and the opening and closing state of the gripper; The key actions of the preset type include: a clamp opening action or a clamp closing action.

3. The method according to claim 2, characterized in that The key actions of identifying whether there is a preset type between the first time point and the second time point include: By using the degree of opening and closing of the robot's gripper jaws between the first time point and the second time point, it is identified whether there is a gripper jaw opening action or a gripper jaw closing action between the first time point and the second time point.

4. The method according to claim 3, characterized in that The method of using the degree of opening and closing of the robot's gripper between the first time point and the second time point to identify whether there is a gripper opening action or a gripper closing action between the first time point and the second time point includes: Preprocessing the training data to extract the gripper action value at the first time point and the gripper action value at the second time point; When the gripper action value at the first time point is less than the closing threshold, and the gripper action value at the second time point is greater than the closing threshold, determining that there is a gripper closing action between the first time point and the second time point; When the gripper action value at the first time point is greater than the opening threshold, and the gripper action value at the second time point is less than the opening threshold, it is determined that there is a gripper opening action between the first time point and the second time point; Otherwise, it is determined that there is no jaw opening action or jaw closing action between the first time point and the second time point.

5. The method according to claim 1, characterized in that The action prediction model is implemented based on a multimodal autoregressive model.

6. A method for motion prediction, characterized in that: include: Acquiring input data, wherein the input data includes instructions received by the robot at a first time point, collected images, and action states; Inputting the input data into an action prediction model to obtain an action predicted by the action prediction model according to the input data; The action prediction model is pre-trained using the method according to any one of claims 1 to 5.

7. The method according to claim 6, characterized in that The method further comprises: According to the predicted action, the robot is controlled to perform a corresponding action.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method described in any one of claims 1 to 7 are implemented.

9. An electronic device, characterized in that: include: one or more processors; as well as A memory associated with the one or more processors, the memory being used to store program instructions, wherein the program instructions, when read and executed by the one or more processors, execute the steps of the method described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images