Method and device for determining trajectory data, robot controller and storage medium
By obtaining action data during the robot's execution of target actions and using a trajectory prediction model for processing, the long-term problem in the prior art is solved, and efficient and accurate object trajectory prediction is achieved.
Patent Information
- Application Number
- CN202410165427.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-02-02
- Publication Date
- 2025-08-05
AI Technical Summary
In the prior art, robots predict the object trajectory by using a motion capture sensor to complete the object for a period of time, which takes a long time.
By obtaining action data during the robot's process of performing target actions on an object and processing it using a trajectory prediction model, the trajectory data of the object is directly predicted, including the architecture of the encoder and decoder, and the LSTM model is used to encode and decode action features.
It realizes that the object's trajectory can be predicted immediately after the robot completes the target action on the object, saving time, improving prediction efficiency, and ensuring the accuracy of the trajectory data.
Smart Images

Figure CN120422212A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of computer technology, and in particular, to a method, an apparatus, a robot controller, and a storage medium for determining trajectory data. Background Art
[0002] With the development of computer technology and the wide application of robots, the accurate prediction of robot motion has become a research hotspot. Among them, how to predict the trajectory of the object manipulated by the robot is particularly challenging.
[0003] In the related art, a method for predicting the trajectory of an object in the scenario where a robot throws an object is proposed. After the robot completes the throwing action on the object, the object is photographed by a motion capture sensor for a period of time, and based on the trajectory of the object that has been photographed and the physical characteristics of the object, the trajectory of the object after the photographed trajectory is predicted.
[0004] However, the above solution requires photographing the object by a motion capture sensor for a period of time before predicting the subsequent trajectory, which takes a long time. Summary of the Invention
[0005] The embodiments of the present application provide a method, an apparatus, a robot controller, and a storage medium for determining trajectory data, which save time consumption and improve efficiency. The technical solutions are as follows:
[0006] On the one hand, a method for determining trajectory data is provided. The method includes:
[0007] During the process of the robot performing a target action on an object, obtaining action data of the robot, where the action data represents the state of the robot itself during the process of the robot performing the target action on the object;
[0008] Processing the action data through a trajectory prediction model to obtain trajectory data of the object;
[0009] Where the trajectory data represents the trajectory of the object after the robot completes the target action on the object.
[0010] On the other hand, a device for determining trajectory data is provided. The device includes:
[0011] An action data acquisition module, configured to obtain action data of the robot during the process of the robot performing a target action on an object, where the action data represents the state of the robot itself during the process of the robot performing the target action on the object;
[0012] A prediction module, configured to process the action data through a trajectory prediction model to obtain trajectory data of the object;
[0013] Among them, the trajectory data represents the trajectory of the object after the robot completes the target action on the object.
[0014] In a possible implementation, the trajectory prediction model includes an encoder and a decoder, and the prediction module includes:
[0015] An encoding unit, configured to encode the action data through the encoder to obtain action features;
[0016] A decoding unit, configured to decode the action features through the decoder to obtain the trajectory data of the object.
[0017] In a possible implementation, the action data includes at least two types, the encoder includes at least two encoding blocks and a fusion layer, and the encoding unit is configured to:
[0018] Encode the action data of at least two types corresponding to at least two of the encoding blocks through the at least two encoding blocks to obtain at least two action features;
[0019] Fuse at least two of the action features through the fusion layer to obtain the fused action features.
[0020] In a possible implementation, the encoder further includes an association encoding block, and the encoding unit is configured to encode the action data of at least two types through the association encoding block to obtain association action features, where the association action features represent the association between at least two of the action features; fuse at least two of the action features and the association action features through the fusion layer to obtain the fused action features.
[0021] In a possible implementation, the association encoding block includes at least two encoding layers, and the encoding unit is configured to encode the action data of at least two types through the first encoding layer in the association encoding block to obtain first association action features; encode the first association action features through the second encoding layer in the association encoding block to obtain second association action features, until the association action features are obtained through the last encoding layer in the association encoding block.
[0022] In a possible implementation, the trajectory prediction model is a long short-term memory network (LSTM) model, and the encoding unit is configured to, when obtaining the action data at a first time point each time, encode the action data through the encoder to obtain the action features at the first time point, where the first time point is a time point during the process of the robot performing the target action on the object;
[0023] The decoding unit is configured to combine the action features at at least two of the first time points obtained when the robot completes the target action on the object; and decode the combined action features through the decoder to obtain the trajectory data.
[0024] In a possible implementation, the action data of the robot includes the action data of the robot at at least two first time points, and the trajectory data of the object includes the pose parameters of the object at at least two second time points, where the pose parameters include position and attitude; wherein, the first time point is a time point during the process of the robot performing the target action on the object, and the second time point is a time point after the robot completes the target action on the object;
[0025] The apparatus further includes:
[0026] A training module, configured to obtain sample data from a sample data set and train the trajectory prediction model based on the sample data;
[0027] The sample data includes sample action data and a sample trajectory, the sample action data includes the action data of the robot at at least two first sample time points, and the sample trajectory includes the pose parameters of a sample object at at least two second sample time points; wherein, the first sample time point is a time point during the process of the robot performing the target action on the sample object, and the second sample time point is a time point after the robot completes the target action on the sample object.
[0028] In a possible implementation, the sample data set includes at least one of the following:
[0029] A first sample data set, the sample data in the first sample data set is collected under a first condition, and the first condition is that the robot uses the same end effector to perform the target action on the same sample object, and the pose and speed of the end effector when releasing the sample object after completing the target action are not completely the same;
[0030] A second sample data set, the sample data in the second sample data set is collected under a second condition, and the second condition is that the robot uses a random end effector to perform the target action on the same sample object, and the pose and speed of the robot when releasing the sample object after completing the target action are random;
[0031] The third sample data set, where the sample data in the third sample data set is collected under a third condition, and the third condition is: the robot uses a random end effector to perform the target action on different sample objects respectively, and the pose and speed of the robot when releasing the current sample object after completing the target action are random.
[0032] In a possible implementation, the robot includes an end effector, and the action data acquisition module includes at least one of the following:
[0033] A first acquisition unit, configured to acquire the pose parameters of the end effector at at least two first time points, where the pose parameters include position and orientation;
[0034] A second acquisition unit, configured to acquire the speed of the end effector at at least two first time points;
[0035] A force / torque sensor is configured on the joint corresponding to the end effector, and a third acquisition unit is configured to acquire the force and torque of the end effector at at least two first time points through the force / torque sensor;
[0036] Wherein, the first time point is a time point during the process that the robot performs the target action on the object.
[0037] In a possible implementation, the robot includes an end effector; the device further includes:
[0038] A coordinate system creation module, configured to create a target attached coordinate system, where the target attached coordinate system takes the centroid of the object grasped by the end effector as the origin, and the directions of each coordinate axis in the target attached coordinate system are determined based on the directions of each coordinate axis of the end effector, and the action data represents the state of the robot in the target attached coordinate system during the process that the robot performs the target action on the object.
[0039] On the other hand, a robot controller is provided, and the robot controller includes a processor and a memory, and at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the operations performed by the method for determining trajectory data as described in the above aspect.
[0040] On the other hand, a computer-readable storage medium is provided, and at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor to implement the operations performed by the method for determining trajectory data as described in the above aspect.
[0041] On the other hand, a computer program product is provided, including a computer program which is loaded and executed by a processor to implement the operations performed by the method for determining trajectory data as described in the above aspects.
[0042] The solution provided by the embodiments of the present application can obtain the motion data of the robot during the process of the robot performing a target action on an object. After being processed by a trajectory prediction model, the trajectory data of the object can be predicted. The trajectory of the object can be predicted immediately after the robot finishes performing the target action on the object, without waiting to take pictures of the object for a period of time by a motion capture sensor after the robot finishes performing the target action on the object, saving time consumption and improving efficiency. Moreover, the embodiments of the present application summarize and learn from human daily actions, and determine that the state of the robot during the process of the robot performing a target action on an object determines the trajectory of the object. Therefore, a trajectory prediction model with the motion data of the robot during the process of the robot performing a target action on the object as the input and the trajectory data of the object as the output is used for prediction, ensuring the accuracy of the predicted trajectory data. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.
[0044] Figure 1 is a schematic diagram of an implementation environment provided by the embodiments of the present application;
[0045] Figure 2 is a flowchart of a method for determining trajectory data provided by the embodiments of the present application;
[0046] Figure 3 is a schematic diagram of an object trajectory provided by the embodiments of the present application;
[0047] Figure 4 is a flowchart of another method for determining trajectory data provided by the embodiments of the present application;
[0048] Figure 5 is a schematic structural diagram of a trajectory prediction model provided by the embodiments of the present application;
[0049] Figure 6 is a flowchart of another method for determining trajectory data provided by the embodiments of the present application;
[0050] Figure 7 is a schematic diagram of different positions where a robot grasps an object provided by the embodiments of the present application;
[0051] Figure 8 It is a schematic diagram of the change curve of the force / torque of a robot provided by an embodiment of the present application;
[0052] Figure 9 It is a schematic diagram of the change curve of an action data provided by an embodiment of the present application;
[0053] Figure 10 It is a schematic diagram of different positions where a robot grasps an object provided by an embodiment of the present application;
[0054] Figure 11 It is a schematic diagram of the displacement of an object during a robot executing a target action provided by an embodiment of the present application;
[0055] Figure 12 It is a schematic diagram of the attached body coordinate system of an object provided by an embodiment of the present application;
[0056] Figure 13 It is a schematic diagram of another attached body coordinate system of an object provided by an embodiment of the present application;
[0057] Figure 14 It is a schematic diagram of another attached body coordinate system of an object provided by an embodiment of the present application;
[0058] Figure 15 It is a schematic diagram of a coordinate system provided by an embodiment of the present application;
[0059] Figure 16 It is a schematic diagram of the structure of an associated coding block provided by an embodiment of the present application;
[0060] Figure 17 It is a schematic diagram of action data and trajectory data provided by an embodiment of the present application;
[0061] Figure 18 It is a schematic diagram of the structure of a device for determining trajectory data provided by an embodiment of the present application;
[0062] Figure 19 It is a schematic diagram of the structure of another device for determining trajectory data provided by an embodiment of the present application;
[0063] Figure 20 It is a schematic diagram of the structure of a robot controller provided by an embodiment of the present application. Detailed implementation manners
[0064] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the following will further describe the embodiments of the present application in detail with reference to the accompanying drawings.
[0065] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of this application, the first time point may be referred to as the second time point, and similarly, the second time point may be referred to as the first time point.
[0066] Among them, at least two means two or more. For example, at least two time points can be two time points, three time points, or any integer greater than or equal to two, such as three time points. Each refers to each of the at least two. For example, each time point refers to each of the at least two time points. If the at least two time points are three time points, then each time point refers to each of the three time points.
[0067] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, input data, etc.), and signals (including but not limited to signals transmitted between user terminals and other devices, etc.) involved in this application are all fully authorized by the user or relevant parties, and the collection, use, and processing of relevant data need to comply with the relevant laws, regulations, and standards of relevant countries and regions.
[0068] Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a way similar to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines to enable the machines to have the functions of perception, reasoning, and decision-making.
[0069] Artificial intelligence technology is an interdisciplinary subject with a wide range of fields, including both hardware-level technologies and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, the pre-trained model, also known as the large model or the foundation model, can be widely applied to downstream tasks in various directions of artificial intelligence after fine-tuning. The software technologies of artificial intelligence mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, intelligent transportation, and other major directions.
[0070] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in multiple fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, driverless, autonomous driving, drones, digital twins, virtual humans, robots, artificial intelligence-generated content, conversational interaction, smart healthcare, smart customer service, game AI, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.
[0071] A pre-trained model (PTM), also known as a foundation model or a large model, refers to a deep neural network (DNN) with a large number of parameters. It is trained on a large amount of unlabeled data, and the function approximation ability of the large-parameter DNN is used to enable the PTM to extract common features from the data. Through techniques such as fine-tuning, parameter-efficient fine-tuning (PEFT), and prompt-tuning, it is applicable to downstream tasks. Therefore, the pre-trained model can achieve ideal results in few-shot or zero-shot scenarios. PTMs can be classified into language models, visual models, speech models, multimodal models, etc. according to the data modalities they process. Among them, multimodal models refer to models that establish feature representations of two or more data modalities. The pre-trained model is an important tool for outputting artificial intelligence-generated content and can also serve as a general interface connecting multiple specific task models.
[0072] The solution provided in the embodiments of this application relates to artificial intelligence technology and is specifically described through the following embodiments:
[0073] The method for determining the trajectory of an object provided in the embodiments of this application is used in a robot controller. A robot controller is a device that controls a robot to complete certain actions or tasks according to instructions and sensing information, that is, the robot controller is used to control the robot, including controlling the trajectory of the robot's movement, the actions performed by the robot, and the attitude changes of various parts of the robot.
[0074] In one possible implementation, the robot controller is located on the robot and is a component of the robot. In another possible implementation, the robot controller is an electronic device independent of the robot, and the robot controller is wired or wirelessly connected to the robot.
[0075] A robot is an intelligent machine that can work semi-autonomously or fully autonomously. Robots can perform tasks such as moving, throwing, hitting, etc. through programming and automatic control. The robot can be a semi-mobile robot, that is, the whole robot is fixed at a certain position and only part of it can move, or the robot is a mobile robot, and the whole mobile robot can move. The robot can include various types such as drones, delivery robots, cooking robots, juggling robots, etc. The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, intelligent transportation, etc.
[0076] In a possible implementation manner, the computer program involved in the embodiments of the present application can be deployed to be executed on a robot controller, or on multiple robot controllers located at one place, or on multiple robot controllers distributed at multiple places and interconnected through a communication network. The multiple robot controllers distributed at multiple places and interconnected through a communication network can form a blockchain system.
[0077] In a possible implementation manner, the robot controller in the embodiments of the present application is connected to a server. The server is a node in the blockchain system, and this node can store the data of the robot in the blockchain. Then, this node or other nodes in the blockchain can query the data of the robot by accessing the blockchain.
[0078] Figure 1 is a schematic diagram of an implementation environment provided by the embodiments of the present application. Refer to Figure 1 , this implementation environment includes: a robot 101 and an object 102. The robot 101 includes an end effector 1011 and a robot controller 1012. The robot controller 1012 controls the end effector 1011 to perform any target action on the object 102, such as grasping, throwing, hitting, or pushing the object 102, etc. When the end effector 1011 finishes performing the target action, it will release the object 1011, and then the object 1011 will move a certain distance, thereby forming a trajectory.
[0079] Among them, the end effector 1011 refers to any tool with a certain function connected to the edge of the robot. It includes robot grippers, robot collision sensors, robot rotary connectors, etc.
[0080] In the embodiments of the present application, during the process of the end effector 1011 performing a target action on the object 102, the robot controller 1012 obtains the action data of the robot. In this way, when the end effector 1011 finishes performing the target action on the object, the robot controller 1012 can predict the trajectory of the object based on the action data.
[0081] Figure 1The illustrated implementation environment is only an exemplary implementation environment, and the embodiments of the present application can also be applied in other implementation environments. For example, the robot controller is a handle independent of the robot, and the handle is wirelessly connected to the robot. The user manipulates the handle to control the end effector on the robot to perform a target action.
[0082] The embodiments of the present application are applied to any scenario where a robot performs a target action on an object.
[0083] For example, in the scenario where a juggling robot performs juggling, the juggling robot needs to use multiple arms to throw or catch small balls. After throwing a small ball with one arm each time, the solution provided by the embodiments of the present application is used to predict the trajectory of the small ball, and then based on the trajectory of the small ball, another arm is controlled to catch the small ball.
[0084] Or, in the scenario where a delivery robot delivers items, after the delivery robot holds the item and arrives at the recipient's address, it needs to place the item at the position designated by the recipient by means of actions such as throwing the item or pushing the item. Then the delivery robot first performs multiple simulated actions on the item, that is, after each action on the item is completed, the item is not actually released, but only the method provided by the embodiments of the present application is used to predict the trajectory of the item after each action on the object is completed. Then, a trajectory that can place the item in the correct position is selected from the predicted multiple trajectories, and then the target action corresponding to the trajectory is performed on the item, and the item is released after the target action is completed, so as to place the item in the correct position.
[0085] Figure 2 It is a flowchart of a method for determining trajectory data provided by the embodiments of the present application, which is executed by a robot controller. Refer to Figure 2 The method includes:
[0086] 201. During the process of the robot performing a target action on an object, the robot controller obtains the action data of the robot.
[0087] Among them, the robot includes various types such as a drone, a delivery robot, a cooking robot, a juggling robot, etc. The embodiments of the present application are applied to the scenario where the object is invisible, that is, the embodiments of the present application can predict the trajectory data for any unknown object without the need to pre-understand the physical characteristics of the object, so there is no need to obtain additional prior knowledge of the object. Therefore, the object shown in the embodiments of the present application can be any type of object. In addition, the object can be a rigid object, and the object will not deform due to force when the robot performs an action on the object.
[0088] Among them, the target action includes actions such as grasping, throwing, hitting, or pushing. For example, refer to Figure 3 When the target action is a throwing action, the robot uses the end effector to hold the object (Figure 3 Taking an airplane model as an example), after moving a certain distance and then releasing the object, the object is thrown, and the object will move a certain distance and then land. From the moment the object is released until it lands, a trajectory is formed. Or the target action is a hitting action. After the robot swings the end effector, touches the object and then leaves the object, thus hitting the object. Then the object will move a certain distance and then stop moving. From the moment the object leaves the end effector until it stops moving, a trajectory is formed.
[0089] In addition, the action data represents the state of the robot itself during the process of the robot performing the target action on the object, such as the position, speed, etc. of the robot. Since the object moves due to the action of the robot and thus forms a trajectory, the state of the robot during the process of the robot performing the target action on the object determines the trajectory of the object. Therefore, in the case of ignoring the influence of air resistance, if the action data during the process of the robot performing the target action can be accurately obtained, even without shooting the object, the trajectory data of the object can be directly predicted. This principle stems from the daily decision-making process of humans: for example, through appropriate training and practice, an experienced player can immediately predict the trajectory of a basketball after picking it up and throwing it. This prediction ability is the judgment ability of the player on the result of the shooting action formed after a long time of training and practice. The player mainly relies on proprioception, such as touch, arm muscle tension, and throwing force, etc., to predict the movement of the basketball. Then at the moment when the player throws the basketball, the player can rely on proprioception to predict the trajectory of the basketball, rather than predicting the subsequent trajectory based on the existing trajectory of the basketball after it has moved for a period of time.
[0090] 202. The robot controller processes the action data through the trajectory prediction model to obtain the trajectory data of the object.
[0091] Among them, the trajectory prediction model is used to predict the trajectory data of the object after the robot completes the target action based on the action data of the robot. It takes the action data of the robot as the input and the trajectory data of the object as the output. The trajectory data represents the trajectory of the object after the robot completes the target action on the object.
[0092] In a possible implementation manner, the trajectory prediction model is a deep learning model, a recurrent neural network model, an LSTM (Long Short-Term Memory) model or other models, and the embodiments of the present application do not make limitations thereto.
[0093] In the embodiments of the present application, a task of predicting the trajectory of a random object for robot control is formulated. Therefore, a trajectory prediction model Φ: X→Y is first obtained. The action data X collected by the robot during the process of the robot performing a target action on the object is used as the input of the trajectory prediction model, and the trajectory data Y corresponding to the trajectory starting from the time point when the robot finishes performing the target operation on the object is processed and obtained. The embodiments of the present application associate the trajectories of any unknown objects with the action data perceived by the robot body, use the action data perceived by the robot body to predict the trajectory data of the object, do not require prior knowledge of the object or observing the object from the outside, the method is more portable and reliable, and is also easy to be applied to cluttered and unstructured environments, expanding the application scope of the robot.
[0094] In a possible implementation manner, the trajectory prediction model is an LSTM model. The action data of the robot includes the action data at at least two first time points. Here, the first time point is the time point during the process of the robot performing a target action on the object, that is, the time point from the start of performing the target action on the object to before the completion of the target action. Therefore, the LSTM model can perform predictions in multiple time steps. At each time step, the action data at a first time point is obtained. The action data is processed through the LSTM model. Then, as the time for the robot to perform the target action on the object becomes longer, more action data at the first time points will be obtained, and thus more action data processed by the LSTM model. Then, when the robot finishes performing the target action on the object, the action data at the last first time point can be obtained. After the action data at the last first time point is processed through the LSTM model, the trajectory data of the object can be quickly determined according to the processing results of the action data at each first time point. The recursiveness of the LSTM model can process the process of performing the target action and the process of predicting the trajectory data in parallel. Once the robot finishes performing the target action, the entire trajectory data can be output, greatly shortening the prediction time, which is particularly beneficial for time-critical application scenarios.
[0095] The method provided by the embodiments of the present application can obtain the action data of the robot during the process of the robot performing a target action on the object, and after being processed through the trajectory prediction model, predict the trajectory data of the object. After the robot finishes performing the target action on the object, the trajectory of the object can be predicted immediately, without the need to predict after shooting the object for a period of time by a motion capture sensor after the robot finishes performing the target action on the object, saving time consumption and improving efficiency. Moreover, the embodiments of the present application summarize and learn from the daily actions of humans, and determine that the state of the robot during the process of the robot performing a target action on the object determines the trajectory of the object. Therefore, a trajectory prediction model with the action data during the process of the robot performing a target action on the object as the input and the trajectory data of the object as the output is used for prediction, ensuring the accuracy of the predicted trajectory data.
[0096] Based on the above Figure 2 shown embodiments, the embodiments of the present application further provide a trajectory prediction model. The trajectory prediction model adopts an "encoder-decoder" architecture and includes an encoder and a decoder. Figure 4 is a flowchart of a method for determining trajectory data provided by the embodiments of the present application, which is executed by a robot controller. Refer to Figure 4 and the method includes:
[0097] 401. During the process of the robot performing a target action on an object, the robot controller acquires the action data of the robot.
[0098] Step 401 is the same as step 201 and will not be elaborated here.
[0099] 402. The robot controller encodes the action data through the encoder in the trajectory prediction model to obtain action features.
[0100] Among them, the trajectory prediction model is used to predict the trajectory data of the object after the robot completes the target action based on the action data of the robot. The trajectory prediction model adopts an "encoder-decoder" architecture and includes an encoder and a decoder. The encoder is used to take the action data of the robot as input, encode the action data, and thus convert the action data into action features. The action features can describe the state during the process of the robot performing the target action on the object in numerical form.
[0101] In a possible implementation manner, the encoder in the trajectory prediction model is φ toss , the action data is X, and the action feature is e, then e = φ toss (X). Here, e represents the learned action features, including the features during the process of the robot performing the target action and the features of the interaction between the robot and the object due to the physical properties of the object, etc.
[0102] In a possible implementation manner, the encoder includes one or more LSTM modules.
[0103] 403. The robot controller decodes the action features through the decoder in the trajectory prediction model to obtain the trajectory data of the object.
[0104] Among them, the decoder is used to decode the action features and thus convert the action features into the trajectory data of the object. The trajectory data represents the trajectory of the object after the robot completes the target action on the object.
[0105] In a possible implementation manner, the decoder in the trajectory prediction model is φ obj, if the action feature is e and the trajectory data is Y, then Y = φ obj (e).
[0106] In one possible implementation, the decoder includes one or more LSTM modules.
[0107] In one possible implementation, the trajectory prediction model is an LSTM model. For example, the encoder in the trajectory prediction model includes an LSTM module. Through the encoder, the action data is encoded to obtain action features, including: when the action data at a first time point is obtained each time, the action data is encoded through the encoder to obtain the action feature at the first time point, where the first time point is a time point during the process of the robot performing a target action on an object. As the robot performs the target action on the object for a longer time, more action data at the first time points will be obtained, so that more action features corresponding to the action data can be obtained through the encoder. Then, when the robot finishes performing the target action on the object, the action data at the last first time point can be obtained, and the action data at the last first time point is encoded through the encoder to obtain the last action feature. Then, when the robot finishes performing the target action on the object, at least two obtained action features at the first time points are combined; through the decoder, the combined action features are decoded to obtain the trajectory data. The recursiveness of the LSTM model can process the process of performing the target action and the process of predicting the trajectory data in parallel. The encoder has already started encoding the action data during the process of the robot performing the target action, so that once the robot finishes performing the target action, the entire trajectory data can be output, greatly shortening the prediction time, which is particularly beneficial for time-critical application scenarios.
[0108] The method provided by the embodiments of the present application can obtain the action data of the robot during the process of the robot performing a target action on an object, convert the action data into action features through the encoder in the trajectory prediction model, and then obtain the trajectory data of the object through the decoder in the trajectory prediction model. Therefore, the trajectory of the object can be predicted immediately after the robot finishes performing the target action on the object, without having to predict after using a motion capture sensor to photograph the object for a period of time after the robot finishes performing the target action on the object, saving time consumption and improving efficiency. Moreover, the embodiments of the present application summarize and learn from human daily actions, and determine that the state of the robot during the process of the robot performing a target action on an object determines the trajectory of the object. Therefore, a trajectory prediction model with the action data during the process of the robot performing a target action on an object as the input and the trajectory data of the object as the output is used, ensuring the accuracy of the predicted trajectory data.
[0109] In the above Figure 2 and Figure 4Based on the embodiments shown, the embodiments of the present application further provide another trajectory prediction model. Refer to Figure 5 , the trajectory prediction model includes an encoder and a decoder. The encoder includes at least two encoding blocks ( Figure 5 taking 3 encoding blocks as an example) and a fusion layer. One encoding block corresponds to one type and is used to encode the action data of one type, while the fusion layer is used to fuse the action features corresponding to the action data of at least two types, so as to comprehensively consider the influence of different types of action data on the object trajectory.
[0110] The embodiments of the present application further provide another method for determining trajectory data. This method applies Figure 5 the trajectory prediction model shown. Figure 6 FIG. is a flowchart of another method for determining trajectory data provided by the embodiments of the present application. The embodiments of the present application are executed by a robot controller. Refer to Figure 6 , this method includes:
[0111] 601. During the process of the robot performing a target action on an object, the robot controller acquires the action data of the robot. The action data represents the state of the robot itself during the process of the robot performing the target action on the object.
[0112] Step 601 is the same as step 201 and will not be elaborated here.
[0113] In addition, it should be noted that considering that the state of the robot itself can be described from multiple different perspectives, the types of action data can include multiple types, such as pose parameters, speed, force or torque, etc. In the embodiments of the present application, the robot controller can acquire any one or more types of action data.
[0114] In a possible implementation manner, the robot includes an end effector for performing an action on an object. The action data of the robot includes the action data of the end effector. Then, during the process of the robot performing a target action on an object, acquiring the action data of the robot includes at least one of the following:
[0115] 6011. Acquire the pose parameters of the end effector at at least two first time points. The pose parameters include position and orientation.
[0116] Among them, the first time point is the time point during the process of the robot performing the target action on the object, that is, the time point from when the end effector starts to perform the target action on the object to when the target action is completed. For example, the robot controller receives a target instruction indicating to perform a target action on an object. Then, the robot controller responds to the target instruction and starts to perform the target action. At this time, the pose parameters of the end effector can be collected until the end effector completes the target action and then stops collecting the pose parameters of the end effector.
[0117] Considering that, when ignoring the influence of the grasping force of the robot on the object and the physical characteristics of the robot, the changes in the position and posture of the robot directly reflect the action characteristics during the process of the robot performing the target action. Therefore, by collecting the pose parameters of the end effector at multiple first time points on the robot body, a sequence of pose parameters is obtained, effectively capturing the dynamic operation process of the end effector. The state changes of the end effector in time and space can be learned from the sequence of pose parameters, so as to subsequently predict the trajectory data of any unknown object based on the state of the end effector in time and space.
[0118] In a possible implementation, an interval duration is set. After the end effector starts to perform the target action, the pose parameters of the end effector are collected every such interval duration until the end effector completes the execution of the target action.
[0119] For example, the obtained sequence of pose parameters where i represents the serial number of the pose parameter, its initial value is 0, the number of pose parameters is m1 + 1, that is, the number of the first time points is m1 + 1, m1 is a positive integer, x t represents the pose parameter at the first time point t, for example x t ∈SE(3), indicating that x t includes a rotation matrix and a translation matrix. In a possible implementation, the interval duration between any two first time points is T1, then the duration for detecting the pose parameters is m1T1, and the time point when the end effector completes the execution of the target action on the object is set as the time point t = 0, then the following condition needs to be satisfied: t i =(i - m1)T1, where t i represents the i-th first time point.
[0120] 6012. Obtain the speeds of the end effector at at least two first time points.
[0121] Among them, the first time point is the time point during the process of the robot performing the target action on the object, that is, the time point from when the end effector starts to perform the target action on the object to before the completion of the execution of the target action. For example, the robot controller receives a target instruction, and the target instruction indicates to perform a target action on the object. Then the robot controller responds to the target instruction and starts to perform the target action. At this time, the speed of the end effector can be collected, and the collection of the speed of the end effector stops until the end effector completes the execution of the target action. This speed can include translational speed, angular velocity, etc.
[0122] Considering that, when ignoring the influence of the grasping force of the robot on the object and the physical characteristics of the robot, the change in the speed of the robot directly reflects the action characteristics during the execution of the target action by the robot. Therefore, by collecting the speeds of the end effector at multiple first time points on the robot body, a sequence of speeds is obtained, effectively capturing the dynamic operation process of the end effector. The state of the end effector in time and space can be learned from the sequence of speeds, so as to predict the trajectory data of any unknown object based on the state of the end effector in time and space in the future.
[0123] In a possible implementation, an interval duration is set. After the end effector starts to execute the target action, the speed of the end effector is collected every such interval duration until the end effector completes the execution of the target action.
[0124] For example, the obtained speed sequence where i represents the serial number of the speed, its initial value is 0, and the number of speeds is m2 + 1, that is, the number of first time points is m2 + 1, and m2 is a positive integer. represents the speed at the first time point t. That is belongs to a six-dimensional real number space. In a possible implementation, the interval duration between any two first time points is T2, then the duration for detecting the pose parameters is m2T2, and the time point when the end effector completes the execution of the target action on the object is set as the time point t = 0, then the following condition t i =(i - m2)T2, where t i represents the i-th first time point.
[0125] The above step 6012 and the above step 6011 can be combined, that is, the robot controller can obtain the pose parameters of the end effector at multiple first time points and the speeds at multiple first time points. In one case, the robot controller can obtain the pose parameters and speeds asynchronously, then the first time points corresponding to the pose parameters and the first time points corresponding to the speeds may be the same or different. In another case, the robot controller can obtain the pose parameters and speeds synchronously, then the first time points corresponding to the pose parameters and the first time points corresponding to the speeds are the same.
[0126] 6013. A force / torque sensor is configured on the joint corresponding to the end effector, and the force and torque of the end effector at at least two first time points are obtained through the force / torque sensor.
[0127] The robot controller controls the movement or rotation of the joints corresponding to the end effector, so as to control the end effector to perform the target action. A force / torque sensor is configured on the joints corresponding to the end effector, which is equivalent to configuring a force / torque sensor on the "wrist" of the robot, so as to obtain the force and torque of the end effector of the robot. For example, the force / torque sensor is a six-axis force / torque sensor and is installed between the robotic arm and the end effector of the robot.
[0128] Among them, the first time point is the time point during the process of the robot performing the target action on the object, that is, the time point from when the end effector starts to perform the target action on the object to before the target action is completed. For example, the robot controller receives a target instruction, and the target instruction indicates to perform a target action on the object. Then the robot controller responds to the target instruction and starts to perform the target action. At this time, the force and torque of the end effector can be collected, and the collection of the force and torque of the end effector stops until the end effector completes the target action.
[0129] By collecting the force and torque of the end effector at multiple first time points on the robot body, a sequence of force and torque is obtained, and the action characteristics inherited from the interaction between the robot and the object can be extracted, effectively capturing the dynamic operation process of the end effector. The state of the end effector in time and space can be learned from the sequence of force and torque, so that the trajectory data of any unknown object can be predicted based on the state of the end effector in time and space in the future. Moreover, collecting the force and torque of the end effector through the force / torque sensor is an efficient, lightweight and reliable method, which can be applied to many robot systems. Therefore, it is ensured that the method provided by the embodiments of the present application can be easily applied to different robot systems, and the applicable range is relatively wide.
[0130] In a possible implementation manner, an interval duration is set. After the end effector starts to perform the target action, the force and torque of the end effector are collected every such interval duration until the end effector completes the target action.
[0131] For example, the obtained sequence of force and torque where i represents the serial number of the force and torque, its initial value is 0, the number of forces and torques is m3 + 1, that is, the number of first time points is m3 + 1, m3 is a positive integer, f t represents the force and torque at the first time point t, f t ∈R 6 That is, f tBelongs to a six-dimensional real space. In one possible implementation, the time interval between any two first time points is T3, the duration for detecting pose parameters is m3T3, and the time point when the end effector completes the target action on the object is set as the time point t = 0. Then the following condition t i =(i - m3)T3, where t i represents the i-th first time point.
[0132] See Figure 7 , the end effector of the robot grasps different positions of the same object respectively and performs a throwing action on the object, which will result in different F / T (force / torque) obtained during the throwing action. See Figure 8 , Figure 8 The horizontal axis of the curve shown in Figure 7 represents time, with the unit of 0.01 seconds. The force / torque sensor is a six-axis force / torque sensor. If Figure 7 the robot shown in the first figure in
[0133] performs ten throwing actions on the object, then the change curves of the average values of the ten forces in the x-axis, y-axis, and z-axis directions obtained through the force / torque sensor are 802, 804, and 806 respectively, and the shaded area where each curve is located represents the range composed of the maximum and minimum values of the ten forces. The change curves of the average values of the ten torques in the x-axis, y-axis, and z-axis directions are 808, 810, and 812 respectively, and the shaded area where each curve is located represents the range composed of the maximum and minimum values of the ten torques. If Figure 9 , Figure 9 the first figure in Figure 9 shows the change curve of the positions (represented by p) of the end effectors on the 6 robotic arms of the robot, Figure 9The third figure in [figure number] shows the variation curve of the force / torque (denoted as F / T) of the end effectors on the six robotic arms of the robot.
[0134] In one case, the robot controller can asynchronously obtain at least two types of motion data, and the first time points corresponding to at least two types of motion data among the pose parameters, speed, and force and torque can be the same or different. In another case, the robot controller can synchronously obtain at least two types of motion data, and the first time points corresponding to at least two types of motion data among the pose parameters, speed, and force and torque are the same.
[0135] If the object is photographed by a motion capture sensor, it is very likely that the captured image of the object obtained by the motion capture sensor will be inaccurate due to the object being blocked or the captured image being damaged, which will in turn lead to an inaccurate predicted trajectory for the object. However, in the embodiments of the present application, the object is not photographed by a motion capture sensor, but the motion data of the robot itself is obtained, avoiding inaccurate trajectories caused by the object being blocked or the captured image being damaged, and improving reliability.
[0136] Moreover, the process of the robot performing a target action on an object is essentially a continuous and highly dynamic interaction among the robot, the object, and the environment (such as air resistance). Therefore, the trajectory of the object is affected by the robot, the object, and the environment, and features that can accurately measure the interaction among the robot, the object, and the environment need to be extracted in order to accurately predict the trajectory of the object. Referring to Figure 10 , for the same robot and the same object, when the robot grasps the object from different positions on the object, even if the robot performs the same target action on the object, the trajectory of the object is different. Referring to Figure 11 , after the robot grasps the object and during the process of performing the target action on the object, the object will gradually shift, resulting in a change in the relative position between the object and the robot. For example, due to the limited speed and friction of the robot, the object will slide inside the end effector of the robot, causing a change in the trajectory of the object. And this interaction between the robot and the object is complex and variable, making it difficult to capture. In the method provided by the embodiments of the present application, by obtaining multiple types of motion data of the robot, including pose parameters, speed, force, and torque, the interaction between the robot and the object is characterized by multi-modal motion data, which is more accurate and comprehensive, and helps to accurately predict the trajectory of the object.
[0137] In a possible implementation, the time interval between any two adjacent first time points is equal, and the duration of obtaining the motion data can be set according to requirements, such as set to 0.1 seconds, 0.5 seconds, etc. The longer the duration, the longer the sequence of the obtained motion data, and the more motion information can be included, thus improving the prediction accuracy.
[0138] In an embodiment of the present application, the robot controller needs to create an attached coordinate system of an object, so as to obtain the motion data of the robot based on the attached coordinate system, so that the motion data represents the state of the robot in the attached coordinate system during the process of the robot performing a target action on the object, and then use the attached coordinate system to parameterize the trajectory of the object. The pose of the object can be represented by the translation matrix and rotation matrix of the attached coordinate system relative to the reference coordinate system, and the reference coordinate system is the world coordinate system or the robot coordinate system, etc.
[0139] The creation of the attached coordinate system will affect the trajectory of the object. The traditional attached coordinate system takes a point on the object as the origin, and determines the coordinate system of the attached coordinate system based on the pose of the object. Then, as the position of the object moves, the position of the attached coordinate system also moves, and when the pose of the object changes, the direction of the coordinate axes of the attached coordinate system also changes with the change of the object's pose. For example, refer to Figure 12 , for two different objects with the same physical characteristics (such as mass, centroid position, and moment of inertia): Object 1 and Object 2, if the same robot performs a throwing operation on these two objects, the trajectories of the two objects should be the same, and the motion data obtained during the prediction process should also be the same. However, if different attached coordinate systems are used for these two objects, it will result in different motion data being obtained and different predicted trajectories (dashed trajectories). In addition, for the same symmetric and uniformly distributed object, if the robot throws the object with the same throwing action, the trajectory of the object should be the same. But refer to Figure 13 A represents one end of the object, Figure 13 in the first figure of Figure 13 , the end effector of the robot grabs one end opposite to end A,
[0140] Therefore, in order to avoid the above problems, the embodiments of the present application create a target attached coordinate system. The target attached coordinate system takes the centroid of the object grasped by the end effector as the origin. The centroid of the object is unique, and the motion pattern of the centroid is consistent in different objects, and the motion pattern of the centroid of the object is relatively simple, which is convenient for the model to learn the motion characteristics of the object. And the directions of each coordinate axis in the target attached coordinate system are determined based on the directions of each coordinate axis of the end effector. The action data represents the state of the robot in the target attached coordinate system during the process of the robot performing the target action on the object. Even if the object changes, the target attached coordinate system adopted by the robot will not change, or if the target action changes, the target attached coordinate system adopted by the robot will not change. Then refer to Figure 12 , the target attached coordinate system takes the centroid b' on object 1 and object 2 as the origin, and the x-axis and z-axis of the target attached coordinate system are as Figure 12 shown. The directions of each coordinate axis of the target attached coordinate system are determined based on the posture of the end effector. Then, if the same robot performs a throwing operation on these two objects, the trajectories (solid trajectories) of the two objects are the same. Refer to Figure , the target attached coordinate system takes the centroid b' on the object as the origin, and the x-axis and z-axis of the target attached coordinate system are as shown. The directions of each coordinate axis of the target attached coordinate system are determined based on the posture of the end effector. Then, although the object has flipped, the position of the end effector has not changed. Therefore, the target attached coordinate system is the same, and the trajectories (solid trajectories) generated by the two throwing actions are the same.
[0141] In a possible implementation manner, the directions of each coordinate axis in the target attached coordinate system are determined based on the directions of each coordinate axis of the end effector, including: determining the initial posture of the end effector at the start of performing the target action as the initial posture of the target attached coordinate system, that is, determining the directions of each coordinate axis in the initial posture of the end effector as the initial directions of each coordinate axis of the target attached coordinate system. That is, R b’ =I, R b’ represents the initial posture of the target attached coordinate system, and I represents the initial posture of the end effector. Or, at the start of performing the target action, the initial posture of the end effector is not equal to the initial posture of the target attached coordinate system, but there is a fixed conversion relationship between the two. For example, R b’ =B*I, R b’ represents the initial posture of the target attached coordinate system, R b’ SO(3), I represents the initial posture of the end effector, B represents the conversion matrix, and B is a constant with a constant value.
[0142] In one possible implementation, the action data is represented based on a reference coordinate system, but the action data represents the state of the robot in the target attached body coordinate system during the process of the robot performing a target action on an object. For example, the position of the object is represented as the translational displacement of the target attached body coordinate system relative to the reference coordinate system, and the attitude of the object is represented as the rotational displacement of the target attached body coordinate system relative to the reference coordinate system. Additionally, the attitude of the object can be represented in the form of an axis angle, i.e., o t ∈R 3 ,o t represents the attitude of the object.
[0143] In another possible implementation, if the robot creates the above-mentioned target attached body coordinate system and also creates other attached body coordinate systems of the object, then the position and attitude of the object, as well as the trajectory data of the object, can be represented based on the target attached body coordinate system or based on other attached body coordinate systems, and they can be mutually converted through the constant transformation matrix between the target attached body coordinate system and other attached body coordinate systems.
[0144] See , the dashed line represents the axes of the traditional attached body coordinate system, the solid line represents the axes of the target attached body coordinate system, A represents one side of the object, then by comparing the three figures in , it can be seen that when the attitude of the object changes, the attitude of the traditional attached body coordinate system also changes, while the attitude of the target attached body coordinate system is consistent with the attitude of the end effector grasping the object, and through the constant transformation matrix R between these two attached body coordinate systems b b’ can perform conversion between the two attached body coordinate systems. See , three coordinate systems are created: r represents the reference coordinate system, g represents the joint coordinate system of the robot, b′ represents the target attached body coordinate system. The shape or attitude of the object changes, but since the attitude of the end effector does not change, the attitude of the target attached body coordinate system does not change.
[0145] 602. The robot controller encodes the action data of at least two corresponding types through at least two encoding blocks to obtain at least two action features.
[0146] The trajectory prediction model in the embodiments of the present application is an end-to-end model, which is independent of the object, can perform body detection, obtain the action data of the robot, learn the action features of the robot during the process of performing a target action on any unknown object from the action data, combine the action features during the process of the robot performing the target action, and map the action features to the estimation of the object trajectory, thereby predicting the trajectory data of the object. It is a lightweight but effective and accurate operation method.
[0147] In a possible implementation, the trajectory prediction model is an end-to-end, sequence-to-sequence deep neural network. The input is a sequence composed of action data at at least two time points, and the output is a sequence composed of pose parameters of at least two waypoints on the trajectory.
[0148] The action data includes at least two types. The encoder includes at least two encoding blocks, and one encoding block corresponds to one type and is used to encode one type of action data. Then, at least two types of action data can be encoded by at least two encoding blocks to obtain action features corresponding to at least two types of action data. The action features represent the features of the state of the robot during the process of the robot performing the target action on the object. Among them, each encoding block can include an LSTM module or other types of encoding blocks.
[0149] See , the encoder includes a first encoding block, a second encoding block, and a third encoding block. The first encoding block is used to encode pose parameters, the second encoding block is used to encode speed, and the third encoding block is used to encode force and torque.
[0150] 603. The robot controller fuses at least two types of action features through a fusion layer to obtain the fused action features.
[0151] The encoder further includes a fusion layer, and the fusion layer is used to fuse the action features corresponding to different types of action data to obtain the fused action features. Among them, the fusion layer can be a fully connected layer or other types of layers.
[0152] Although each type of action data describes the action execution of the robot to a certain extent, it is found through research that the spatio-temporal correlation between different types of action data also contains key features that affect the object trajectory. Therefore, the embodiment of the present application adopts a fine-grained fusion strategy to extract the correlation between different types of action data, further improving the operation accuracy.
[0153] In a possible implementation, the encoder further includes an associated encoding block. The associated encoding block is used to extract features representing the correlation between different types of action features. Then, through the fusion layer, at least two types of action features are fused to obtain the fused action features, including: encoding at least two types of action data through the associated encoding block to obtain associated action features, where the associated action features represent the correlation between at least two types of action features; and through the fusion layer, fusing at least two types of action features and the associated action features to obtain the fused action features, such that the fused action features not only contain the influence of each type of action feature but also the influence of the correlation between different types of action features. For example, during the process of a robot throwing an object, the actions of the robot are related to the action data such as force / moment at each path point detected. This correlation may contain additional clues helpful for learning action features.
[0154] Exemplarily, the associated encoding block includes at least two encoding layers. Encoding at least two types of action data through the associated encoding block to obtain associated action features includes: encoding at least two types of action data through the first encoding layer in the associated encoding block to obtain the first associated action feature; encoding the first associated action feature through the second encoding layer in the associated encoding block to obtain the second associated action feature, until the associated action feature is obtained through the last encoding layer in the associated encoding block. For example, is a schematic structural diagram of an associated encoding block provided by an embodiment of the present application. Refer to the associated encoding block includes at least two cascaded encoding layers. By encoding sequentially through the at least two encoding layers, the accuracy of the associated action features can be improved.
[0155] In another possible implementation, if different types of action data cannot be synchronously acquired due to different sampling frequencies and transmission delays, resulting in the misalignment of the first time points corresponding to different types of action data, then encoding is no longer performed through the associated encoding block.
[0156] 604. The robot controller decodes the action features through a decoder to obtain the trajectory data of the object. The trajectory data represents the trajectory of the object after the robot completes the target action on the object.
[0157] Among them, the decoder can be an LSTM module or other types of modules.
[0158] In a possible implementation, the decoder uses an autoregressive algorithm to decode the action features to obtain the trajectory data, so as to improve the accuracy of the trajectory data.
[0159] In another possible implementation, the trajectory data of the object includes the pose parameters of the object at at least two second time points, where the pose parameters include position and attitude, and the second time point is the time point after the robot completes the target action on the object. Exemplarily, the time interval between any two adjacent second time points is equal.
[0160] The decoder in the embodiments of the present application does not need to rely on the prior knowledge of the object or the previous trajectory of the object after the target action on the object is completed, and only predicts the trajectory data of the object according to the learned action features. Therefore, it can be realized immediately after the target action on the object is completed.
[0161] For example, refer to , the robot controller obtains the action data set The action data set includes multi-modal action data, and then it is processed through the trajectory prediction model φ: X→Y to obtain the trajectory data corresponding to the trajectory of the object starting from the time point (t = 0) when the robot completes the target operation on the object. Where Y represents the trajectory data, t represents the time point, j represents the serial number of the path point in the trajectory of the object, its initial value is 1, the maximum value is n, n is the number of path points included in the trajectory of the object, n is a positive integer, t = jT, T represents the time interval between any two adjacent path points, and nT represents the duration of the trajectory of the object, p t represents the position of the object at the time point t, and o t represents the attitude of the object at the time point t.
[0162] After predicting the trajectory data of the object, the robot controller can control the end effector to release the object so that the object moves along the trajectory indicated by the trajectory data. Alternatively, the robot controller may not control the end effector to release the object, but instead re-execute other target actions to predict the trajectory data corresponding to other target actions, and then select the trajectory data that meets the conditions from multiple trajectory data, control the end effector to perform the target action corresponding to the selected trajectory data on the object and then release the object, or other operations may also be performed after predicting the trajectory data of the object. The embodiments of the present application do not limit this.
[0163] In a possible implementation, the trajectory prediction model is an LSTM model. For example, the encoder in the trajectory prediction model includes an LSTM module. Through the encoder, the action data is encoded to obtain action features, including: in the case of obtaining the action data at a first time point each time, the above steps 602 and 603 are executed to obtain the action features at the first time point, where the first time point is a time point during the process of the robot performing the target action on the object. As the robot performs the target action on the object for a longer time, more action data at the first time points will be obtained, so that more action features corresponding to the action data can be obtained through the encoder. Then, when the robot finishes performing the target action on the object, the action data at the last first time point can be obtained, and thus the above steps 602 and 603 are executed to encode the action data at the last first time point to obtain the last action feature. Then, when the robot finishes performing the target action on the object, at least two obtained action features at the first time points are combined; through the decoder, the combined action features are decoded to obtain the trajectory data. The recursiveness of the LSTM model can process the process of performing the target action and the process of predicting the trajectory data in parallel. During the process of the robot performing the target action, the encoder has already started to encode the action data, so that once the robot finishes performing the target action, the entire trajectory data can be output, greatly shortening the prediction time, which is particularly beneficial for time-critical application scenarios.
[0164] The method provided by the embodiments of the present application can obtain the action data of the robot during the process of the robot performing the target action on the object. After being processed by the encoder and decoder of the trajectory prediction model, the trajectory data of the object is predicted. After the robot finishes performing the target action on the object, the trajectory of the object can be predicted immediately, without the need to predict after the robot finishes performing the target action on the object and shooting the object for a period of time by a motion capture sensor. Almost the entire trajectory of the object can be predicted in real time, saving time consumption and improving efficiency. Moreover, the embodiments of the present application summarize and learn from human daily actions, and determine that the state of the robot during the process of the robot performing the target action on the object determines the trajectory of the object. Therefore, a trajectory prediction model with the action data during the process of the robot performing the target action on the object as the input and the trajectory data of the object as the output is used for prediction, ensuring the accuracy of the predicted trajectory data. The solution of the embodiments of the present application can improve the flexibility and versatility of the robot in various scenarios, and contribute to the development of more versatile robots.
[0165] In addition, in the related art, an object is photographed by a motion capture sensor, such as a high-speed camera, that is, the state of the object is observed from the outside, which is easily affected by the external environment, thus encountering difficulties in processing data based on external observations, restricting the application scenarios. For example, it is difficult to obtain an undamaged and fully complete photographed image in a wide open space. Therefore, the related art cannot be applied to predicting the trajectory of an object in a wide open space, or the external environment is dynamically changing, resulting in an unstable photographed image and insufficient accuracy. However, the method provided in the embodiments of the present application does not photograph the object through a motion capture sensor, but obtains the motion data of the robot itself through the body perception mode, and then predicts the trajectory data through the trajectory prediction model. The application scenario of this method is not restricted and is very suitable for use in a more dynamic environment, expanding the application range.
[0166] In the embodiments of the present application, considering that when ignoring the influence of the grasping force of the robot on the object and the physical characteristics of the robot, the changes in the position and posture of the robot directly reflect the action characteristics during the execution of the target action by the robot. Therefore, by collecting the pose parameters of the end effector at multiple first time points on the robot body, a sequence of pose parameters is obtained, effectively capturing the dynamic operation process of the end effector. The state changes of the end effector in time and space can be learned from the sequence of pose parameters, so as to be able to predict the trajectory data of any unknown object based on the state of the end effector in time and space in the future, improving the prediction accuracy.
[0167] In the embodiments of the present application, considering that when ignoring the influence of the grasping force of the robot on the object and the physical characteristics of the robot, the changes in the speed of the robot directly reflect the action characteristics during the execution of the target action by the robot. Therefore, by collecting the speeds of the end effector at multiple first time points on the robot body, a sequence of speeds is obtained, effectively capturing the dynamic operation process of the end effector. The state of the end effector in time and space can be learned from the sequence of speeds, so as to be able to predict the trajectory data of any unknown object based on the state of the end effector in time and space in the future, improving the prediction accuracy.
[0168] In the embodiments of the present application, by collecting the forces and torques of the end effector at multiple first time points on the robot body, a sequence of forces and torques is obtained, and action features inherited from the interaction between the robot and the object can be extracted, effectively capturing the dynamic operation process of the end effector. The state of the end effector in time and space can be learned from the sequence of forces and torques, so that the trajectory data of any unknown object can be predicted based on the state of the end effector in time and space. Moreover, collecting the forces and torques of the end effector through a force / torque sensor is an efficient, lightweight, and reliable method, which can be applied to many robot systems. Therefore, it is ensured that the method provided by the embodiments of the present application can be easily applied to different robot systems, and has a wide range of applications.
[0169] In the embodiments of the present application, by obtaining various types of action data of the robot, including pose parameters, speed, force, and torque, the interaction between the robot and the object is characterized by multi-modal action data, which is more accurate and comprehensive. Each type of action data describes the action execution of the robot to a certain extent, and the spatio-temporal correlation between different types of action data also contains key features that affect the object trajectory. Therefore, a fine-grained fusion strategy is adopted to extract the correlation between different types of action data, further improving the operation accuracy and helping to accurately predict the object trajectory.
[0170] In the embodiments of the present application, at least two types of action data are encoded by an association coding block to obtain association action features, which represent the correlation between at least two of the action features. Through a fusion layer, at least two action features and the association action features are fused to obtain fused action features, so that the fused action features not only contain the influence of each type of action feature, but also contain the influence of the correlation between different types of action features, further improving the operation accuracy.
[0171] In the embodiments of the present application, the process of executing the target action and the process of predicting the trajectory data are processed in parallel. During the process of the robot executing the target action, the encoder has already started encoding the action data, so that once the robot completes the execution of the target action, the entire trajectory data can be output, greatly shortening the prediction time.
[0172] In the embodiment of the present application, the target attached body coordinate system takes the centroid of the object as the origin. Since the centroid of the object is unique, the motion patterns of the centroids in different objects are consistent, and the motion pattern of the centroid of the object is relatively simple, which is convenient for the model to learn the motion characteristics of the object. Moreover, the directions of each coordinate axis in the target attached body coordinate system are determined based on the directions of each coordinate axis of the end effector. The action data represents the state of the robot in the target attached body coordinate system during the process of the robot performing the target action on the object. For different objects or different target actions, the target attached body coordinate system adopted by the robot will not change, or even if the target action changes, the target attached body coordinate system adopted by the robot will not change, avoiding the situation where the predicted trajectory data is inconsistent when the robot performs the same target action on objects with the same physical characteristics due to the influence of the attached body coordinate system.
[0173] The following embodiments will illustrate the training process of the trajectory prediction model:
[0174] The process of the robot performing the target action on the object mainly depends on the interaction between the robot and the object. Therefore, it is particularly important to accurately perceive the action characteristics of the robot. In many related technologies, it is assumed that the physical characteristics of the object are known, and then the trajectory of the object is predicted based on human insights into the physical characteristics of the object. However, the models constructed according to such assumptions have great limitations in terms of accuracy and generalization. This is because there are non-linear relationships in the process of object interaction, such as the end effector of the robot being worn due to repeated use. The embodiment of the present application creates a trajectory prediction model for any unknown object, which can predict the trajectory of the object based on the action data of the robot without considering the physical characteristics of the object, so that it can be applied to any type of object and improve the generalization.
[0175] In a possible implementation manner, the action data of the robot includes the action data of the robot at at least two first time points, and the trajectory data of the object includes the pose parameters of the object at at least two second time points. The pose parameters include position and attitude; wherein, the first time point is the time point during the process of the robot performing the target action on the object, and the second time point is the time point after the robot completes the target action on the object.
[0176] Training the trajectory prediction model includes: obtaining sample data from the sample dataset, and training the trajectory prediction model based on the sample data. The sample data includes sample action data and sample trajectories. The sample action data includes the action data of the robot at at least two first sample time points, and the sample trajectories include the pose parameters of the sample object at at least two second sample time points; wherein, the first sample time point is the time point during the process of the robot performing the target action on the sample object, and the second sample time point is the time point after the robot completes the target action on the sample object.
[0177] In the embodiments of the present application, a robot executes a target action on a sample object to obtain sample data, and then trains a trajectory prediction model based on the sample data, so that the trajectory prediction model has the ability to predict the object trajectory based on the action data during the process of the robot executing the target action on any unknown object, thereby improving the prediction accuracy of the trajectory prediction model.
[0178] In the embodiments of the present application, the robot executes a target action on the sample object and records the trajectory of the sample object after the execution of the target action is completed, so as to obtain sample data. In one possible implementation, the robot executes a target action on the sample object in a simulation environment to create a sample data set. In another possible implementation, the robot executes a target action on the sample object in a real environment to create a sample data set. For example, a dual-robot system is created. The dual-robot system includes a recycling robot and a throwing robot. In each acquisition cycle, the recycling robot transfers the object in the container to the throwing robot. After the throwing robot catches the object, it executes a throwing action and releases the object, so as to throw the object into the container. The recycling robot catches the object again to start the next acquisition cycle. The pose parameters, speed, and force / torque of the throwing robot are obtained as input data in each acquisition cycle, and an optical motion capture system is used to record the trajectory of the object from the moment it is released by the throwing robot to the moment it lands.
[0179] In one possible implementation, the sample data set includes at least one of the following:
[0180] (1) The first sample data set. The sample data in the first sample data set is collected under a first condition. The first condition is that the robot uses the same end effector to execute a target action on the same sample object, and the pose and speed of the end effector when releasing the sample object after the execution of the target action are not completely the same. This first sample data set is used to evaluate the ability of the trajectory prediction model to capture the action data of the robot.
[0181] (2) The second sample data set. The sample data in the second sample data set is collected under a second condition. The second condition is that the robot uses a random end effector to execute a target action on the same sample object, and the pose and speed of the robot when releasing the sample object after the execution of the target action are random. This second sample data set is used to evaluate the ability of the trajectory prediction model to capture the action data of the robot and determine the end effector used by the robot.
[0182] (3) The third sample dataset, where the sample data in the third sample dataset is collected under the third condition. The third condition is that the robot uses a random end effector to perform target actions on different sample objects respectively, and when the robot releases the current sample object after completing the target action, the pose and speed are random. This third sample dataset is used to evaluate the ability of the trajectory prediction model to capture the physical characteristics of the object.
[0183] In the embodiments of the present application, sample data is collected under different conditions to create a sample dataset, increasing the data volume of the sample dataset. By obtaining sample data from the above sample dataset and training a trajectory prediction model based on the sample data, the trajectory prediction model is enabled to accurately extract action features and predict the trajectory data of the object, improving the prediction accuracy of the trajectory prediction model.
[0184] In a possible implementation manner, to reduce the side effects of noise, the control mode of the robot is set to the speed control mode, so that the movement of the robot is smoother at both the position and speed levels. The control mode of the end effector is set to the current-based control mode, so that the robot can autonomously determine the width of the end effector to tightly grasp different objects through the end effector.
[0185] In a possible implementation manner, for each sample dataset, the sample data in the sample dataset can be divided into three subsets. One subset is used as the training set, and the sample data in the training set is used to train the trajectory prediction model. Another subset is used as the validation set, and the sample data in the validation set is used to validate the trajectory prediction model. Another subset is used as the test set, and the sample data in the test set is used to evaluate the trained trajectory prediction model. Among them, the test set does not participate in the training process. The trajectory prediction model is trained based on the sample data in the training set, and the trajectory prediction model is validated through the sample data in the validation set, so as to continue training the trajectory prediction model according to the validation results. After the training of the trajectory prediction model is completed, the performance of the trajectory prediction model is evaluated based on the sample data in the test set.
[0186] In addition, to improve accuracy, the cross-validation method can be adopted. That is, for multiple subsets of the same sample dataset, one subset is used as the validation set, another subset is used as the training set to train the trajectory prediction model, and the last subset is used as the test set to obtain the evaluation index of the trajectory prediction model. Next, the validation set, training set, and test set are replaced, and finally, the average value of the multiple evaluation indexes is used as the final evaluation index of the trajectory prediction model.
[0187] In a possible implementation, a fourth sample data set can also be obtained. The sample data in the fourth sample data set is collected under a fourth condition, and the fourth condition is that the robot uses a random end effector to perform a target action on a random sample object respectively, and the pose and speed of the robot when releasing the current sample object are random after the target action is completed. The fourth sample data set is the test set of the trajectory prediction model and does not participate in the training process of the trajectory prediction model. Based on the fourth sample data set, the performance of the trajectory prediction model can be evaluated.
[0188] For example, collect 11 objects and perform 400 random throwing actions, and each throwing action is generated by a random release position, speed and end effector. The geometric shapes of the physical objects are different, such as cylinders, cubes and polyhedrons, etc., and the materials include rubber, aluminum, wood and resin, etc.
[0189] In another possible implementation, the evaluation metrics of the trajectory prediction model include: ADE (Average Displacement Error), that is, the mean square error between the waypoints on the predicted trajectory and the corresponding ground truth values, and the overall prediction accuracy of the trajectory prediction model is evaluated by using the average error distribution along the trajectory.
[0190] In another possible implementation, the evaluation metrics of the trajectory prediction model include: FDE (Final Displacement Error): FDE refers to the average of the absolute displacements between the waypoints on the predicted trajectory and their ground truth values. Compared with ADE, FDE reflects how the prediction error develops along the predicted trajectory.
[0191] The embodiments of this application evaluate the following four trajectory prediction models:
[0192] Trajectory prediction model 1: Only uses the time series of the robot's pose parameters and speed to predict the trajectory data of the object, without considering the robot's force / torque.
[0193] Trajectory prediction model 2: Only uses the time series of the robot's force / torque to predict the trajectory data of the object, without considering the robot's pose parameters and speed.
[0194] Trajectory prediction model 3: Uses the time series of the robot's pose parameters, speed and force / torque to predict the trajectory data of the object, but does not include an association coding block, so the above three time series are encoded separately and not encoded through the association coding block.
[0195] Trajectory prediction model 4: Using the time series of the pose parameters, speed, and force / torque of the robot to predict the trajectory data of the object, and including an association encoding block, which not only encodes the above three time series respectively, but also encodes through the association encoding block, so as to fuse each type of action feature with the associated action feature.
[0196] Based on the above first sample dataset, second sample dataset, and third sample dataset, the above 4 trajectory prediction models are trained and evaluated respectively. The evaluation results show that the prediction accuracy of trajectory prediction model 2 is higher than that of trajectory prediction model 1, which indicates that the force / torque of the robot can not only implicitly reflect the action features of the robot performing the target action, but also contains rich information about the physical characteristics of the object. However, the convergence speed of trajectory prediction model 1 is faster than that of trajectory prediction model 2. The prediction accuracies of trajectory prediction model 3 and trajectory prediction model 4 are higher than those of trajectory prediction model 1 and trajectory prediction model 2. Therefore, trajectory prediction model 3 and trajectory prediction model 4 have the characteristics of fast convergence speed and accurate prediction.
[0197] In addition, by conducting experiments on the method provided in the embodiments of the present application in various real environments, such as a robot throwing a bottle, a robot juggling, a robot throwing random objects into a small container, etc., the experimental results show that the method provided in the embodiments of the present application can predict trajectories at the second level within dozens of milliseconds, and the prediction error of the pose parameters of the road section is reduced to millimeters / degree.
[0198] In addition, the prediction duration of the trajectory prediction model is also evaluated, which is crucial for object-centered application scenarios. For example, in the scenario of a robot juggling, if it takes too much time to predict the trajectory, the robot will have no reaction time, resulting in the failure of juggling. By evaluating the prediction duration of the trajectory prediction model, it is determined that for trajectories with durations of 0.1 second, 0.2 second, and 0.5 second respectively, the required prediction durations are 12.2 milliseconds, 14.5 milliseconds, and 20.1 milliseconds. From the evaluation results, it can be seen that the method provided in the embodiments of the present application has high computational efficiency in predicting trajectories.
[0199] It is a schematic structural diagram of a device for determining trajectory data provided by an embodiment of the present application. See This device includes:
[0200] An action data acquisition module 1801, configured to acquire the action data of the robot during the process of the robot performing a target action on an object, and the action data represents the state of the robot itself during the process of the robot performing the target action on the object;
[0201] A prediction module 1802, configured to process the action data through a trajectory prediction model to obtain the trajectory data of the object;
[0202] The trajectory data represents the trajectory of the object after the robot completes the target action on the object.
[0203] The device provided by the embodiment of the present application can obtain the action data of the robot during the process of the robot performing the target action on the object, and through processing by the trajectory prediction model, predict the trajectory data of the object. After the robot completes the target action on the object, the trajectory of the object can be predicted immediately, without waiting for a period of time after the robot completes the target action on the object and then predicting by using a motion capture sensor, which saves time and improves efficiency. Moreover, the embodiment of the present application summarizes and learns from human daily actions, and determines that the state of the robot during the process of the robot performing the target action on the object determines the trajectory of the object. Therefore, a trajectory prediction model with the action data during the process of the robot performing the target action on the object as the input and the trajectory data of the object as the output is used for prediction, ensuring the accuracy of the predicted trajectory data.
[0204] In a possible implementation manner, the trajectory prediction model includes an encoder and a decoder. Refer to , the prediction module 1802 includes:
[0205] An encoding unit 18021, configured to encode the action data through the encoder to obtain action features;
[0206] A decoding unit 18022, configured to decode the action features through the decoder to obtain the trajectory data of the object.
[0207] In a possible implementation manner, the action data includes at least two types, the encoder includes at least two encoding blocks and a fusion layer, and the encoding unit 18021 is configured to:
[0208] Encode the action data of at least two types corresponding to at least two encoding blocks respectively through the at least two encoding blocks to obtain at least two action features;
[0209] Fuse the at least two action features through the fusion layer to obtain the fused action features.
[0210] In a possible implementation manner, the encoder further includes an associated encoding block, and the encoding unit 18021 is configured to encode the action data of at least two types through the associated encoding block to obtain associated action features, where the associated action features represent the correlation between at least two action features; and fuse the at least two action features and the associated action features through the fusion layer to obtain the fused action features.
[0211] In a possible implementation, the associated coding block includes at least two coding layers. The coding unit 18021 is configured to encode at least two types of action data through the first coding layer in the associated coding block to obtain the first associated action feature; and encode the first associated action feature through the second coding layer in the associated coding block to obtain the second associated action feature, until the associated action feature is obtained through the last coding layer in the associated coding block.
[0212] In a possible implementation, the trajectory prediction model is a long short-term memory network (LSTM) model. The coding unit 18021 is configured to encode the action data through an encoder each time action data at a first time point is obtained, to obtain the action feature at the first time point, where the first time point is a time point during the process in which the robot performs a target action on an object.
[0213] The decoding unit 18022 is configured to combine at least two action features at the first time points that have been obtained when the robot finishes performing the target action on the object; and decode the combined action features through a decoder to obtain trajectory data.
[0214] In a possible implementation, the action data of the robot includes the action data of the robot at at least two first time points, and the trajectory data of the object includes the pose parameters of the object at at least two second time points, where the pose parameters include position and attitude; the first time point is a time point during the process in which the robot performs a target action on the object, and the second time point is a time point after the robot finishes performing the target action on the object.
[0215] See , the apparatus further includes:
[0216] A training module 1803, configured to obtain sample data from a sample data set and train the trajectory prediction model based on the sample data.
[0217] The sample data includes sample action data and a sample trajectory. The sample action data includes the action data of the robot at at least two first sample time points, and the sample trajectory includes the pose parameters of a sample object at at least two second sample time points; the first sample time point is a time point during the process in which the robot performs a target action on the sample object, and the second sample time point is a time point after the robot finishes performing the target action on the sample object.
[0218] In a possible implementation, the sample data set includes at least one of the following:
[0219] The first sample dataset, where the sample data in the first sample dataset is collected under the first condition, and the first condition is that the robot uses the same end effector to perform the target action on the same sample object, and the pose and speed of the end effector when releasing the sample object after completing the target action are not exactly the same;
[0220] The second sample dataset, where the sample data in the second sample dataset is collected under the second condition, and the second condition is that the robot uses a random end effector to perform the target action on the same sample object, and the pose and speed of the robot when releasing the sample object after completing the target action are random;
[0221] The third sample dataset, where the sample data in the third sample dataset is collected under the third condition, and the third condition is that the robot uses a random end effector to perform the target action on different sample objects respectively, and the pose and speed of the robot when releasing the current sample object after completing the target action are random.
[0222] In a possible implementation, the robot includes an end effector. Refer to , and the motion data acquisition module 1801 includes at least one of the following:
[0223] The first acquisition unit 18011 is used to acquire the pose parameters of the end effector at at least two first time points, and the pose parameters include position and orientation;
[0224] The second acquisition unit 18012 is used to acquire the speed of the end effector at at least two first time points;
[0225] A force / torque sensor is configured on the joint corresponding to the end effector. The third acquisition unit 18013 is used to acquire the force and torque of the end effector at at least two first time points through the force / torque sensor;
[0226] Wherein, the first time point is a time point during the process of the robot performing the target action on the object.
[0227] In a possible implementation, the robot includes an end effector; Refer to , and the device further includes:
[0228] The coordinate system creation module 1804 is used to create a target attached coordinate system. The target attached coordinate system takes the centroid of the object grasped by the end effector as the origin, and the directions of each coordinate axis in the target attached coordinate system are determined based on the directions of each coordinate axis of the end effector. The motion data represents the state of the robot in the target attached coordinate system during the process of the robot performing the target action on the object.
[0229] It should be noted that: The device for determining trajectory data provided in the above embodiments is only illustrated by dividing the above functional modules. In practical applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the robot controller is divided into different functional modules to complete all or part of the functions described above. In addition, the device for determining trajectory data provided in the above embodiments and the method embodiments for determining trajectory data belong to the same concept. For the specific implementation process, please refer to the method embodiments and will not be elaborated here.
[0230] An embodiment of the present application also provides a robot controller, which includes a processor and a memory. At least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the operations performed by the method for determining trajectory data in the above embodiments.
[0231] The structural schematic diagram of a robot controller 2000 provided by an exemplary embodiment of the present application is shown.
[0232] The robot controller 2000 includes: a processor 2001 and a memory 2002.
[0233] The processor 2001 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 2001 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field Programmable Gate Array), and PLA (Programmable Logic Array). The processor 2001 may also include a main processor and a coprocessor. The main processor is a processor used to process data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor used to process data in the standby state. In a possible implementation manner, the processor 2001 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 2001 may further include an AI (Artificial Intelligence) processor, and the AI processor is used to process computational operations related to machine learning.
[0234] The memory 2002 may include one or more computer-readable storage media, which may be non-transitory. The memory 2002 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In a possible implementation, the non-transitory computer-readable storage media in the memory 2002 is used to store at least one computer program, and the at least one computer program is used to be possessed by the processor 2001 to implement the method for determining trajectory data provided in the method embodiments of this application.
[0235] In a possible implementation, the robot controller 2000 may further optionally include: a peripheral device interface 2003 and at least one peripheral device. The processor 2001, the memory 2002, and the peripheral device interface 2003 may be connected through a bus or signal lines. Each peripheral device may be connected to the peripheral device interface 2003 through a bus, signal lines, or circuit board. Optionally, the peripheral device includes at least one of a radio frequency circuit 2004, a display screen 2005, a camera component 2006, and a power supply 2007.
[0236] The peripheral device interface 2003 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 2001 and the memory 2002. In a possible implementation, the processor 2001, the memory 2002, and the peripheral device interface 2003 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 2001, the memory 2002, and the peripheral device interface 2003 can be implemented on a separate chip or circuit board, and this embodiment does not limit this.
[0237] The radio frequency circuit 2004 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 2004 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 2004 converts electrical signals into electromagnetic signals for transmission, or converts the received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 2004 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and so on. The radio frequency circuit 2004 can communicate with other devices through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: metropolitan area network, generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area network, and / or WiFi (Wireless Fidelity) network. In a possible implementation, the radio frequency circuit 2004 may further include a circuit related to NFC (Near Field Communication), which is not limited in this application.
[0238] The display screen 2005 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 2005 is a touch display screen, the display screen 2005 also has the ability to collect touch signals on or above the surface of the display screen 2005. The touch signal can be input as a control signal to the processor 2001 for processing. At this time, the display screen 2005 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In a possible implementation, there can be one display screen 2005, which is set on the front panel of the robot controller 2000; in other embodiments, there can be at least two display screens 2005, which are respectively set on different surfaces of the robot controller 2000 or in a folded design; in other embodiments, the display screen 2005 can be a flexible display screen, which is set on the curved surface or the folding surface of the robot controller 2000. Even, the display screen 2005 can also be set into an irregular non-rectangular shape, that is, a special-shaped screen. The display screen 2005 can be prepared using materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).
[0239] The camera component 2006 is used to collect images or videos. Optionally, the camera component 2006 includes a front camera and a rear camera. The front camera is disposed on the front panel of the robot controller 2000, and the rear camera is disposed on the rear of the robot controller 2000. In a possible implementation, there are at least two rear cameras, which can be any one of a main camera, a depth camera, a wide-angle camera, and a telephoto camera, so as to implement functions such as background blurring by fusing the main camera and the depth camera, panoramic shooting by fusing the main camera and the wide-angle camera, and VR (Virtual Reality) shooting function or other fused shooting functions. In a possible implementation, the camera component 2006 may further include a flash. The flash can be a single-color temperature flash or a two-color temperature flash. The two-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.
[0240] The power supply 2007 is used to supply power to each component in the robot controller 2000. The power supply 2007 can be alternating current, direct current, a disposable battery, or a rechargeable battery. When the power supply 2007 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.
[0241] Those skilled in the art can understand that the structure shown in does not constitute a limitation on the robot controller 2000, and it may include more or fewer components than shown in the figure, or combine some components, or adopt different component arrangements.
[0242] This application embodiment also provides a computer-readable storage medium, in which at least one computer program is stored, and the at least one computer program is loaded and executed by a processor to implement the operations performed by the method for determining trajectory data in the above embodiment.
[0243] This application embodiment also provides a computer program product, including a computer program, and the computer program is loaded and executed by a processor to implement the operations performed by the method for determining trajectory data in the above embodiment.
[0244] Those of ordinary skill in the art can understand that all or part of the steps of implementing the above embodiment can be completed by hardware, or can be completed by a program instructing relevant hardware. The program can be stored in a computer-readable storage medium, and the above-mentioned storage medium can be a read-only memory, a magnetic disk, or an optical disc, etc.
[0245] The above are only optional embodiments of the embodiments of the present application, and are not intended to limit the embodiments of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the embodiments of the present application shall be included within the protection scope of the present application.
Claims
1. A method for determining trajectory data, characterized in that: The method comprises: Acquiring action data of the robot during the process of the robot performing the target action on the object, wherein the action data represents a state of the robot during the process of the robot performing the target action on the object; Processing the motion data through a trajectory prediction model to obtain trajectory data of the object; The trajectory data represents the trajectory of the object after the robot completes the target action on the object.
2. The method according to claim 1, characterized in that The trajectory prediction model includes an encoder and a decoder. The motion data is processed by the trajectory prediction model to obtain the trajectory data of the object, including: Encoding the motion data by the encoder to obtain motion features; The motion features are decoded by the decoder to obtain the trajectory data of the object.
3. The method according to claim 2, characterized in that The motion data includes at least two types, the encoder includes at least two encoding blocks and a fusion layer, and the encoder encodes the motion data to obtain motion features, including: Using at least two coding blocks, respectively encode the action data of the types corresponding to the at least two coding blocks to obtain at least two action features; At least two of the motion features are fused through the fusion layer to obtain the fused motion feature.
4. The method according to claim 3, characterized in that The encoder further includes an associated coding block, and the fusion layer is used to fuse at least two of the motion features to obtain the fused motion features, including: encoding at least two types of the motion data by the association coding block to obtain an associated motion feature, wherein the associated motion feature represents the association between the at least two motion features; At least two of the action features and the associated action features are fused through the fusion layer to obtain the fused action feature.
5. The method according to claim 4, characterized in that The associated coding block includes at least two coding layers, and encoding at least two types of motion data through the associated coding block to obtain associated motion features includes: encoding at least two types of the action data through a first coding layer in the associated coding block to obtain a first associated action feature; The first associated action feature is encoded through the second coding layer in the associated coding block to obtain a second associated action feature, until the associated action feature is obtained through the last coding layer in the associated coding block.
6. The method according to claim 2, characterized in that The trajectory prediction model is a long short-term memory network (LSTM) model. The encoder encodes the action data to obtain action features, including: Each time action data at a first time point is acquired, encoding the action data by the encoder to obtain an action feature at the first time point, wherein the first time point is a time point when the robot performs the target action on the object; Decoding the motion feature by the decoder to obtain the trajectory data of the object includes: When the robot completes the target action on the object, combining the obtained action features of at least two first time points; The combined motion features are decoded by the decoder to obtain the trajectory data.
7. The method according to claim 1, characterized in that The robot's motion data includes motion data of the robot at at least two first time points, and the object's trajectory data includes posture parameters of the object at at least two second time points, the posture parameters including position and posture; wherein the first time point is a time point in the process of the robot performing the target action on the object, and the second time point is a time point after the robot completes the target action on the object; Training the trajectory prediction model includes: Get sample data from the sample dataset; Training the trajectory prediction model based on the sample data; The sample data includes sample action data and sample trajectory, the sample action data includes the action data of the robot at at least two first sample time points, and the sample trajectory includes the posture parameters of the sample object at at least two second sample time points; wherein, the first sample time point is the time point in the process of the robot performing the target action on the sample object, and the second sample time point is the time point after the robot completes the target action on the sample object.
8. The method according to claim 7, characterized in that The sample data set includes at least one of the following: a first sample data set, wherein sample data in the first sample data set is collected under a first condition, wherein the first condition is that the robot uses the same end effector to perform the target action on the same sample object, and the posture and speed of the end effector when releasing the sample object after completing the target action are not completely the same; a second sample data set, wherein sample data in the second sample data set is collected under a second condition, wherein the second condition is that the robot uses a random end effector to perform the target action on the same sample object, and after completing the target action, the robot releases the sample object at a random position and speed; The third sample data set, the sample data in the third sample data set are collected under a third condition, and the third condition is: the robot uses a random end effector to perform the target action on different sample objects respectively, and after the target action is completed, the posture and speed of the robot when releasing the current sample object are random.
9. The method according to any one of claims 1 to 8, characterized in that The robot includes an end effector, and obtaining the motion data of the robot during the process of the robot performing a target motion on the object includes at least one of the following: Acquire posture parameters of the end effector at at least two first time points, the posture parameters including position and posture; Acquiring the velocity of the end effector at at least two first time points; A force / torque sensor is configured on the joint corresponding to the end effector, and the force and torque of the end effector at at least two first time points are obtained through the force / torque sensor; The first time point is a time point in the process of the robot performing the target action on the object.
10. The method according to any one of claims 1 to 8, characterized in that The robot includes an end effector; and the method further includes: A target body coordinate system is created, wherein the target body coordinate system takes the center of mass of the object grasped by the end effector as its origin, and the directions of the respective coordinate axes in the target body coordinate system are determined based on the directions of the respective coordinate axes of the end effector, and the action data represents the state of the robot in the target body coordinate system during the process of the robot performing the target action on the object.
11. A device for determining trajectory data, characterized in that: The device comprises: an action data acquisition module, configured to acquire action data of the robot during the process of the robot performing a target action on an object, wherein the action data represents a state of the robot itself during the process of the robot performing the target action on the object; A prediction module, configured to process the motion data using a trajectory prediction model to obtain trajectory data of the object; The trajectory data represents the trajectory of the object after the robot completes the target action on the object.
12. A robot controller, characterized in that: The robot controller includes a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the operations performed by the method for determining trajectory data according to any one of claims 1 to 10.
13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the operations performed by the method for determining trajectory data according to any one of claims 1 to 10.
14. A computer program product comprising a computer program, characterized in that The computer program is loaded and executed by a processor to implement the operations performed by the method for determining trajectory data according to any one of claims 1 to 10.
Citation Information
Cited By
Aircraft recovery dynamic scheduling method based on long short-term memory network
CN120725394A
Positioning method and system based on visual perception
CN121074123A
