Action prediction method based on key points and imitation learning, terminal and storage medium

Abstract

The invention discloses an action prediction method based on key points and imitation learning, a terminal and a storage medium, and relates to the technical field of intelligent equipment. According to the method, the thought of a world model is combined with imitation learning, scene representation more conforming to the environment context is learned through the environment dynamics prediction model, and the motion trail of the key point is predicted through the trained trajectory prediction model so as to assist the motion head module in motion decision making. The key point motion trail is used as auxiliary information, and the decision accuracy of the intelligent agent can be effectively improved. Firstly, the key points are not affected by environmental changes such as illumination, even if an object is partially shielded, the rest of key point tracks can still provide the same information, and therefore good robustness is achieved. And secondly, the motion trail of the key point and the motion trail of the intelligent mechanical device in the three-dimensional space have a certain mapping relation, so that the neural network can better fit the moving action of the intelligent mechanical device.

Description

Technical Field

[0001] The present invention relates to the field of embodied intelligence technology, and in particular to an action prediction method, a terminal and a storage medium based on key points and imitation learning. Background Art

[0002] Imitation learning aims to learn complex tasks by imitating the behavior of humans or other intelligent agents, and is used to understand and build intelligent systems with autonomous behavior capabilities. Multi-task imitation learning is a method that trains intelligent agents to master multiple skills at the same time by imitating expert demonstrations of multiple different tasks. Multi-task imitation learning requires the model to be able to share knowledge and capabilities between different tasks, improve the generalization ability and efficiency of the model, and is suitable for complex scenarios such as robot control and natural language processing.

[0003] For robotic arm tasks that use visual information as input information, due to the complexity of visual perception and the diversity of objects, when the robotic arm operates in a complex and diverse environment, visual perception faces many challenges, which in turn affects the grasping accuracy. For example, objects are obscured, making it difficult for the visual system to obtain complete object information; changes in light make it difficult for the visual system to accurately identify object boundaries and textures. Scene understanding and multi-tasking also bring challenges to model algorithms. When the robotic arm faces a complex working environment, it must not only identify individual objects, but also understand the context of the entire scene.

[0004] The multi-task imitation learning decision model currently used for robotic arm tasks has a poor understanding of the environmental scene, which leads to low decision accuracy.

[0005] Therefore, the existing technology still needs to be improved and developed. Summary of the invention

[0006] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, an action prediction method, terminal and storage medium based on key points and imitation learning are provided, aiming to solve the problem that the multi-task imitation learning decision model applied to robotic arm tasks in the prior art has poor understanding of the environmental scene, which leads to low decision accuracy.

[0007] The technical solution adopted by the present invention to solve the problem is as follows: In a first aspect, an embodiment of the present invention provides an action prediction method based on key points and imitation learning, the method comprising: Obtain mission information and observation data of intelligent mechanical devices; Inputting the task information and the observation data into a trained multi-task imitation learning decision model to obtain a predicted execution action of the intelligent mechanical device; Wherein, the multi-task imitation learning decision model includes: An encoding module, used for encoding the task information and the observation data, and generating a plurality of semantic units according to the encoded data; A world model, used to generate a predicted state sequence according to each of the semantic units through a decoder and an environmental dynamics prediction model; A trajectory prediction model, used to generate a predicted key point trajectory according to the predicted state sequence; An attention model, used to generate a latent vector according to each of the semantic units; wherein the latent vector is used to reflect the overall information of the environment and the task; The action head module is used to generate a predicted execution action of the intelligent mechanical device according to the predicted key point trajectory and the latent vector.

[0008] In one implementation, the observation data includes image observation data from a plurality of camera perspectives and state observation data of the intelligent mechanical device.

[0009] In one implementation, the task information and the observation data are encoded, and a plurality of semantic units are generated according to the encoded data, including: Acquire the semantic unit corresponding to the task information through a language encoder; Obtaining semantic units corresponding to the image observation data through a visual encoder and a SimNorm layer; wherein the SimNorm layer is used to divide the encoding vector into a number of segments of a preset length, perform a softmax operation on each segment separately, and reassemble them; The semantic unit corresponding to the state observation data is obtained through the state encoder.

[0010] In one embodiment, generating a predicted state sequence according to each of the semantic units through a decoder and an environmental dynamics prediction model includes: Based on all the semantic units, generating reconstructed observation data through the decoder; Based on the reconstructed observation data, the predicted states of several future time steps are cyclically generated through the environmental dynamics prediction model to obtain the predicted state sequence.

[0011] In one embodiment, the training method of the multi-task imitation learning decision model includes: The expert trajectories are collected through the expert strategy module, and an expert data set is established according to the expert trajectories; wherein each expert data includes historical observation data, corresponding expert strategy actions and corresponding historical task information; The encoding module, the world model, the attention model and the action head module are used as a strategy model; The strategy model and the trajectory prediction model are trained using the expert data set, and model parameters of the strategy model and the trajectory prediction model are updated respectively.

[0012] In one embodiment, the training method of the policy model includes: Calculating the loss value of the world model and the loss value of the action head module through the expert data set; wherein the loss value of the world model is determined based on the reconstruction observation loss value of the decoder and the prediction loss value of the environment dynamics prediction model; Determining the loss value of the strategy model according to the loss values ​​corresponding to the world model and the action head module respectively; The model parameters of the strategy model are updated according to the loss value of the strategy model.

[0013] In one embodiment, the training method of the trajectory prediction model includes: Determine whether the current number of training steps meets the training interval of the trajectory prediction model; If the conditions are met, the predicted state sequence output by the environmental dynamics prediction model is obtained, and a training prediction key point trajectory is generated according to the predicted state sequence by the trajectory prediction model; Based on the historical observation data of the corresponding number of frames in the expert dataset, a reference prediction key point trajectory is generated through a point tracking model; According to the training predicted key point trajectory and the reference predicted key point trajectory, a loss value of the trajectory prediction model is calculated; and according to the loss value of the trajectory prediction model, a model parameter of the trajectory prediction model is updated.

[0014] In one implementation, the historical observation data corresponding to the number of frames in the expert dataset includes a plurality of consecutive frames of real image observation data; based on the historical observation data corresponding to the number of frames in the expert dataset, generating a reference predicted key point trajectory through a point tracking model includes: A plurality of feature points are obtained by sampling the real image observation data of the first frame; Based on the real image observation data of each frame and the position of each feature point, generating trajectory change data of each feature point in each frame through a point tracking model; Calculating the change amplitude of each feature point according to each trajectory change data; A number of target feature points are selected according to the magnitude of each change, and a reference prediction key point trajectory is generated according to the trajectory of each target feature point.

[0015] In a second aspect, an embodiment of the present invention further provides a terminal comprising a memory and at least one processor; the memory stores a program; the program includes instructions for executing any of the action prediction methods based on key points and imitation learning as described above; and the processor is used to execute the program.

[0016] In a third aspect, an embodiment of the present invention further provides a computer-readable storage medium on which a plurality of instructions are stored, wherein the instructions are suitable for being loaded and executed by a processor to implement the steps of any of the above-mentioned action prediction methods based on key points and imitation learning.

[0017] Beneficial effects of the present invention: The embodiments of the present invention combine the idea of ​​the world model with imitation learning, learn a scene representation that is more in line with the environmental context through the environmental dynamics prediction model, and predict the motion trajectory of the key points through the trained trajectory prediction model to assist the action head module in making action decisions. The present invention uses the motion trajectory of key points as auxiliary information to effectively improve the decision-making accuracy of the intelligent agent. First of all, the key points will not be affected by environmental changes such as lighting. Even if the object is partially blocked, the trajectory of the remaining key points can still provide the same information, so it has good robustness. Secondly, there is a definite mapping relationship between the motion trajectory of the key points and the motion trajectory of the intelligent mechanical device in three-dimensional space, so the neural network can better fit the moving movements of the intelligent mechanical device. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0019] Figure 1 It is a flowchart of an action prediction method based on key points and imitation learning provided by an embodiment of the present invention.

[0020] Figure 2 It is a schematic diagram of the overall algorithm architecture of the action prediction method based on key points and imitation learning provided by an embodiment of the present invention.

[0021] Figure 3 It is a logical schematic diagram of key point trajectory prediction provided by an embodiment of the present invention.

[0022] Figure 4 It is a principle block diagram of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0023] The present invention discloses an action prediction method, terminal and storage medium based on key points and imitation learning. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0024] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0025] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as herein.

[0026] In view of the above-mentioned defects of the prior art, the present invention provides an action prediction method based on key points and imitation learning, wherein the method obtains task information and observation data of an intelligent mechanical device; inputs the task information and the observation data into a trained multi-task imitation learning decision model to obtain the predicted execution action of the intelligent mechanical device; wherein the multi-task imitation learning decision model includes: an encoding module for encoding the task information and the observation data, and generating a plurality of semantic units according to the encoded data; a world model for generating a predicted state sequence according to each of the semantic units through a decoder and an environmental dynamics prediction model; a trajectory prediction model for generating a predicted key point trajectory according to the predicted state sequence; an attention model for generating a latent vector according to each of the semantic units; wherein the latent vector is used to reflect the overall information of the environment and the task; and an action head module for generating the predicted execution action of the intelligent mechanical device according to the predicted key point trajectory and the latent vector. The present invention combines the idea of ​​the world model with imitation learning, learns a scene representation that is more in line with the environmental context through the environmental dynamics prediction model, and predicts the key point motion trajectory through the trained trajectory prediction model to assist the action head module in making action decisions. The present invention uses the motion trajectory of key points as auxiliary information, which can effectively improve the decision-making accuracy of the intelligent agent. First, the key points will not be affected by environmental changes such as lighting. Even if the object is partially blocked, the remaining key point trajectory can still provide the same information, so it has good robustness. Secondly, there is a definite mapping relationship between the motion trajectory of the key points and the motion trajectory of the intelligent mechanical device in three-dimensional space, so the neural network can better fit the movement of the intelligent mechanical device.

[0027] like Figure 1 As shown, the method specifically comprises the following steps: Step S100: Acquire task information and observation data of the intelligent mechanical device.

[0028] Step S200: input the task information and the observation data into a trained multi-task imitation learning decision model to obtain a predicted execution action of the intelligent mechanical device.

[0029] Specifically, task information can also be called language input or task description, which is used to reflect the current task of the intelligent mechanical device, such as task type and task goal. The intelligent mechanical device can be a robotic arm, and the observation data of the intelligent mechanical device includes image observation data from several camera perspectives and its own state observation data. Task information and observation data are the basic data for realizing the autonomous decision-making actions of the intelligent mechanical device. The task information and observation data are input into a pre-built and trained multi-task imitation learning decision model, and the multi-task imitation learning decision model automatically decides the next action of the intelligent mechanical device.

[0030] Furthermore, the multi-task imitation learning decision model includes: An encoding module, used for encoding the task information and the observation data, and generating a plurality of semantic units according to the encoded data; A world model, used to generate a predicted state sequence according to each of the semantic units through a decoder and an environmental dynamics prediction model; A trajectory prediction model, used to generate a predicted key point trajectory according to the predicted state sequence; An attention model, used to generate a latent vector according to each of the semantic units; wherein the latent vector is used to reflect the overall information of the environment and the task; The action head module is used to generate a predicted execution action of the intelligent mechanical device according to the predicted key point trajectory and the latent vector.

[0031] Specifically, Figure 2 As shown in the figure, the multi-task imitation learning decision model mainly consists of five functional modules: encoding module, world model (i.e. Figure 2 The dynamic model in the environment), trajectory prediction model, attention model and action head module. Among them, the role of the encoding module is to encode both the observation data and the task information into vectors in the latent space, thereby obtaining multiple semantic units (Tokens). The world model includes a decoder and an environment dynamics model, which is used to imagine the future state trajectory at each time step to obtain a predicted state sequence. The trajectory prediction model has been pre-trained with supervised learning, and can accurately map the predicted state sequence to the motion trajectory of the key points, thereby obtaining the predicted key point trajectory, and using the predicted key point trajectory as auxiliary information for subsequent action decisions. The attention model (Transformer) is used to generate a latent vector that can reflect the overall information of the environment and the task based on all semantic units, and input it into the action head module for action decision. The action head module analyzes the latent vector output by the attention model and the predicted key point trajectory, and decides the action to be executed by the intelligent mechanical device, that is, to obtain the predicted execution action, and sends it to the intelligent mechanical device for execution. The multi-task imitation learning decision model trained in this embodiment can be deployed to the corresponding scenario, providing observation data at each time step, and the model outputs the action to be executed by the intelligent mechanical device.

[0032] In one implementation, the task information and the observation data are encoded, and a plurality of semantic units are generated according to the encoded data, including: Acquire the semantic unit corresponding to the task information through a language encoder; Obtaining semantic units corresponding to the image observation data through a visual encoder and a SimNorm layer; wherein the SimNorm layer is used to divide the encoding vector into a number of segments of a preset length, perform a softmax operation on each segment separately, and reassemble them; The semantic unit corresponding to the state observation data is obtained through the state encoder.

[0033] Specifically, this embodiment needs to map the task information and observation data into semantic units (Tokens) that can be used by the Transformer. The image observation data is passed through the visual encoder and the SimNorm layer to generate visual Tokens, the state observation data (or called state vector) is passed through the state encoder to generate state Tokens, and the task information is passed through the language encoder to generate language Tokens. The three will be input into the attention model (Transformer) together later. Among them, the language encoder can use a pre-trained lightweight sentence embedding model to encode different task descriptions into fixed-dimensional embedding vectors.

[0034] For example, the observation data at each step , , Represents the image observations from different camera perspectives and the body state vector observations of the robot. For each task, the corresponding task description is encoded by the pre-trained model, denoted as Lang. The input data for each time step is The language encoder and visual encoder are mapped into several vector codes respectively. The vector codes obtained by the visual encoder are then passed through the SimNorm layer, and together with other codes, several Tokens that can be input into the Transformer are obtained.

[0035] In one implementation, generating a predicted state sequence according to each of the semantic units through a decoder and an environmental dynamics prediction model includes: Based on all the semantic units, generating reconstructed observation data through the decoder; Based on the reconstructed observation data, the predicted states of several future time steps are cyclically generated through the environmental dynamics prediction model to obtain the predicted state sequence.

[0036] Specifically, the decoder and the environmental dynamics prediction model in this embodiment have been pre-trained and have learned the complex mapping relationship between input and output. Therefore, in practical applications, the decoder can learn to reconstruct observations based on the semantic units (Tokens) generated above and output relatively accurate reconstructed observation data. Figure 3 As shown, the environmental dynamics prediction model learns the current state based on the reconstructed observation data output by the decoder , and imagine the future state backwards , etc. (s represents state, t represents time step), thus obtaining the predicted state sequence.

[0037] In one implementation, the training method of the multi-task imitation learning decision model includes: The expert trajectories are collected through the expert strategy module, and an expert data set is established according to the expert trajectories; wherein each expert data includes historical observation data, corresponding expert strategy actions and corresponding historical task information; The encoding module, the world model, the attention model and the action head module are used as a strategy model; The strategy model and the trajectory prediction model are trained using the expert data set, and model parameters of the strategy model and the trajectory prediction model are updated respectively.

[0038] Specifically, this embodiment will train the multi-task imitation learning decision model based on expert data and through the behavior cloning algorithm. Before training, the expert trajectory of the intelligent mechanical device (such as a robotic arm) in completing various tasks is first collected through expert strategies, and the expert trajectory is recorded and saved in the form of observation action pairs. Each observation action pair is an expert data, which contains historical observation data, expert strategy actions and historical task information with corresponding relationships, wherein the historical observation data and historical task information can be used as input data for subsequent model training, and the expert strategy actions can be used as the real label for subsequent model training to evaluate model performance. During training, the multi-task imitation learning decision model is trained using the expert data set generated above, so that the multi-task imitation learning decision model can learn multiple tasks by imitating expert behavior. The encoding module, the world model including the decoder and the environmental dynamics prediction model, the attention model and the action head module are used as a strategy model. When parameter optimization is required, the strategy model and the trajectory prediction model are optimized separately.

[0039] For example, the training process of the multi-task imitation learning decision model includes: initializing the parameters of the Transformer backbone and action head module, the parameters of the world model, and the parameters of the trajectory prediction model. The world model includes the environment dynamics prediction model and the decoder that learns to reconstruct observations. The parameters of the policy model are , the parameters of the trajectory prediction model are . Read the collected expert trajectories into the memory Buffer and set the hyperparameters. The hyperparameters include the batch size B and the setting parameters of the SimNorm operation (performed in the aforementioned SimNorm layer) after the image encoder encodes the observation. The SimNorm operation divides the encoded vector into L segments of length d, performs a softmax operation on each segment separately, and finally splices them back together. Therefore, its setting parameters include: the division length d, the length H of the strategy training imagined trajectory (that is, the time step of backward prediction when training the environmental dynamics prediction model), and the number of key points used. , use the key point trajectory time length T (i.e. key point trajectory length), where T is less than or equal to H. It should be noted that the SimNorm operation is only for the visual encoder, and the language encoder and state encoder are not SimNorm operated. The multi-task imitation learning decision model is trained using the expert dataset. If the current training step number has reached the maximum step number , the training ends; otherwise, randomly take out group B data from the Buffer and continue to train and update the current multi-task imitation learning decision model.

[0040] In one implementation, the training method of the policy model includes: Calculating the loss value of the world model and the loss value of the action head module through the expert data set; wherein the loss value of the world model is determined based on the reconstruction observation loss value of the decoder and the prediction loss value of the environment dynamics prediction model; Determining the loss value of the strategy model according to the loss values ​​corresponding to the world model and the action head module respectively; The model parameters of the strategy model are updated according to the loss value of the strategy model.

[0041] Specifically, the loss value of the policy model is composed of the loss value of the world model and the loss value of the action head module. The numerical value of the loss value can reflect the gap between the predicted result and the actual result. The loss value of the world model is composed of the reconstruction observation loss value of the decoder and the prediction loss value of the environmental dynamics prediction model. The reconstruction observation loss value can reflect the gap between the reconstructed observation data output by the decoder and the actual observation data, and the prediction loss value can reflect the gap between the predicted state of the environmental dynamics prediction model and the future actual state. During the training process, the model parameters are updated guided by the loss value of the policy model.

[0042] For example, for the world model, the decoder (Decoder) and the environment dynamics prediction model (Dynamic) are trained separately. The loss value of the world model is calculated as: ; In the formula, represents the encoder, represents the loss of reconstructing the observations, Represents the prediction loss of the state sequence, which can also be called the state trajectory. First, the Token obtained in the above step is decoded by the Decoder to reconstruct the observation, and the reconstruction observation loss value is calculated with the true value of the observation. This item is used as an auxiliary loss for learning the dynamics of the environment, because the rich supervision signal provided by the reconstructed image can enable the network to effectively learn the environment representation. Secondly, the future state is predicted cyclically (state It can also be represented by Token), and the state trajectory is obtained , that is, a series of states, and uses the hidden states encoded by future real observations to generate the true labels corresponding to the state trajectories, so that the environmental dynamics prediction model can learn to accurately predict future states.

[0043] For the attention model (Transformer), all the semantic units mentioned above (including language token, image token and state token) are input into the Transformer, and the latent vector containing the overall environment and task information is obtained through the attention mechanism. .

[0044] For the trajectory prediction model, the predicted key point trajectory output by the trajectory prediction model is recorded as .

[0045] For the action head module, the action head module receives the latent vector , the predicted key point trajectory is recorded as These two inputs and output the predicted action of the intelligent mechanical device (robotic arm) , as the pending action of the machine: .

[0046] Compare the predicted action with the corresponding expert policy action in the expert data and calculate the loss value of the action head module : ; in, represents expert data; State the task objectives, i.e. Lang mentioned above; Indicates the current status.

[0047] In summary, the loss function of the strategy model is: ; in, is the action loss term weight.

[0048] In one implementation, the training method of the trajectory prediction model includes: Determine whether the current number of training steps meets the training interval of the trajectory prediction model; If the conditions are met, the predicted state sequence output by the environmental dynamics prediction model is obtained, and a training prediction key point trajectory is generated according to the predicted state sequence by the trajectory prediction model; Based on the historical observation data of the corresponding number of frames in the expert dataset, a reference prediction key point trajectory is generated through a point tracking model; Calculating a loss value of the trajectory prediction model according to the training predicted key point trajectory and the reference predicted key point trajectory; The model parameters of the trajectory prediction model are updated according to the loss value of the trajectory prediction model.

[0049] Specifically, for the model parameter update process of the trajectory prediction model, the current training step number is first judged. When the training interval of the trajectory prediction model is met, the model parameters of the trajectory prediction model are updated. The environmental dynamics prediction model trained in the above steps is used to cyclically generate the prediction state of T time steps. The state of this time series is the prediction state sequence, which is used as the input data of the trajectory prediction model. The trajectory prediction model outputs the training prediction key point trajectory based on the input data. At the same time, the point tracking model (Co-tracker) is used to track the key points of the real image observation data of the future T frames to obtain the reference prediction key point trajectory, where the point tracking model is a pre-trained tracking model. The reference prediction key point trajectory is used as the true label (i.e., the supervisory signal) of the trajectory prediction model. By comparing the reference prediction key point trajectory with the training prediction key point trajectory output by the trajectory prediction model, the performance of the trajectory prediction model can be evaluated, and the loss value of the trajectory prediction model can be calculated. The model parameters are updated based on the loss value.

[0050] For example, Figure 3 As shown, the current status is , using environmental dynamics prediction models to imagine future states backwards , etc., which are input into the trajectory prediction model trained by the point tracking model, and the trajectory prediction model outputs The coordinate trajectory of the key points in the observed image is the predicted key point trajectory, where is a custom hyperparameter.

[0051] In one implementation, the historical observation data corresponding to the number of frames in the expert dataset includes a plurality of consecutive frames of real image observation data; based on the historical observation data corresponding to the number of frames in the expert dataset, generating a reference prediction key point trajectory through a point tracking model includes: A plurality of feature points are obtained by sampling the real image observation data of the first frame; Based on the real image observation data of each frame and the position of each feature point, generating trajectory change data of each feature point in each frame through a point tracking model; Calculating the change amplitude of each feature point according to each trajectory change data; A number of target feature points are selected according to the magnitude of each change, and a reference prediction key point trajectory is generated according to the trajectory of each target feature point.

[0052] Specifically, the specific process of using the point tracking model (Co-tracker) to track the key points of the real image observation data of the future T frames is as follows: First, sample in the first frame image to obtain feature points (or called initial points, sampling points), input the positions of the consecutive T frame images and feature points into the point tracking model, and obtain the trajectory changes of the feature points in each frame. Secondly, calculate the total length of the trajectory changes of all feature points in T time steps, and rank all feature points according to the magnitude of change. Finally, select the feature point with the largest magnitude of change as the key point , that is, a set of target feature points are obtained, and the reference predicted key point trajectory is determined according to the trajectories corresponding to these target feature points as the true label (or predicted label) of the trajectory prediction model.

[0053] The advantages of the present invention are: 1. The present invention uses a pre-trained lightweight sentence embedding model to encode different task descriptions into fixed-dimensional embedding vectors. The expert strategy is effectively fitted through a multi-task imitation learning decision model assisted by key points. The multi-task imitation learning decision model can process multiple inputs such as language and images, and performs well in terms of convergence speed and task success rate.

[0054] 2. While learning the expert strategy through the behavior cloning algorithm, the present invention builds a world model to realize the trajectory prediction of the future state. It can predict the movement trajectory of the key points in the observation at each time step, and use this information as the auxiliary input of the action head module for the final decision.

[0055] 3. The present invention can standardize scene representation by adding a world model, and combined with dynamic trajectory auxiliary decision-making information, it can better realize autonomous decision-making of the robotic arm in multi-task scenarios.

[0056] Based on the above embodiment, the present invention further provides a terminal, whose principle block diagram can be shown as follows: Figure 4As shown. The terminal includes a processor, a memory, a network interface, and a display screen connected through a system bus. Among them, the processor of the terminal is used to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the terminal is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an action prediction method based on key points and imitation learning is implemented. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0057] Those skilled in the art will understand that Figure 4 The principle block diagram shown in the figure is only a block diagram of a partial structure related to the scheme of the present invention, and does not constitute a limitation on the terminal to which the scheme of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0058] In one implementation, the memory of the terminal stores one or more programs, and is configured to be executed by one or more processors. The one or more programs include instructions for performing a motion prediction method based on key points and imitation learning.

[0059] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0060] In summary, the present invention discloses an action prediction method, terminal and storage medium based on key points and imitation learning, which relates to the field of embodied intelligent technology. The method obtains task information and observation data of an intelligent mechanical device; inputs the task information and the observation data into a trained multi-task imitation learning decision model to obtain the predicted execution action of the intelligent mechanical device; wherein the multi-task imitation learning decision model includes: an encoding module for encoding the task information and the observation data, and generating a number of semantic units according to the encoded data; a world model for generating a predicted state sequence according to each of the semantic units through a decoder and an environmental dynamics prediction model; a trajectory prediction model for generating a predicted key point trajectory according to the predicted state sequence; an attention model for generating a latent vector according to each of the semantic units; wherein the latent vector is used to reflect the overall information of the environment and the task; an action head module for generating the predicted execution action of the intelligent mechanical device according to the predicted key point trajectory and the latent vector. The present invention combines the idea of ​​the world model with imitation learning, learns a scene representation that is more in line with the environmental context through the environmental dynamics prediction model, and predicts the key point motion trajectory through the trained trajectory prediction model to assist the action head module in making action decisions. The present invention uses the motion trajectory of key points as auxiliary information, which can effectively improve the decision-making accuracy of the intelligent agent. First, the key points will not be affected by environmental changes such as lighting. Even if the object is partially blocked, the remaining key point trajectory can still provide the same information, so it has good robustness. Secondly, there is a definite mapping relationship between the motion trajectory of the key points and the motion trajectory of the intelligent mechanical device in three-dimensional space, so the neural network can better fit the movement of the intelligent mechanical device.

[0061] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. An action prediction method based on key points and imitation learning, characterized in that: The method comprises: Obtain mission information and observation data of intelligent mechanical devices; Inputting the task information and the observation data into a trained multi-task imitation learning decision model to obtain a predicted execution action of the intelligent mechanical device; Wherein, the multi-task imitation learning decision model includes: An encoding module, used for encoding the task information and the observation data, and generating a plurality of semantic units according to the encoded data; A world model, used to generate a predicted state sequence according to each of the semantic units through a decoder and an environmental dynamics prediction model; A trajectory prediction model, used to generate a predicted key point trajectory according to the predicted state sequence; An attention model, used to generate a latent vector according to each of the semantic units; wherein the latent vector is used to reflect the overall information of the environment and the task; The action head module is used to generate a predicted execution action of the intelligent mechanical device according to the predicted key point trajectory and the latent vector.

2. The action prediction method based on key points and imitation learning according to claim 1, characterized in that: The observation data includes image observation data from a plurality of camera perspectives and state observation data of the intelligent mechanical device.

3. The action prediction method based on key points and imitation learning according to claim 2, characterized in that: The task information and the observation data are encoded, and a plurality of semantic units are generated according to the encoded data, including: Acquire the semantic unit corresponding to the task information through a language encoder; Obtaining semantic units corresponding to the image observation data through a visual encoder and a SimNorm layer; wherein the SimNorm layer is used to divide the encoding vector into a number of segments of a preset length, perform a softmax operation on each segment separately, and reassemble them; The semantic unit corresponding to the state observation data is obtained through the state encoder.

4. The action prediction method based on key points and imitation learning according to claim 1, characterized in that: According to each of the semantic units, a predicted state sequence is generated through a decoder and an environmental dynamics prediction model, including: Based on all the semantic units, generating reconstructed observation data through the decoder; Based on the reconstructed observation data, the predicted states of several future time steps are cyclically generated through the environmental dynamics prediction model to obtain the predicted state sequence.

5. The action prediction method based on key points and imitation learning according to claim 1, characterized in that: The training method of the multi-task imitation learning decision model includes: The expert trajectories are collected through the expert strategy module, and an expert data set is established according to the expert trajectories; wherein each expert data includes historical observation data, corresponding expert strategy actions and corresponding historical task information; The encoding module, the world model, the attention model and the action head module are used as a strategy model; The strategy model and the trajectory prediction model are trained using the expert data set, and model parameters of the strategy model and the trajectory prediction model are updated respectively.

6. The action prediction method based on key points and imitation learning according to claim 5, characterized in that: The training method of the strategy model includes: Calculating the loss value of the world model and the loss value of the action head module through the expert data set; wherein the loss value of the world model is determined based on the reconstruction observation loss value of the decoder and the prediction loss value of the environment dynamics prediction model; Determining the loss value of the strategy model according to the loss values ​​corresponding to the world model and the action head module respectively; The model parameters of the strategy model are updated according to the loss value of the strategy model.

7. The action prediction method based on key points and imitation learning according to claim 5, characterized in that: The training method of the trajectory prediction model includes: Determine whether the current number of training steps meets the training interval of the trajectory prediction model; If the conditions are met, the predicted state sequence output by the environmental dynamics prediction model is obtained, and a training prediction key point trajectory is generated according to the predicted state sequence by the trajectory prediction model; Based on the historical observation data of the corresponding number of frames in the expert dataset, a reference prediction key point trajectory is generated through a point tracking model; According to the training predicted key point trajectory and the reference predicted key point trajectory, a loss value of the trajectory prediction model is calculated; and according to the loss value of the trajectory prediction model, a model parameter of the trajectory prediction model is updated.

8. The action prediction method based on key points and imitation learning according to claim 7, characterized in that: The historical observation data corresponding to the number of frames in the expert data set includes a plurality of consecutive frames of real image observation data; Based on the historical observation data of the corresponding frame number in the expert dataset, a reference prediction key point trajectory is generated through a point tracking model, including: A plurality of feature points are obtained by sampling the real image observation data of the first frame; Based on the real image observation data of each frame and the position of each feature point, generating trajectory change data of each feature point in each frame through a point tracking model; Calculating the change amplitude of each feature point according to each trajectory change data; A number of target feature points are selected according to the magnitude of each change, and a reference prediction key point trajectory is generated according to the trajectory of each target feature point.

9. A terminal, characterized in that: The terminal includes a memory and at least one processor; the memory stores a program; the program contains instructions for executing the action prediction method based on key points and imitation learning as described in any one of claims 1-8; and the processor is used to execute the program.

10. A computer-readable storage medium having a plurality of instructions stored thereon, characterized in that: The instructions are suitable for being loaded and executed by a processor to implement the steps of the action prediction method based on key points and imitation learning as described in any one of claims 1-8.

Images