Training method of motion simulation model, storage medium, equipment and program product

Abstract

The invention provides a training method of an action simulation model, and relates to the technical field of robots. The method comprises the following steps: acquiring an action feature corresponding to a target action to obtain a target action feature; determining a known action matched with the target action according to the target action feature to obtain a target known action; acquiring an action simulation model corresponding to the target known action from a model library to obtain a target action simulation model; and performing model training on the target action simulation model based on the target action to obtain an action simulation model corresponding to the target action. According to the method, the to-be-simulated target action is compared with the existing known action, the known action similar to the target action is found, the trained model corresponding to the known action serves as the initialization model, further training is conducted, model reuse of the similar action is achieved, the model training efficiency is improved, and consumption of computing resources is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of robots, in particular to a training method of a motion simulation model, a storage medium, an equipment and a program product. BACKGROUND

[0002] With the development of robot technology, more and more fields need to use robots to replace humans to do some work, which requires robots to be able to imitate human actions in order to adapt to various application environments.

[0003] At present, when training the motion simulation model used for robots to imitate human actions, it is often repeated to train similar actions, which leads to low model training efficiency and waste of computing resources. SUMMARY

[0004] Therefore, the embodiments of the present application provide a training method of a motion simulation model, a storage medium, an equipment and a program product.

[0005] In a first aspect, an embodiment of the present application provides a training method of a motion simulation model, comprising: obtaining an action feature corresponding to a target action to obtain a target action feature; determining a known action matched with the target action according to the target action feature to obtain a target known action; obtaining a motion simulation model corresponding to the target known action from a model library to obtain a target motion simulation model, the model library comprising at least one trained motion simulation model corresponding to a known action; and performing model training on the target motion simulation model based on the target action to obtain a motion simulation model corresponding to the target action.

[0006] In combination with the first aspect, in some implementation manners of the first aspect, the determining of the known action matched with the target action according to the target action feature to obtain the target known action comprises: mapping the target action feature to a latent space to obtain first latent space representation information corresponding to the target action; querying second latent space representation information matched with the first latent space representation information from a latent space representation library, the latent space representation library comprising at least one latent space representation information corresponding to a known action; and in a case where the second latent space representation information exists in the latent space representation library, determining the known action corresponding to the second latent space representation information as the target known action.

[0007] In combination with the first aspect, in some implementation manners of the first aspect, the mapping of the target action feature to the latent space to obtain the first latent space representation information corresponding to the target action comprises: mapping the target action feature to the latent space by using a trained variational autoencoder to obtain the first latent space representation information corresponding to the target action.

[0008] With reference to the first aspect, in some implementations of the first aspect, before the target action feature is mapped to the latent space by using the trained variational autoencoder to obtain the first latent space representation information corresponding to the target action, the method further includes: obtaining an action feature corresponding to an action sample based on a preset action dataset to obtain a first action feature; inputting the first action feature to an encoder in the variational autoencoder, and mapping the first action feature to the latent space by using the encoder to obtain third latent space representation information corresponding to the action sample; inputting the third latent space representation information to a decoder in the variational autoencoder, and performing feature reconstruction on the third latent space representation information by using the decoder to obtain a second action feature; training the variational autoencoder according to the first action feature and the second action feature to obtain the trained variational autoencoder; and mapping the target action feature to the latent space by using the trained variational autoencoder to obtain the first latent space representation information corresponding to the target action, including: inputting the target action feature to the encoder in the trained variational autoencoder, and mapping the target action feature to the latent space by using the encoder to obtain the first latent space representation information corresponding to the target action.

[0009] With reference to the first aspect, in some implementations of the first aspect, obtaining an action feature corresponding to an action sample based on a preset action dataset to obtain a first action feature includes: obtaining device action data corresponding to a plurality of action samples based on the preset action dataset; determining position information of a zero moment point corresponding to each action sample according to the device action data corresponding to each action sample; determining a position relationship between the zero moment point corresponding to each action sample and a preset support polygon according to the position information; performing screening on the device action data corresponding to the plurality of action samples based on the position relationship to obtain device action data corresponding to screened action samples; and obtaining an action feature corresponding to the screened action samples based on the device action data corresponding to the screened action samples to obtain the first action feature.

[0010] With reference to the first aspect, in some implementations of the first aspect, performing screening on the device action data corresponding to the plurality of action samples based on the position relationship to obtain the device action data corresponding to the screened action samples includes: in a case where a position relationship corresponding to a target action sample satisfies a first preset condition, deleting the device action data corresponding to the target action sample, the target action sample including any action sample in the plurality of action samples; and wherein the first preset condition includes that the zero moment point corresponding to the target action sample is outside a boundary of the preset support polygon, and a distance between the zero moment point and the boundary of the preset support polygon is greater than a preset distance threshold.

[0011] In some implementations of the first aspect, the deleting the device action data corresponding to the target action sample comprises: obtaining a duration when a position relationship corresponding to the target action sample satisfies a first preset condition; and deleting the device action data corresponding to the target action sample when the duration is greater than a preset duration threshold.

[0012] In some implementations of the first aspect, the querying, from the latent space representation library, the second latent space representation information matching the first latent space representation information comprises: determining, from the latent space representation library, a latent space representation information most similar to the first latent space representation information to obtain a target latent space representation information; and determining the target latent space representation information as the second latent space representation information when a similarity between the first latent space representation information and the target latent space representation information is greater than a preset similarity threshold.

[0013] In some implementations of the first aspect, the model training, based on the target action, of the target action simulation model to obtain the action simulation model corresponding to the target action comprises: obtaining model training element information corresponding to the target action; and model training, based on a Markov decision process, of the target action simulation model according to the model training element information to obtain the action simulation model corresponding to the target action; wherein the action simulation model comprises an input layer, an output layer, and a target linear projection layer, and the input layer and the output layer are connected through a residual connection of the target linear projection layer.

[0014] In some implementations of the first aspect, before obtaining the action simulation model corresponding to the target action, the method further comprises: obtaining privileged observation information corresponding to the target action; and performing error tracking according to the privileged observation information in the model training process.

[0015] In the second aspect, an embodiment of the present application provides a training device of an action simulation model, the device comprising: a first obtaining module configured to obtain an action feature corresponding to a target action to obtain a target action feature; a determining module configured to determine a known action matching the target action according to the target action feature to obtain a target known action; a second obtaining module configured to obtain an action simulation model corresponding to the target known action from a model library to obtain a target action simulation model, the model library comprising at least one trained action simulation model corresponding to a known action; and a model training module configured to model train, based on the target action, of the target action simulation model to obtain the action simulation model corresponding to the target action.

[0016] In the third aspect, an embodiment of the present application provides a computer readable storage medium storing a computer program, the computer program being configured to execute the training method of the action simulation model of the first aspect.

[0017] In a fourth aspect, an embodiment of the present application provides an electronic device, comprising: a processor; a memory for storing processor-executable instructions; the processor is configured to execute the training method of the action simulation model of the first aspect.

[0018] In a fifth aspect, an embodiment of the present application provides a computer program product, comprising instructions for causing an electronic device to implement the training method of the action simulation model of the first aspect when the instructions are executed on the electronic device.

[0019] In the present application, by comparing the target action to be simulated with the existing known action, the known action similar to the target action is determined, and then the known action corresponding to the trained model is used as an initialization model for further training, thereby realizing model reuse of similar actions, further improving the model training efficiency and reducing the consumption of computing resources. BRIEF DESCRIPTION OF DRAWINGS

[0020] The above and other objects, features and advantages of the present application will become more apparent from the following detailed description when taken in conjunction with the accompanying drawings in which:

[0021] Figure 1 Fig. 1 shows a flowchart of a training method of an action simulation model according to an embodiment of the present application.

[0022] Figure 2 Fig. 2 shows a flowchart of a training method of an action simulation model according to another embodiment of the present application.

[0023] Figure 3 Fig. 3 shows a structural diagram of an action simulation model according to an embodiment of the present application.

[0024] Figure 4 Fig. 4 shows a flowchart of a training method of an action simulation model according to an embodiment of the present application.

[0025] Figure 5 Fig. 5 shows a structural diagram of a training device of an action simulation model according to an embodiment of the present application.

[0026] Figure 6 Fig. 6 shows a structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] With reference to the drawings and the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts are within the scope of the present application.

[0028] In addition, in order to better illustrate the present application, a large number of specific details are given in the specific embodiments below. Those skilled in the art should understand that the present application can also be implemented without some specific details. In some examples, methods and means well known to those skilled in the art are not described in detail in order to highlight the main ideas of the present application.

[0029] It should be noted that: similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0030] In addition, if the terms "first", "second", "third" and "fourth" are used only to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0031] In related technologies, action imitation of agents or robots is mainly based on methods such as behavior cloning, inverse reinforcement learning or diffusion models, which extracts pose trajectories from demonstration videos or motion capture data, and uses neural network training control strategy to reproduce actions, but all have some problems. For example, behavior cloning can directly map observation states to actions, but is easily affected by data distribution offset, resulting in weak generalization ability; inverse reinforcement learning uses reward function optimization strategy, but is computationally intensive, takes longer to train, and is less efficient when processing large-scale action data sets; diffusion models can generate diverse action sequences, but the full-batch training mode easily leads to gradient conflicts, and the adaptation to new actions requires retraining the entire model, making it difficult to support real-time incremental learning. In summary, the action imitation in related technologies has problems such as low training efficiency, weak generalization ability, and slow adaptation to new actions, and therefore cannot meet the needs of real-time robot control or interactive scenarios. Therefore, a new action simulation model training method is needed to solve the problem of low training efficiency in existing technologies.

[0032] The technical scheme of the embodiment of the present application can query the known action matched with the target action, use the action simulation model corresponding to the known action as the initialization model of the target action, and perform model training of the target action based on the initialization model, so that the training results of the known action model can be fully utilized, repeated training of similar actions can be avoided, the training efficiency is improved, and the consumption of computing resources is reduced.

[0033] Figure 1 Fig. 1 shows a flowchart of a training method of an action simulation model provided by an example embodiment of the present application. As shown in Fig. 1, the training method of the action simulation model provided by the embodiment of the present application includes the following steps. Figure 1

[0034] In step S110, action features corresponding to a target action are obtained to obtain target action features.

[0035] The target action can be an action that an intelligent agent or a humanoid robot currently wants to learn.

[0036] In the embodiment of the present application, the action features can include at least one of joint position, speed and posture information, and correspondingly, the target action features can be at least one of joint position, speed and posture information corresponding to the target action.

[0037] In some embodiments, the target action features can be a feature vector corresponding to the target action, for example, the target action features can be a feature vector composed of joint position, speed and posture information corresponding to the target action.

[0038] In some embodiments, the target action can include a plurality of decomposed actions, and correspondingly, the target action features can be action features corresponding to the plurality of decomposed actions respectively.

[0039] Optionally, the action features can include time sequence features, spatial features and quantitative features, etc. The time sequence features are used to describe the sequence of actions, for example, jumping first and then waving. The spatial features are used to describe the specific position of the action, for example, can be joint angle, limb position, center of gravity coordinates, etc. The quantitative features are used to describe the parameter size related to the action, for example, can be action speed, force size, duration, etc.

[0040] Optionally, the feature vector can also be standardized and reduced in dimension, so that subsequent processing is more efficient, and the training effect and generalization ability of the model are also improved. Specifically, the standardization processing can avoid that some features have a dominant effect on model training due to a too large value range. The dimension reduction processing can reduce the dimension of the feature vector, remove redundant information, and extract the most critical features for action simulation, so as to reduce the computational complexity on the premise of ensuring the performance of the model. ​

[0041] Step S120, determining a known action matching the target action according to the target action feature, to obtain a target known action.

[0042] The known action can be an action that the agent or humanoid robot has completed training, i.e., an action that has been learned. In the embodiments of the present application, the known action is an action for which a corresponding trained action simulation model already exists in the model library.

[0043] The target known action can be an action with the highest matching degree with the target action. Correspondingly, determining a known action matching the target action according to the target action feature, to obtain a target known action includes: for each known action in the plurality of known actions, determining the corresponding action feature thereof, and matching it with the target action feature to obtain the highest matching degree, and determining the known action corresponding to the action feature with the highest matching degree as the target known action.

[0044] In the embodiments of the present application, the target known action can be a known action with the highest similarity to the target action.

[0045] In some embodiments, a feature matching algorithm can be used to calculate the similarity, calculate the similarity between the target action feature and the action feature of the known action in the model library, and determine the known action with the highest similarity as the target known action. Optionally, the feature matching algorithm can include at least one of cosine similarity, Euclidean distance, Manhattan distance, Pearson correlation coefficient, etc.

[0046] Step S130, obtaining an action simulation model corresponding to the target known action from the model library, to obtain a target action simulation model.

[0047] The model library includes at least one trained action simulation model corresponding to a known action.

[0048] Specifically, each known action in the model library has a unique corresponding trained action simulation model, and the correspondence between the plurality of action simulation models in the model library and the plurality of known actions is pre-set. Based on the target known action and the correspondence, the action simulation model in the model library corresponding to the target known action is determined as the target action simulation model.

[0049] Optionally, the identification information of each of the plurality of known actions is pre-set, and the correspondence between the identification information and the plurality of action simulation models in the model library is established. Correspondingly, obtaining an action simulation model corresponding to the target known action from the model library, to obtain a target action simulation model, includes: based on the correspondence and the identification information of the target known action, determining the action simulation model in the model library corresponding to the target known action as the target action simulation model.

[0050] In step S140, the target action simulation model is trained based on the target action to obtain an action simulation model corresponding to the target action.

[0051] The target action simulation model is an initial model of the target action currently trained. Correspondingly, the target action simulation model is trained based on the target action to obtain an action simulation model corresponding to the target action, which includes: determining model input data corresponding to the target action, inputting the model input data into the target action simulation model to obtain a simulation result, and correcting model parameters in the target action simulation model based on the simulation result and a loss function of the target action; taking convergence of the loss function in the target model as a training target to obtain the action simulation model corresponding to the target action.

[0052] It can be understood that, before training the target action simulation model, a training sample set corresponding to the target action simulation model needs to be constructed, so as to train the target action simulation model based on each training sample in the training sample set to obtain the action simulation model corresponding to the target action.

[0053] In order to make the target action simulation model learn the time sequence execution rule of the target action more accurately and improve the convergence efficiency and simulation accuracy of model training, an optional embodiment is provided in the present application, which trains the target action simulation model based on Markov decision process by explicitly defining the core elements of model training and using a specific network structure, so as to accurately reproduce the target action. The specific implementation is as follows.

[0054] In the present application, the target action simulation model is trained based on the target action to obtain an action simulation model corresponding to the target action, which includes: obtaining model training element information corresponding to the target action; and training the target action simulation model based on Markov decision process according to the model training element information to obtain the action simulation model corresponding to the target action.

[0055] The model training element information is a core element for training the action simulation model, and can be a set of one or more model training element information, which is the input basis of Markov decision process (MDP). In the present application, the model training element information includes state space features, action space features, reward functions, etc. The state space features are used to describe the features at a certain moment during the execution of the action, for example, joint angles, positions, speeds, etc. The action space features are used to describe the next action instruction output by the action simulation model in the current state, for example, joint angle change amount, force adjustment parameters, etc. The reward function is used to determine the matching degree between the output action of the action simulation model and the target action, for example, the smaller the error, the higher the reward.

[0056] For example, the state space feature can be based on the key features of the target action at each time point, such as the hip joint angle, the center of gravity height, the rotation speed, etc. The reward function can be determined based on the joint angle error between the output action of the action simulation model and the target action. For each first angle reduction in the error, the reward increases by a first preset score. If the error exceeds a preset value, the reward decreases by a second preset score. The action space feature can be based on the next step action adjustment parameter of the model, such as the hip joint angle increasing the target angle and the rotation speed increasing the target speed.

[0057] Optionally, the state space feature can also include a state transition parameter for describing the rule of transition from the current state to the next state, i.e., the process of transition from the first state to the second state after the simulation determines the transition to the new state, for example, after the hip joint angle is increased by the target angle, the new state of the hip joint angle becomes the second angle.

[0058] The Markov decision process is a mathematical framework for describing sequential decision problems, and the core is the Markov property, i.e., the future state only depends on the current state and the current selected action, and is independent of the past state. The advantage of this is that the action simulation model can learn to select a specific action in a specific state to obtain the maximum cumulative reward, thereby optimizing the action sequence.

[0059] In some examples, the action simulation model includes an input layer, an output layer, and a target linear projection layer, and the input layer and the output layer are connected through the target linear projection layer. Specifically, the input layer is used to receive the input data of the action simulation model (e.g., joint angle, position, etc.) and convert it into a feature vector that can be processed by the action simulation model; the target linear projection layer is a linear transformation layer, which is implemented through matrix multiplication and bias, and is mainly used to map the feature vector of the input layer to a space matching the dimension of the output layer, providing a basis for residual connection; the output layer is used to output the action simulation result. For example, the action simulation result can be an action instruction, which is used to describe the next step joint angle change, speed adjustment, etc.

[0060] The input layer is connected to the output layer through the target linear projection layer, which can alleviate the gradient vanishing problem of deep network, ensure that the gradient can be effectively returned, make the action simulation model converge faster, reduce the number of training times, increase the training efficiency of the action simulation model, and at the same time retain the core information of the original input action feature, improve the action simulation accuracy.

[0061] To further improve the accuracy and convergence speed of model training, on the basis of completing policy optimization based on a Markov decision process, an optional solution is provided in the embodiments of the present application, which introduces privileged observation information to assist in supervising the training process, so as to realize more fine-grained perception and correction of tracking errors, thereby enhancing the stability and fidelity of the action simulation model. The specific implementation is as follows.

[0062] In the embodiments of the present application, before obtaining the action simulation model corresponding to the target action, the method further comprises: obtaining privileged observation information corresponding to the target action; and tracking errors according to the privileged observation information in the model training process.

[0063] The privileged observation information is additional auxiliary information obtained in the training stage of the action simulation model, and cannot be obtained in the actual application (inference) stage of the model, and is used to describe the deep law of the action or the error source. For example, the privileged observation information can be simulation data, including real center of gravity position, global velocity, accurate torque of each joint, and foot contact force data.

[0064] In some scenarios, considering that some data on the real robot has large errors due to large sensor noise, or cannot be collected at all, and therefore cannot be used as input of the action simulation model. Therefore, error tracking can be realized based on the above-mentioned frame number data, which is converted into reward punishment or gradient weight, to pull back the policy parameters in real time, so that the action simulation model can approach the action with higher accuracy more quickly.

[0065] Specifically, for each simulation step in the simulation environment, the privileged observation is used to determine the immediate error (for example, it can be attitude drift, ZMP out of bounds, action deviation, etc.), and the policy gradient or reward is immediately modified based on the immediate error to reduce the error from the target action and avoid error accumulation.

[0066] It can be understood that, in the early stage of training, there is an error between the simulation action output by the action simulation model and the target action; in the conventional training, the action simulation model can only perceive the error size through reward feedback, and cannot determine the specific reason for the error; after introducing the privileged observation information, the deep feature information related to the simulation action output by the action simulation model and the target action is compared with the privileged observation information, the error source is located, the parameters of the action simulation model are adjusted based on the error source, the prediction accuracy of the action simulation model for the key feature information is optimized, and the error between the simulation action output by the action simulation model and the target action is reduced.

[0067] Optionally, the error source can be the prediction deviation of the action simulation model for the key parameters in the execution of the action.

[0068] For example, in the simulation of the target shooting action in the simulation environment, at the initial stage of training, the simulated shooting action output by the action simulation model has errors with the target shooting action. For example, the error can be that the flight angle of the ball corresponding to the simulated shooting action is smaller than the flight angle of the ball corresponding to the target shooting action, and the hit rate is low. In the conventional training, the action simulation model only sets a low reward based on the error, but cannot determine whether the reason for the error is insufficient wrist force or body center of gravity deviation. After introducing privileged observation information, the wrist torque predicted by the action simulation model is compared with the privileged information. For example, the torque output by the action simulation model can be 5 Nm, and the torque of the target action can be 8 Nm, and it is further determined that the torque error causes the insufficient flight angle of the ball. The parameters associated with torque prediction in the action simulation model are adjusted to improve the torque output by the action simulation model.

[0069] The technical scheme of the embodiment of the present application uses privileged observation information to analyze the error between the simulated action output by the action simulation model and the target action in real time during model training, and locates the specific reason for the error, which can provide a reference for parameter adjustment of the action simulation model and further improve the model training efficiency.

[0070] In order to determine the termination condition of model training and ensure that the action simulation model obtained by training meets the preset accuracy requirement, a specific error judgment standard can be set based on the device action to determine whether the model training is completed, which can not only avoid insufficient training to cause insufficient simulation action accuracy of the action simulation model, but also prevent waste of computing resources caused by overtraining, and further ensure the reliability and practicality of the final action simulation model.

[0071] In the embodiment of the present application, the model training of the target action simulation model is performed based on the Markov decision process according to the model training element information, and the action simulation model corresponding to the target action is obtained, including: performing the model training of the target action simulation model based on the Markov decision process according to the model training element information until a second preset condition is met, and obtaining the action simulation model corresponding to the target action.

[0072] The second preset condition includes that the height or direction error of the device reference coordinate system is less than a preset threshold, and the spatial position of the device action executor does not deviate from the reference trajectory.

[0073] The device reference coordinate system is a coordinate system for describing the position and posture of a device (such as a robot, a mechanical arm, or an intelligent agent). The center of the device base or a fixed reference point is usually taken as the origin, and the height (Z-axis coordinate of the origin) and direction (direction of the coordinate axis) are the bases for measuring the overall positioning accuracy of the device.

[0074] The device reference coordinate system height error can be a difference between a device reference coordinate system origin height (Z-axis coordinate) corresponding to a simulated action output by the action simulation model and an actual height of the device reference coordinate system in the target action.

[0075] The device reference coordinate system direction error can be an angular deviation between a direction corresponding to a coordinate axis (such as an X-axis or a Y-axis) of the device reference coordinate system corresponding to the simulated action output by the action simulation model and a direction corresponding to an actual coordinate axis of the device reference coordinate system in the target action.

[0076] In some scenarios, the device reference coordinate system height or direction error is negatively correlated with the model accuracy. Therefore, a preset threshold can be set to determine the device reference coordinate system error or the degree of position deviation of the device action executor. If the error is greater than or equal to the threshold, the action simulation model accuracy is insufficient and needs to be continuously trained. If the error is less than the threshold, the training is stopped.

[0077] The device action executor can be a component of the device that performs a specific action, and the posture of the device action executor can reflect the accuracy of the simulated action output by the action simulation model. For example, the device action executor can be a gripper of a mechanical arm, a welding gun, or the like. When the device is a robot, the device action executor can be a hand, a leg, or the like of the robot.

[0078] The reference trajectory can be posture data of the device action executor when performing the target action, and can be used to measure the motion accuracy of the device action executor.

[0079] Optionally, the accuracy of the simulated action output by the action simulation model can be determined based on a deviation between the actual posture data of the device action executor and standard posture data at a corresponding time on the reference trajectory. Specifically, if the actual posture data of the device action executor deviates from the standard posture data at the corresponding time on the reference trajectory, the action simulation model accuracy is insufficient and needs to be continuously trained. If the actual posture data of the device action executor does not deviate from the standard posture data at the corresponding time on the reference trajectory, the training is stopped.

[0080] For example, the reference trajectory can be a rotation angle of each joint of the hand of the robot when performing the target action. Further, a preset angle threshold can be set to determine the degree of deviation of the rotation angle of each joint of the hand of the robot relative to the rotation angle of each joint corresponding to the reference trajectory. If the deviation between the actual rotation angle of each joint of the hand of the robot and the standard rotation angle of each joint on the reference trajectory exceeds the preset angle threshold, the action simulation model accuracy is insufficient and needs to be continuously trained. If the deviation between the actual rotation angle of each joint of the hand of the robot and the standard rotation angle of each joint on the reference trajectory does not exceed the preset angle threshold, the training is stopped.

[0081] In the embodiments of the present application, a reinforcement learning method is used to train a target action simulation model with real data of the target action as a reference. In the training process, the reward function generally includes pose tracking reward, speed matching reward, collision penalty, and energy penalty, etc. The final action simulation model is obtained as the action simulation model corresponding to the target action.

[0082] The technical solutions of the embodiments of the present application realize the rapid imitation and model training of new actions through action feature extraction, similar action retrieval, model weight reuse, and reinforcement learning fine-tuning, significantly improving the training efficiency and the accuracy of action imitation.

[0083] Considering that the original action features of the target action are usually high-dimensional and have a large amount of data, in order to improve the calculation efficiency, the original action features of the target action can be reduced in dimension. In the embodiments of the present application, after the action features are mapped to the latent space, the action features are compressed and abstracted, thereby reducing the amount of subsequent calculation data and improving the calculation efficiency.

[0084] In order to more accurately match the reference object that best fits the target action from the known actions, and improve the accuracy and efficiency of action matching, the action features can be further mapped to the latent space for matching. In order to realize efficient comparison of known actions and target actions at the feature level, a latent space representation library containing latent space representation information of known actions can be constructed to support the matching process.

[0085] In the embodiments of the present application, the known action matched with the target action is determined according to the target action features, and the target known action is obtained, including: mapping the target action features to the latent space to obtain first latent space representation information corresponding to the target action; querying second latent space representation information matched with the first latent space representation information from the latent space representation library, the latent space representation library including at least one latent space representation information corresponding to a known action; in the case that the second latent space representation information exists in the latent space representation library, determining the known action corresponding to the second latent space representation information as the target known action.

[0086] The latent space is a high-dimensional abstract space constructed by a model after learning the internal laws of data, which maps the original high-dimensional and complex data (such as action features) to low-dimensional and compact latent space vectors.

[0087] The latent space representation information includes a low-dimensional vector obtained after mapping the target data to the latent space, which contains the core features of the original data and is easier to calculate the similarity. It should be noted that the latent space representation information involved in the present application can be a latent space vector. The first latent space representation information includes a low-dimensional vector obtained after mapping the target action features to the latent space.

[0088] Optionally, the encoder can be pre-trained, and the encoder can be used to encode the input action feature into corresponding hidden space representation information. Further, the target action feature is mapped to the hidden space based on the pre-trained encoder to obtain the first hidden space representation information corresponding to the target action. Specifically, the target action feature is input into the pre-trained encoder, and the input data of the encoder is determined as the first hidden space representation information corresponding to the target action.

[0089] The second hidden space representation information includes a low-dimensional vector obtained after the action feature of the known action is mapped to the hidden space.

[0090] The hidden space representation library includes hidden space representation information of a plurality of known actions, and each hidden space representation information in the hidden space representation library corresponds to one known action. In the case where the second hidden space representation information exists in the hidden space representation library, the known action corresponding to the second hidden space representation information is determined as the target known action, including: calculating the similarity between the first hidden space representation information and the hidden space representation information in the hidden space representation library, and determining the hidden space representation information with the highest similarity in the hidden space representation library as the second hidden space representation information. Further, the known action corresponding to the second hidden space representation information in the hidden space representation library is determined as the target known action.

[0091] In the embodiments of the present application, the variational autoencoder can be used to map the target action feature to the hidden space. Correspondingly, the target action feature is mapped to the hidden space to obtain the first hidden space representation information corresponding to the target action, including: using the trained variational autoencoder to map the target action feature to the hidden space to obtain the first hidden space representation information corresponding to the target action.

[0092] The variational autoencoder (VAE) is a neural network-based generative model composed of an encoder and a decoder, which is used to learn the probability distribution of data and map it to the hidden space. The output hidden space representation information not only preserves the core features of the original data, but also conforms to a specific probability distribution (usually normal distribution), making the hidden space vector more regular and generalizable.

[0093] It can be understood that before the variational autoencoder is trained, a training sample set needs to be constructed first, and the encoder and the decoder of the variational autoencoder are trained based on the training sample set. Specifically, a large amount of original action feature data of actions is collected, and the action features are input into the encoder to compress the action features into a latent space vector; further, the decoder restores the latent space vector to action feature data, outputs the to-be-processed action feature data, and optimizes the parameters of the encoder and the decoder based on the error minimization between the to-be-processed action feature data output by the decoder and the original action feature data. The trained encoder can stably output a latent space vector conforming to a probability distribution, and the decoder can restore the original action feature from the latent space vector.

[0094] In addition, the above-mentioned querying, from the latent space representation library, second latent space representation information matching the first latent space representation information can specifically include: determining, from the latent space representation library, latent space representation information most similar to the first latent space representation information to obtain target latent space representation information; in a case where a similarity between the first latent space representation information and the target latent space representation information is greater than a preset similarity threshold, determining that the target latent space representation information is the second latent space representation information.

[0095] Exemplarily, the similarity between the latent space representation information can be determined based on a Euclidean distance or a cosine distance, for example, a Euclidean distance between the first latent space representation information and the latent space representation information in the latent space representation library is calculated to obtain a minimum distance, and the target latent space representation information corresponding to the minimum distance in the latent space representation library is determined as the second latent space representation information, and the known action corresponding to the second latent space representation information in the latent space representation library is determined as the target known action.

[0096] Reference Figure 2 In the embodiment of the present application, before the target action feature is mapped to the latent space by using the trained variational autoencoder to obtain the first latent space representation information corresponding to the target action, the method further includes the following steps.

[0097] In step S210, based on a preset action data set, action features corresponding to action samples are obtained to obtain first action features.

[0098] The preset action data set can include a large number of action samples. The first action feature is original action feature data corresponding to each action sample.

[0099] Optionally, the preset action data set can include a large number of motion data of actions, and each action data includes an action sample and original action feature data corresponding to the action sample.

[0100] It should be noted that the action sample can be determined based on a known action, or can be determined based on other actions, and the embodiments of the present application do not make specific limitations thereto.

[0101] Optionally, the preset action data set can be a large-scale public action data set, for example, can be an Archive of Motion Capture as Surface Shapes (AMASS) data set, an Ubisoft La Forge Animation Dataset (LAFAN1), or the like.

[0102] In the embodiments of the present application, based on the preset action data set, the action features corresponding to the action samples are obtained to obtain the first action features, including: based on the preset action data set, obtaining device action data corresponding to a plurality of action samples respectively; determining the position information of the zero moment point corresponding to each action sample according to the device action data corresponding to each action sample; determining the position relationship between the zero moment point corresponding to each action sample and the preset support polygon according to the position information; filtering the device action data corresponding to the plurality of action samples based on the position relationship to obtain the device action data corresponding to the filtered action samples; and obtaining the action features corresponding to the filtered action samples based on the device action data corresponding to the filtered action samples to obtain the first action features.

[0103] The device action data can be specific action data of an action performer when performing an action corresponding to an action sample, for example, the device action data can be specific action parameters (such as position, angle, force, mass, acceleration, torque, etc.) of an agent or a robot.

[0104] Optionally, the device action data can be original action data of an action performer obtained through an action acquisition device, for example, the device action data can be joint positions recorded through a motion capture camera and foot pressure recorded through a pressure sensor when a robot is walking.

[0105] In the embodiments of the present application, the action samples in the preset action data set can be real action data of a human being. Correspondingly, based on the preset action data set, device action data corresponding to the plurality of action samples is obtained, including: based on simulation software, device action data corresponding to the plurality of action samples is obtained. For example, based on a simulation environment of Multi-Joint dynamics with Contact (MuJoCo), action data corresponding to the plurality of action samples is obtained, including at least one of mass, position, acceleration, torque and the like, which is taken as device action data. The advantage of this is that MuJoCo can support multi-thread / multi-process batch simulation running, thereby improving the action simulation model training efficiency.

[0106] A Zero Moment Point (ZMP) includes a point of action of a resultant force of a ground reaction force on a robot foot.

[0107] A preset support polygon is a polygon area formed by a support surface (such as a foot bottom, a robot foot) in contact with the ground when the robot performs an action, and is a reference boundary for determining whether the zero moment point is in a stable range. The vertices of the polygon are usually edge points of the support surface (such as the front palm, the heel, the lateral toe of the foot bottom, etc.).

[0108] The device action data corresponding to the screened action samples is stable and effective device action data.

[0109] It can be understood that the zero moment point is usually used to determine the dynamic stability of a robot. When the point is located within the preset support polygon, the robot can maintain stable walking, and the horizontal direction net moment of inertial force and gravity is zero. Therefore, based on the relative position relationship between the zero moment point (ZMP) and the preset support polygon, the device action data can be screened, the unbalanced and invalid action samples can be reduced, the stable and effective action samples can be retained, further, the first action features are extracted and the model is trained, the model is prevented from learning incorrect action rules, and the model training efficiency is improved.

[0110] Specifically, based on the preset action data set, action features corresponding to the action samples are obtained to obtain first action features, including the following steps: based on the preset action data set, action data corresponding to the plurality of action samples is obtained, including at least one of mass, position, acceleration, torque and the like, which is taken as device action data; for the device action data corresponding to each action sample, based on formula (1), position information of the zero moment point corresponding to each action sample is determined. , The (x,y) coordinates of the ground contact point (e.g., the foot contact point) are obtained, and a minimum convex polygon is generated using a convex hull algorithm (e.g., SciPy's ConvexHull). This polygon is then identified as the preset support polygon. Further, it is determined whether the zero-moment point is located within the boundary of the preset support polygon. If the zero-moment point corresponding to the target action sample is outside the boundary of the preset support polygon, the device action data corresponding to that target action sample is deleted. If the zero-moment point corresponding to the target action sample is within the boundary of the preset support polygon, the device action data corresponding to that target action sample is retained. This process yields the device action data corresponding to the filtered action samples. Based on the device action data corresponding to the filtered action samples, the first action feature corresponding to that device action data is determined.

[0111] (1) in, n The total number of robot bodies, For the first i The mass of the robot body, ( ) is the first i The position of the center of mass of the robot, ( ) is linear acceleration, ( , ) represents the external torque.

[0112] Optionally, the first motion feature may be a feature that emphasizes leg movement corresponding to the motion data of the device, such as at least one of features such as Cartesian spatial position, root translation, root velocity, foot contact state, and foot velocity.

[0113] Optionally, the relative positional relationship between the zero-moment point and the edge of the preset support polygon can be determined based on the distance between the zero-moment point and the boundary of the preset support polygon.

[0114] Considering that in some scenarios, the distance between the zero-moment point and the boundary of the preset support polygon is inversely proportional to the stability of the action sample, in order to improve the efficiency of action sample selection, an evaluation index can be set to measure the extent to which the ZMP exceeds the boundary of the preset support polygon, so that the selected action samples are more in line with the needs of subsequent model training.

[0115] In this embodiment of the application, a distance threshold is set to measure the degree to which the ZMP exceeds the boundary of the preset supporting polygon. Correspondingly, the device action data corresponding to multiple action samples are filtered based on the positional relationship to obtain the device action data corresponding to the filtered action samples, including: deleting the device action data corresponding to the target action sample when the positional relationship corresponding to the target action sample meets the first preset condition.

[0116] The target action sample includes any action sample in the plurality of action samples; and the first preset condition includes that a zero moment point corresponding to the target action sample is outside a boundary of a preset support polygon, and a distance between the zero moment point and the boundary of the preset support polygon is greater than a preset distance threshold.

[0117] In some scenarios, the imbalance degree is proportional to the distance of the zero moment point corresponding to the target action sample outside the boundary of the preset support polygon, and thus the imbalance degree can be determined based on the distance.

[0118] In some scenarios, the imbalance degree is proportional to the distance of the zero moment point corresponding to the target action sample outside the boundary of the preset support polygon, and thus the imbalance degree can be determined based on the distance.

[0119] In the embodiments of the present application, a preset time threshold is set to determine whether the imbalance state of the current action sample is persistent, and the device action data corresponding to the target action sample is deleted, including: obtaining a duration when the position relationship corresponding to the target action sample satisfies the first preset condition; and in a case where the duration is greater than the preset time threshold, deleting the device action data corresponding to the target action sample.

[0120] Specifically, based on the device action data of each time point of the target action sample, the device action data satisfying the first preset condition and the time point corresponding to the device action data are determined to determine the duration when the first preset condition is satisfied, and if the duration is greater than the preset time threshold, it is determined that the current action sample is in an imbalance state, and the device action data corresponding to the target action sample is deleted.

[0121] Optionally, if there are multiple discontinuous exceedances within a set time period, the maximum value of the durations can be determined, and in a case where the maximum value of the durations is greater than the preset time threshold, the device action data corresponding to the target action sample is deleted.

[0122] In some scenarios, the imbalance degree is proportional to the duration of the imbalance, and thus the imbalance degree can be determined based on the duration of the imbalance.

[0123] In the embodiments of the present application, by setting the preset distance threshold and the preset time, the device action data of the action failure / poor stability is deleted, and more action samples conforming to the dynamic law are retained, so that the action simulation model learns more accurate action rules in subsequent training, and the model training efficiency is improved.

[0124] Step S220, input the first action feature into an encoder in the variational autoencoder, and map the first action feature to the latent space by using the encoder to obtain third latent space representation information corresponding to the action sample.

[0125] The third latent space representation information can be a latent space vector obtained after the first action feature of the action sample is mapped to the latent space by the encoder in the variational autoencoder.

[0126] Specifically, the first action feature is input into the encoder in the variational autoencoder, and a low-dimensional vector corresponding to the first action feature is obtained after the first action feature is mapped to the latent space by the encoder, which is taken as the third latent space representation information corresponding to the action sample.

[0127] Step S230, input the third latent space representation information into a decoder in the variational autoencoder, and perform feature reconstruction on the third latent space representation information by using the decoder to obtain a second action feature.

[0128] The second action feature can be output data of the variational autoencoder, and specifically can be an action feature restored after the third latent space representation information is reconstructed by the decoder in the variational autoencoder.

[0129] Step S240, train the variational autoencoder according to the first action feature and the second action feature to obtain a trained variational autoencoder.

[0130] The action feature restored by the decoder in the variational autoencoder (the second action feature) is compared with the original action feature (the first action feature), and the parameters of the variational autoencoder are optimized based on the error between the first action feature and the second action feature, so that the second action feature reconstructed by the decoder is as close as possible to the first action feature, and the third latent space representation information conforms to the normal distribution (ensuring the regularity of the latent space), thereby obtaining the trained variational autoencoder.

[0131] Further, the trained variational autoencoder is used to map a target action feature to the latent space to obtain first latent space representation information corresponding to a target action, including: inputting the target action feature into an encoder in the trained variational autoencoder, and mapping the target action feature to the latent space by using the encoder to obtain the first latent space representation information corresponding to the target action.

[0132] Specifically, the target action feature is input into the encoder in the trained variational autoencoder, and the target action feature is mapped to the latent space by the encoder based on the learned action rule to obtain the first latent space representation information corresponding to the target action.

[0133] The technical scheme of the embodiment of the application can automatically filter irrelevant noise in the action and retain core features by mapping the original action features to the latent space, and meanwhile, can realize dimension reduction representation of the action features in the form of a low-dimensional vector, greatly improve the similarity calculation efficiency, and can more essentially reflect the action similarity through the latent space distance, effectively reduce the matching error caused by the original feature detail difference, and further improve the accuracy and generality of the action matching.

[0134] The above embodiment describes the training method of the action simulation model in detail, and the following describes the specific process of training the action simulation model based on the above training method in the robot action simulation scene. Figure 3 and Figure 4 The above embodiment describes the training method of the action simulation model in detail, and the following describes the specific process of training the action simulation model based on the above training method in the robot action simulation scene.

[0135] Figure 3 As shown in FIG. 1, which is a structural schematic diagram of an action simulation model provided by an embodiment of the application, as shown in FIG. 2, the action simulation model comprises an input layer, a linear projection layer, a linear projection layer and an ELU activation function layer, and an output layer. Figure 3 The input layer is used to receive the input data a of the action simulation model and convert it into a feature vector that can be processed by the action simulation model.

[0136] The linear projection layer and the ELU activation function layer jointly constitute a nonlinear transformation unit containing an activation function. The linear projection layer is responsible for dimension mapping, and realizes linear transformation through matrix multiplication and bias to ensure that the dimensions are aligned when the residual connection is connected. The ELU activation function is used to introduce nonlinear characteristics, so that the action simulation model can learn more complex action mapping relationships.

[0137] The output layer is used to output the simulation result of the action simulation model. For example, the simulation result of the action simulation model can be a simulated action b.

[0138] The "+" represents a residual connection, which is set before the output layer, can alleviate the gradient disappearance problem of the deep network, ensure that the gradient can be effectively returned, make the action simulation model converge faster, reduce the number of training times, increase the training efficiency of the action simulation model, and at the same time, retain the core information of the original input action features and improve the action simulation accuracy.

[0139]

[0140] ​Specifically, the input data a is input to the input layer, which converts it into a feature vector; the feature vector sequentially passes through a nonlinear transformation unit composed of 3 linear projection layers and ELU activation function layers, and each nonlinear transformation unit performs nonlinear transformation on the feature vector, which can gradually extract deep features of the action. At the same time, the target linear projection layer maps the feature vector of the input layer to a space matching the dimension of the output layer, and is connected in residual with the 3 linear projection layers and ELU activation function layers, enters the last nonlinear transformation unit, and preliminarily generates a predicted simulated action, and outputs the simulated action b through the output layer. After training, the hidden space vector of the new action and the action simulation model of the new action are added to the model library, and the next action is processed, until each action data in the action data set is completed.

[0141] Figure 4 Fig. 1 shows a flowchart of a method for training an action simulation model according to an embodiment of the present application. As shown in Fig. 1, the method for training the action simulation model includes three stages. Figure 4

[0142] The first stage is the training stage of the variational autoencoder.

[0143] Specifically, based on a large-scale public action data set (such as the AMASS, LAFAN1 data set), the human action sequence in SMPL format in the large-scale public action data set is redirected to robot-specific action data (action 1, action 2, action 3, action 4, etc.), including global translation and joint axis angle rotation. Subsequently, the zero moment point (ZMP) is determined based on the Newton-Euler simplified formula (combined with mass, center of mass position, acceleration and torque), and the ZMP is filtered in combination with the support polygon to exclude unstable or infeasible action data. The filtered data emphasizes leg features (such as knee and ankle joint positions and velocities), and ensures that the trajectory of the action data (action 1, action 2, etc.) is suitable for the robot to perform.

[0144] The filtering conditions include a distance threshold (for example, 0.1 meters) and a duration threshold (for example, 0.5 seconds) that the ZMP exceeds the support polygon.

[0145] Further, the variational autoencoder extracts the hidden space vector of the action based on the filtered action data set, and realizes the construction of the action feature distribution. The variational autoencoder includes an encoder and a decoder, and focuses on self-supervised learning of kinematic features. The encoder adopts a Transformer structure, which is used to map the action sequence into a hidden variable Z m . The decoder is also a Transformer structure, which is used to reconstruct the action sequence m of the key joints (for example, head, pelvis, hand and foot) based on z . ​

[0146] The second stage is a rolling training stage.

[0147] Specifically, the rolling training process mainly realizes efficient training of each action of a large-scale action dataset through a latent space retrieval and model initialization mechanism. Specifically, starting from the first action sequence of the dataset, each new action is processed step by step, and the similarity matching is performed using the latent space representation library generated by the variational autoencoder, so as to reuse the action simulation model of the existing action and accelerate the convergence. The training adopts a reinforcement learning framework, and combines an improved neural network structure and parameter setting to ensure the fidelity and robustness of the action simulation.

[0148] The rolling training stage starts from the first action of the action dataset and processes each new action sequence one by one. First, the new action sequence is preprocessed into a standardized format, including the Cartesian space position of the joint, the root translation and velocity, the foot contact state, and the foot velocity, which can highlight the leg-related features to capture the dynamic differences of different motion types (such as jumping or crouching). Then, the preprocessed sequence is input into the encoder of the variational autoencoder to obtain its latent space vector Z _new . The encoder is based on the Transformer structure and can map the action sequence into a low-dimensional latent representation, which can reflect the kinematic essence.

[0149] Further, the Euclidean distance between the latent space vector of the new action (new latent space vector) and the latent space vectors of the existing actions (latent space vector 1, latent space vector 2, latent space vector 3, latent space vector 4, latent space vector 5) is calculated in the latent space representation library, and the closest latent space vector Z _existing (latent space vector 5) of the existing action is found. If the distance between the new latent space vector and the closest latent space vector of the existing action is less than a predetermined threshold, the trained action simulation model corresponding to the latent space vector (latent space vector 5) of the existing action is loaded as the model initialization of the new action; otherwise, the training starts from the base model or random initialization. This similarity retrieval mechanism can ensure knowledge reuse and avoid training similar actions from scratch, thereby significantly accelerating the process.

[0150] The third stage is the training of the action simulation model of the new action.

[0151] Specifically, state space data is acquired, including current posture, velocity, root torsion (represented in the root coordinate system), and reference motion phase information, to provide balance guidance and drift correction; motion space data is acquired, including joint impedance setpoint information, to generate desired torque by normalizing joint position setpoints, promoting shock absorption and noise robustness; and a reward function is acquired, including posture tracking reward, velocity matching reward, collision penalty, and energy penalty, which encourages efficient imitation and avoids excessive torque. Based on the aforementioned state space data, motion space data, and reward function, a motion simulation model for the new motion is trained.

[0152] In this embodiment of the application, in order to improve robustness, an adaptive sampling mechanism is introduced, which divides the action sequence into fixed-length intervals and samples them with weighted average based on the historical failure rate, giving priority to difficult segments; at the same time, randomization parameters are set, including the ground friction coefficient, the default joint position offset, and the torso center of mass position perturbation, to simulate real-world variability.

[0153] The technical solution of this application, on the one hand, transforms motion simulation into incremental learning, which differs from existing model training methods (such as expert models based on fixed clustering or diffusion models trained in full batches). It performs similarity retrieval in the model library based on the VAE latent space, reusing existing motion simulation models and avoiding repeated training of similar actions from scratch. On the other hand, it filters data based on ZMP (zero torque point) and support polygons to ensure dataset quality; it emphasizes VAE encoding of leg kinematic features to capture torque control differences; and it improves the reinforcement learning network structure through linear projection and residual connections to enhance gradient flow. The technical solution of this application supports real-time extraction of motion features from motion data, is suitable for robot control in dynamic environments, achieves rapid imitation, and improves training efficiency and generalization ability.

[0154] The above text combined Figures 1 to 4 The method embodiments of this application are described in detail below, in conjunction with... Figure 5 The present application provides a detailed description of the apparatus embodiments. Furthermore, it should be understood that the descriptions of the method embodiments correspond to the descriptions of the apparatus embodiments; therefore, any parts not described in detail can be found in the foregoing method embodiments.

[0155] Figure 5 The diagram shown is a structural schematic of a training device for a motion simulation model provided in an embodiment of this application. Figure 5As shown, the training device of the action simulation model provided in the embodiments of the present application comprises: a first obtaining module 501 configured to obtain an action feature corresponding to a target action, to obtain a target action feature; a determining module 502 configured to determine a known action matched with the target action according to the target action feature, to obtain a target known action; a second obtaining module 503 configured to obtain an action simulation model corresponding to the target known action from a model library, to obtain a target action simulation model, the model library comprising at least one trained action simulation model corresponding to a known action; and a model training module 504 configured to perform model training on the target action simulation model based on the target action, to obtain an action simulation model corresponding to the target action.

[0156] In an embodiment of the present application, the determining module 502 is specifically configured to: map the target action feature to a latent space to obtain first latent space representation information corresponding to the target action; query second latent space representation information matched with the first latent space representation information from a latent space representation library, the latent space representation library comprising at least one latent space representation information corresponding to a known action; and in the case that the second latent space representation information exists in the latent space representation library, determine the known action corresponding to the second latent space representation information as the target known action.

[0157] In an embodiment of the present application, the determining module 502 is specifically configured to: map the target action feature to a latent space by using the trained variational autoencoder, to obtain first latent space representation information corresponding to the target action.

[0158] In an embodiment of the present application, the determining module 502 is specifically configured to: obtain an action feature corresponding to an action sample based on a preset action data set, to obtain a first action feature; input the first action feature into an encoder in the variational autoencoder, and map the first action feature to a latent space by using the encoder, to obtain third latent space representation information corresponding to the action sample; input the third latent space representation information into a decoder in the variational autoencoder, and perform feature reconstruction on the third latent space representation information by using the decoder, to obtain a second action feature; train the variational autoencoder according to the first action feature and the second action feature, to obtain a trained variational autoencoder; and map the target action feature to the latent space by using the trained variational autoencoder, to obtain the first latent space representation information corresponding to the target action, comprising: inputting the target action feature into the encoder in the trained variational autoencoder, and mapping the target action feature to the latent space by using the encoder, to obtain the first latent space representation information corresponding to the target action.

[0159] In an embodiment of the present application, the determining module 502 is specifically configured to: based on the preset action data set, obtain device action data corresponding to each of the plurality of action samples; determine position information of the zero moment point corresponding to each of the action samples according to the device action data corresponding to each of the action samples; determine a position relationship between the zero moment point corresponding to each of the action samples and the preset support polygon according to the position information; filter the device action data corresponding to each of the plurality of action samples based on the position relationship, to obtain device action data corresponding to filtered action samples; and obtain action features corresponding to the filtered action samples based on the device action data corresponding to the filtered action samples, to obtain the first action features.

[0160] In an embodiment of the present application, the determining module 502 is specifically configured to: in a case where the position relationship corresponding to a target action sample meets a first preset condition, delete the device action data corresponding to the target action sample, the target action sample including any action sample in the plurality of action samples; and wherein the first preset condition includes that the zero moment point corresponding to the target action sample is outside the boundary of the preset support polygon, and the distance between the zero moment point and the boundary of the preset support polygon is greater than a preset distance threshold.

[0161] In an embodiment of the present application, the determining module 502 is specifically configured to: obtain a duration when the position relationship corresponding to a target action sample meets a first preset condition; and in a case where the duration is greater than a preset duration threshold, delete the device action data corresponding to the target action sample.

[0162] In an embodiment of the present application, the determining module 502 is specifically configured to: determine, from the latent space representation library, a latent space representation information most similar to the first latent space representation information, to obtain target latent space representation information; and in a case where the similarity between the first latent space representation information and the target latent space representation information is greater than a preset similarity threshold, determine that the target latent space representation information is the second latent space representation information.

[0163] In an embodiment of the present application, the model training module 504 is specifically configured to: obtain model training element information corresponding to a target action; and based on the model training element information, perform model training on the target action simulation model based on a Markov decision process, to obtain an action simulation model corresponding to the target action; wherein the action simulation model includes an input layer, an output layer and a target linear projection layer, and the input layer and the output layer are connected through a target linear projection layer residual.

[0164] In an embodiment of the present application, the model training module 504 is specifically configured to: obtain privileged observation information corresponding to a target action; and perform error tracking according to the privileged observation information in a model training process.

[0165] In the following, reference is made to Figure 6An electronic device according to an embodiment of the present application is described. Figure 6 FIG. 1 shows a structural diagram of an electronic device according to an exemplary embodiment of the present application.

[0166] As shown in FIG. 1, the electronic device 60 includes one or more processors 601 and a memory 602. Figure 6

[0167] The processor 601 can be a central processing unit (CPU) or other form of processing unit that has data processing capability and / or instruction executing capability, and can control other components in the electronic device 60 to perform desired functions.

[0168] The memory 602 can include one or more computer program products that can include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. For example, the volatile memory can include random access memory (RAM), cache, and / or the like. The non-volatile memory can include read-only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program instructions can be stored on the computer-readable storage media, and the processor 601 can execute the program instructions to implement the development machine access method of various embodiments of the present application described above and / or other desired functions. Various contents such as the port allocation algorithm, the load balancer resource instance, the development machine resource instance, and the like can also be stored in the computer-readable storage media.

[0169] In one example, the electronic device 60 can further include an input device 603 and an output device 604, which are interconnected through a bus system and / or other form of connection mechanism (not shown).

[0170] The input device 603 can include, for example, a keyboard, a mouse, and / or the like.

[0171] The output device 604 can output various information including the port allocation algorithm, the load balancer resource instance, the development machine resource instance, and the like to the outside. The output device 604 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and / or the like.

[0172] Of course, in order to simplify, Figure 6 In FIG. 1, only some of the components related to the present application among the electronic device 60 are shown, and components such as a bus, an input / output interface, and the like are omitted. In addition, the electronic device 60 can include any other appropriate components according to a specific application.

[0173] ​In addition to the methods and devices described above, embodiments of the present application can also be a computer program product that includes computer program instructions that, when run by a processor, cause the processor to perform the steps of the development machine access method described above according to various embodiments of the present application.

[0174] The computer program product can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, etc., and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote computing device or server.

[0175] In addition, embodiments of the present application can also be a computer readable storage medium having stored thereon computer program instructions that, when run by a processor, cause the processor to perform the steps of the development machine access method described above according to various embodiments of the present application.

[0176] The computer readable storage medium can be any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium can include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples (a non-exhaustive list) of the computer readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0177] The above describes the basic principles of the present application in combination with specific embodiments, but it should be noted that the advantages, advantages, effects, etc. mentioned in the present application are only examples and are not limiting, and these advantages, advantages, effects, etc. cannot be considered as the must-have of each embodiment of the present application. In addition, the above specific details are only for the purpose of example and for the purpose of understanding, and are not limiting, and the above details do not limit the present application to the must-use of the above specific details to realize.

[0178] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0179] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0180] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0181] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A training method for an action simulation model, characterized in that, include: Obtain the action features corresponding to the target action to obtain the target action features; Based on the target action characteristics, a known action matching the target action is determined, thus obtaining the target known action; Obtain a motion simulation model corresponding to the known action of the target from the model library to obtain the target motion simulation model. The model library includes at least one trained motion simulation model corresponding to the known action. The target action simulation model is trained based on the target action to obtain the action simulation model corresponding to the target action.

2. The method according to claim 1, characterized in that, The step of determining a known action that matches the target action based on the target action characteristics to obtain the target known action includes: The target action features are mapped to the latent space to obtain the first latent space representation information corresponding to the target action; Query the latent space representation library to find the second latent space representation information that matches the first latent space representation information, wherein the latent space representation library includes at least one latent space representation information corresponding to a known action; If the second latent space representation information exists in the latent space representation library, the known action corresponding to the second latent space representation information is determined to be the target known action.

3. The method according to claim 2, characterized in that, The step of mapping the target action features to the latent space to obtain the first latent space representation information corresponding to the target action includes: The target action features are mapped to the latent space using a trained variational autoencoder to obtain the first latent space representation information corresponding to the target action.

4. The method according to claim 3, characterized in that, Before mapping the target action features to the latent space using a trained variational autoencoder to obtain the first latent space representation information corresponding to the target action, the method further includes: Based on a pre-set action dataset, action features corresponding to action samples are obtained to obtain the first action feature; The first action feature is input into the encoder in the variational autoencoder, and the encoder is used to map the first action feature to the latent space to obtain the third latent space representation information corresponding to the action sample. The third latent space representation information is input into the decoder in the variational autoencoder, and the decoder is used to reconstruct the features of the third latent space representation information to obtain the second action feature. The variational autoencoder is trained based on the first action feature and the second action feature to obtain a trained variational autoencoder. The step of mapping the target action features to a latent space using a trained variational autoencoder to obtain the first latent space representation information corresponding to the target action includes: The target action features are input into the encoder in the trained variational autoencoder, and the encoder is used to map the target action features to the latent space to obtain the first latent space representation information corresponding to the target action.

5. The method according to claim 4, characterized in that, The first action feature is obtained by acquiring action features corresponding to action samples based on a preset action dataset, including: Based on a preset action dataset, obtain device action data corresponding to multiple action samples; Based on the equipment motion data corresponding to each motion sample, determine the position information of the zero torque point corresponding to each motion sample; Based on the position information, determine the positional relationship between the zero-moment point corresponding to each action sample and the preset support polygon; Based on the positional relationship, the device action data corresponding to the multiple action samples are filtered to obtain the device action data corresponding to the filtered action samples. Based on the device action data corresponding to the filtered action samples, the action features corresponding to the filtered action samples are obtained to obtain the first action feature.

6. The method according to claim 5, characterized in that, The step of filtering the device action data corresponding to the multiple action samples based on the positional relationship to obtain the device action data corresponding to the filtered action samples includes: If the positional relationship corresponding to the target action sample meets the first preset condition, delete the device action data corresponding to the target action sample, wherein the target action sample includes any action sample among the plurality of action samples; The first preset condition includes that the zero-moment point corresponding to the target action sample is outside the boundary of the preset support polygon, and the distance between the zero-moment point and the boundary of the preset support polygon is greater than a preset distance threshold.

7. The method according to claim 6, characterized in that, Deleting the device action data corresponding to the target action sample includes: The duration of the positional relationship corresponding to the target action sample when it satisfies the first preset condition is obtained; If the duration exceeds a preset duration threshold, delete the device action data corresponding to the target action sample.

8. The method according to claim 2, characterized in that, The step of querying the latent space representation database for second latent space representation information that matches the first latent space representation information includes: The target latent space representation information is obtained by determining the latent space representation information that is most similar to the first latent space representation information from the latent space representation library. If the similarity between the first latent space representation information and the target latent space representation information is greater than a preset similarity threshold, the target latent space representation information is determined to be the second latent space representation information.

9. The method according to any one of claims 1 to 8, characterized in that, The step of training the target action simulation model based on the target action to obtain an action simulation model corresponding to the target action includes: Obtain model training element information corresponding to the target action; Based on the model training element information, the target action simulation model is trained using a Markov decision process to obtain an action simulation model corresponding to the target action. The motion simulation model includes an input layer, an output layer, and a target linear projection layer, with the input layer and the output layer connected by a residual link through the target linear projection layer.

10. The method according to claim 9, characterized in that, Before obtaining the motion simulation model corresponding to the target motion, the method further includes: Obtain privileged observation information corresponding to the target action; Error tracking is performed based on the privileged observation information during model training.

11. A computer-readable storage medium, characterized in that, The storage medium stores a computer program for executing the training method of the motion simulation model according to any one of claims 1 to 10.

12. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is configured to execute the training method for the action simulation model according to any one of claims 1 to 10.

13. A computer program product, characterized in that, The computer program product includes instructions that, when executed on an electronic device, cause the electronic device to implement the training method for the motion simulation model according to any one of claims 1 to 10.

Images