A mechanical control device state representation learning method with transfer distance metric, electronic device and storage medium

Abstract

A state characterization learning method, electronic device and storage medium of a mechanical control device with transfer distance metric, belonging to the field of deep reinforcement learning and intelligent control technology. In order to accurately measure the contrast characteristics of the transfer distance between states, the present invention sets the motion environment as a Markov decision process, collects the motion trajectory information data of the mechanical control device, performs preprocessing based on the contrast feature learning method, processes it into positive motion trajectory information data and negative motion trajectory information data, and uses the motion trajectory information data, positive motion trajectory information data and negative motion trajectory information data to construct a motion trajectory information training data set; constructs a deep neural network as a TDRP encoder, sets a loss function and an optimizer, inputs the motion trajectory information training data set into the TDRP encoder for training, and obtains a TDRP encoder with transfer distance metric; sets reinforcement learning tasks based on different application environments, and constructs auxiliary rewards.

Description

Technical Field

[0001] The present invention belongs to the technical field of deep reinforcement learning and intelligent control, and specifically relates to a state representation learning method for a mechanical control device with transfer distance metric, an electronic device and a storage medium. Background Art

[0002] Deep reinforcement learning and intelligent control technology are used to study state characterization methods, which mainly include three methods: reconstruction-based feature learning, mutual simulation and contrast feature learning.

[0003] Reconstruction-based feature learning: Earlier image-based deep reinforcement learning generally uses a two-stage learning process, in which the first stage uses the reconstruction error to learn an autoencoder that can compress state information into low-dimensional features, and then the second stage uses this feature to learn a strategy to complete the specified task; this can use a large amount of unlabeled data to learn an encoder in order to obtain a better strategy. However, in actual situations, this approach does not guarantee that the features obtained can extract elements in the state information that are conducive to task solving, and professional knowledge and skills in the task domain are required to use this method effectively. In model-based reinforcement learning, this problem is solved by training the encoder and dynamic model end-to-end at the same time, which also proves that this method can obtain effective task-oriented features. Use reconstruction error to learn a hidden state model, but these methods still have difficulty accurately predicting returns and trajectories over long periods of time, and usually require fine manual adjustment.

[0004] The mutual simulation feature learning method divides the state space into different clusters. When the reward sequences of two states under different action sequences are difficult to distinguish, they will be classified into the same cluster and represented by a feature. The mutual simulation criterion can be used to measure whether the two states behave similarly. A mutual simulation criterion for continuous space MDP proposes a Monte-Carlo algorithm to learn strategies using the bulldozer distance between historical trajectory distributions.

[0005] Contrastive feature learning is a self-supervised learning method that distinguishes positive and negative data by constraining the similarity between data. It is expected that the feature distance of positive data is as close as possible and the feature distance of negative data is as far as possible, so as to learn the characteristics of state information. The similarity measurement can be constructed based on the information of the task domain. For example, data enhancement can be used to maximize the similarity between different data enhanced by a data point. [9] , or maximizing the similarity between adjacent patches while minimizing the similarity between different data points; contrastive feature learning is also used in prediction tasks.

[0006] In the actual application of the above methods, the reconstruction-based feature learning cannot guarantee that the obtained features can extract elements in the state information that are conducive to task solving. At the same time, professional knowledge and skills in the task field are required to effectively use this method, and usually fine manual adjustment is required. The mutual simulation feature learning method is difficult to apply to problems with large state spaces. So far, many successful feature learning methods have been applied to reinforcement learning, but there is still no feature that can measure the "transition distance" between two states. Summary of the invention

[0007] The problem to be solved by the present invention is to accurately measure the contrast characteristics of the transfer distance between states, and propose a mechanical control device state representation learning method, electronic device and storage medium with transfer distance measurement.

[0008] To achieve the above object, the present invention is implemented through the following technical solutions:

[0009] A method for learning state representation of a mechanical control device with transfer distance metric, comprising the following steps:

[0010] S1. Set the motion environment to a Markov decision process and collect motion trajectory information data of the mechanical control device;

[0011] S2. Preprocessing the motion trajectory information data collected in step S1 based on the contrast feature learning method to process it into motion trajectory information positive data and motion trajectory information negative data, and constructing a motion trajectory information training data set using the motion trajectory information data, the motion trajectory information positive data and the motion trajectory information negative data;

[0012] S3. Construct a deep neural network as the TDRP encoder;

[0013] S4. Setting a loss function and an optimizer for the TDRP encoder constructed in step S3, inputting the motion trajectory information training data set obtained in step S2 into the TDRP encoder constructed in step S3, and performing training to obtain a TDRP encoder having a transfer distance metric;

[0014] S5. For the TDRP encoder with transfer distance metric obtained in step S4, reinforcement learning tasks are set based on different application environments, auxiliary rewards are constructed, and then the auxiliary rewards are combined with the environmental rewards of the reinforcement learning tasks to generate the final internal rewards. The internal rewards are used to train the reinforcement learning tasks, and the training results are repeated in steps S1 to S5 to finally obtain a mechanical control device state representation learning method with transfer distance metric.

[0015] Furthermore, the specific implementation method of step S1 includes the following steps:

[0016] S1.1. Set the motion environment as a Markov decision process, represented by M = (S, A, P, R, y), where S is the state space set, A is the action space set, P is the transition probability set, R is the reward value set, and y∈(0,1) is the discount coefficient of the reward function;

[0017] S1.2 Set p = (s'|s, a) as the state transition probability from the first state s∈S to the updated state s'∈S. The mechanical control device determines the action a∈A to be performed in the state s according to the strategy π: a~π(s), and updates the state s'~P(s, a). At the same time, it obtains the reward value r = R(s, a), and uses τ i ={(s 0 ,a 0 )(s 1 ,a 1 )...(s T ,a T )} represents the i-th trajectory with a length of T, and the trajectory set is τ i ∈τ.

[0018] Furthermore, step S2 pre-processes the motion trajectory information data collected in step S1 based on the contrast feature learning method, and converts each state data s in the historical trajectory τ into t ∈S, respectively and Represents the positive and negative data sets of motion trajectory information, and the expression is:

[0019]

[0020]

[0021] Among them, step is the set step size, which is used to control the size of the positive and negative example sets.

[0022] Furthermore, step S3 builds a deep neural network as a TDRP encoder, the deep neural network includes an input layer, 3 hidden layers and an output layer, each layer of the network is implemented using a fully connected network, and a RELU network layer is included between every two networks as an excitation layer;

[0023] Based on the state data being the image data, two convolutional network layers are added before the input layer to realize that the input of the TDRP encoder network is the state data, and the output is the eigenvalue of the state data. θ represents the TDRP encoder network, where θ is the TDRP encoder network parameter.

[0024] Furthermore, the specific implementation method of step S4 includes the following steps:

[0025] S4.1. Set the loss function Loss for the TDRP encoder constructed in step S3 to train and update the TDRP encoder. The calculation expression is:

[0026]

[0027] in, For expectations;

[0028] S4.2. Input the motion trajectory information training data set obtained in step S2 into the TDRP encoder network for end-to-end training update, calculate the error using the loss function constructed in step S4.1, and calculate the gradient through the back propagation algorithm. The calculation expression is:

[0029]

[0030] Among them, ε(t) is a cross-order function:

[0031] S4.3. Set ADAM as the optimizer for the TDRP encoder constructed in step S3, and use the optimizer to update the parameters of the TDRP encoder network to optimize and update the network. The calculation expression for the optimization update of the tth round is:

[0032]

[0033] Among them, m t is the first-order momentum at step t, β 1 is the hyperparameter of mt;

[0034]

[0035] Among them, v t is the first-order momentum of the t-th step, β 2 v t Hyperparameters of

[0036]

[0037]

[0038]

[0039] Among them, η,∈ is a hyperparameter.

[0040] Furthermore, the specific implementation method of step S5 includes the following steps:

[0041] S5.1. Set the application environment to a single reinforcement learning task, then reconstruct the reward function of the single reinforcement learning task, and the calculation expression is:

[0042] r(s,a,sg )=R(s,a)-||φ(s)-φ(s g )|| 2 ;

[0043] S5.2. Set the application environment to the skill connection reinforcement learning task, and set the termination state of the previous skill to To the initial state set of the next technique The TDRP encoder φ with transfer distance metric obtained in step S4 is used for encoding, and then the eigenvalues ​​are aggregated into n state clusters, and the center point of the kth feature state cluster is set to Encode the state of the previous skill, and calculate the minimum distance between it and the center point of the n state clusters as part of the reward to form a new reward function. The calculation expression is:

[0044]

[0045] The reward function is used to fine-tune the strategy of the previous skill, hoping that the state of this skill can approach the initial state of the next skill, thus achieving the connection between the two skills.

[0046] The electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a mechanical control device state representation learning method with transfer distance metric when executing the computer program.

[0047] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the state representation learning method of a mechanical control device with transfer distance metric.

[0048] Beneficial effects of the present invention:

[0049] The mechanical control device state representation learning method with transfer distance metric described in the present invention can be found that for a single reinforcement learning task, in two tasks, compared with the results of training using the original environment rewards, the training using the reward function reconstructed by the TDRP encoder of the present invention has higher efficiency and can converge to the optimal effect faster. It is also more stable. For the grasping task, the strategy trained using the reward function reconstructed by the TDRP encoder has better results when converging, and the success rate is much higher than the strategy trained by the original environment rewards.

[0050] The mechanical control device state representation learning method with transfer distance metric described in the present invention is applied to skill splicing tasks, which can enhance the splicing effect and improve the success rate of the overall task after splicing. The grasping task and the placement task are spliced, and the mechanical control device is expected to complete the coherent task of picking up the nut from the desktop and then placing it on the bolt. The initial states of several successful trajectories of the placement task (the latter task) are extracted, encoded using the TDRP encoder to form a feature set, the feature set is clustered, and the center point of each cluster is obtained as the target feature set of the grasping task (the former task). The reward function of the grasping task is reconstructed using the TDRP encoder and the target feature set, and the grasping task is fine-tuned using this reconstructed reward, and the success rate gradually increases.

[0051] The mechanical control device state representation learning method with transfer distance metric described in the present invention can extract the elements in the state that are most relevant to task execution so as to measure the "transfer distance" between states. It adopts an end-to-end training method, does not require manual professional knowledge and skills in the task field, and can be applied to tasks with a larger state space. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A flowchart of a method for learning a state representation of a mechanical control device with transfer distance metric according to the present invention;

[0053] Figure 2 A schematic diagram of a simulation environment for a single reinforcement learning task of the present invention;

[0054] Figure 3 A schematic diagram of a simulation environment for a skill-connected reinforcement learning task of the present invention;

[0055] Figure 4 This is the comparison curve of the grabbing task of the present invention;

[0056] Figure 5 Place the task comparison curve for the present invention;

[0057] Figure 6 It is the success rate curve of the number of iterations of the present invention;

[0058] Figure 7 To visualize the original state and feature space of the grasping task;

[0059] In the figure, (a) is the original state, (b) is the state with a step of 10, (c) is the state with a step of 30, (d) is the state with a step of 50, and (e) is the state with a step of 80.

[0060] Figure 8 Place the original state and feature space of the task for visualization;

[0061] In the figure, (a) is the original state, (b) is the state with a step of 10, (c) is the state with a step of 30, (d) is the state with a step of 50, and (e) is the state with a step of 80. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present invention more clear, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention, that is, the specific embodiments described are only part of the embodiments of the present invention, rather than all of the specific embodiments. The components of the specific embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations, and the present invention can also have other embodiments.

[0063] Therefore, the following detailed description of the specific embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents the selected specific embodiments of the present invention. Based on the specific embodiments of the present invention, all other specific embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0064] In order to further understand the content, features and effects of the present invention, the following specific implementation methods are given as examples, and the attached Figure 1 -Attached Figure 6 The detailed instructions are as follows: Specific implementation method one:

[0066] A method for learning state representation of a mechanical control device with transfer distance metric, comprising the following steps:

[0067] S1. Set the motion environment to a Markov decision process and collect motion trajectory information data of the mechanical control device;

[0068] Furthermore, the specific implementation method of step S1 includes the following steps:

[0069] S1.1. Set the motion environment as a Markov decision process, represented by M = (S, A, P, R, y), where S is the state space set, A is the action space set, P is the transition probability set, R is the reward value set, and y∈(0,1) is the discount coefficient of the reward function;

[0070] S1.2 Set p = (s'|s, a) as the state transition probability from the first state s∈S to the updated state s'∈s. The mechanical control device determines the action a∈A to be performed in the state s according to the strategy π: a~π(s), and updates the state s'~P(s, a). At the same time, it obtains the reward value r = R(s, a), and uses τ i ={(s 0 ,a 0 )(s 1 ,a 1 )...(s T ,a T )} represents the i-th trajectory with a length of T, and the trajectory set is τ i ∈τ;

[0071] Furthermore, the trajectory information generated by online reinforcement learning training or offline reinforcement learning training can be obtained for the training of the TDRP encoder. For online reinforcement learning training, during the reinforcement learning training, the intelligent body will continuously interact with the environment, and will continuously generate trajectory information τ during the interaction process. These trajectory information are collected for the training of the TDRP encoder; for offline reinforcement learning training, the offline historical trajectory dataset τ is collected as the training data of the TDRP encoder;

[0072] S2. Preprocessing the motion trajectory information data collected in step S1 based on the contrast feature learning method to process it into motion trajectory information positive data and motion trajectory information negative data, and constructing a motion trajectory information training data set using the motion trajectory information data, the motion trajectory information positive data and the motion trajectory information negative data;

[0073] Furthermore, step S2 pre-processes the motion trajectory information data collected in step S1 based on the contrast feature learning method, and converts each state data s in the historical trajectory τ into t ∈S, respectively and Represents the positive and negative data sets of motion trajectory information, and the expression is:

[0074]

[0075]

[0076] Among them, step is the set step size, which is used to control the size of the positive and negative example sets;

[0077] Furthermore, for example, for a history track {(s 0 ,a 0 )(s 1 ,a 1 )...(s100 ,a 100 )}, we can set step to 20, then for each state of 60≥t≥0, we can construct its positive example set and negative example set The larger the set of positive and negative examples in each state of the natural step, the smaller it is.

[0078] S3. Construct a deep neural network as the TDRP encoder;

[0079] Furthermore, in step S3, a deep neural network is constructed as a TDRP encoder. The deep neural network includes an input layer, three hidden layers, and an output layer. Each network layer is implemented using a fully connected network. A RELU network layer is included between every two networks as an excitation layer.

[0080] Based on the state data being the image data, two convolutional network layers are added before the input layer to realize that the input of the TDRP encoder network is the state data, and the output is the eigenvalue of the state data. θ represents the TDRP encoder network, where θ is the TDRP encoder network parameter;

[0081] S4. Setting a loss function and an optimizer for the TDRP encoder constructed in step S3, inputting the motion trajectory information training data set obtained in step S2 into the TDRP encoder constructed in step S3, and performing training to obtain a TDRP encoder having a transfer distance metric;

[0082] Furthermore, the specific implementation method of step S4 includes the following steps:

[0083] S4.1. Set the loss function Loss for the TDRP encoder constructed in step S3 to train and update the TDRP encoder. The calculation expression is:

[0084]

[0085] in, For expectations;

[0086] S4.2. Input the motion trajectory information training data set obtained in step S2 into the TDRP encoder network for end-to-end training update, calculate the error using the loss function constructed in step S4.1, and calculate the gradient through the back propagation algorithm. The calculation expression is:

[0087]

[0088] Among them, ε(t) is a cross-order function:

[0089] S4.3. Set ADAM as the optimizer for the TDRP encoder constructed in step S3, and use the optimizer to update the parameters of the TDRP encoder network to optimize and update the network. The calculation expression for the optimization update of the tth round is:

[0090]

[0091] Among them, m t is the first-order momentum of the tth step, m 0 =0,β 1 is m t Hyperparameters of

[0092]

[0093] Among them, v t is the second-order momentum at step t, v 0 =0,β 2 v t Hyperparameters of

[0094]

[0095]

[0096]

[0097] Among them, η,∈ is a hyperparameter;

[0098] Furthermore, batch training is used during training, and each batch contains multiple states and their positive and negative state sets. A batch of data is used each time the network parameters are updated. During the network training process, the model's hyperparameters are adjusted according to the network's performance on the validation set, including the learning rate, the number of neural network layers, the number of neural network units, the auxiliary reward coefficient, etc., and the TDRP encoder is retrained until the error converges. Finally, a trained encoder that can measure the "transfer distance" is obtained: the TDRP encoder;

[0099] S5. For the TDRP encoder with transfer distance metric obtained in step S4, reinforcement learning tasks are set based on different application environments, auxiliary rewards are constructed, and then the auxiliary rewards are combined with the environmental rewards of the reinforcement learning tasks to generate the final internal rewards. The internal rewards are used to train the reinforcement learning tasks, and the training results are repeated in steps S1 to S5 to finally obtain a mechanical control device state representation learning method with transfer distance metric.

[0100] Furthermore, the specific implementation method of step S5 includes the following steps:

[0101] S5.1. Set the application environment to a single reinforcement learning task, then reconstruct the reward function of the single reinforcement learning task, and the calculation expression is:

[0102] r(s,a,s g )=R(s,a)-||φ(s)-φ(s g )|| 2 ;

[0103] S5.2. Set the application environment to the skill connection reinforcement learning task, and set the termination state of the previous skill to To the initial state set of the next technique The TDRP encoder φ with transfer distance metric obtained in step S4 is used for encoding, and then the eigenvalues ​​are aggregated into n state clusters, and the center point of the kth feature state cluster is set to Encode the state of the previous skill, and calculate the minimum distance between it and the center point of the n state clusters as part of the reward to form a new reward function. The calculation expression is:

[0104]

[0105] The reward function is used to fine-tune the strategy of the previous skill, hoping that the state of this skill can approach the initial state of the next skill, thus achieving the connection between the two skills.

[0106] Furthermore, the trained TDRP encoder is used to construct auxiliary rewards for reinforcement learning tasks. The auxiliary rewards are combined with the environmental rewards of the reinforcement learning tasks to generate the final internal rewards, and this internal reward is used to help train the reinforcement learning tasks. At the same time, during the reinforcement learning task training, the agent constantly interacts with the environment. In each state s, the action a to be performed is determined based on the strategy and the current state, which will generate multiple trajectory information containing states and actions. These trajectory information data can be used to continue training the TDRP encoder.

[0107] like Figure 2 and Figure 3 The experiment includes two simulation environments: the nut grabbing task and the nut placing task. Both simulation environments contain a complex mechanical control device. The nut grabbing task requires controlling the mechanical control device to grab the nut that appears at a random position on the table and lift it into the air, away from the table. Each round of the task includes 100 state transition sequences; the nut placing task requires controlling the mechanical control device to move and place the nut in the claw of the mechanical control device on top of the bolt. Each round of the task includes 200 state transition sequences. Both simulation environments are based on Factory

[12] The build consists of Franka mechanical controls, a tabletop and nuts and bolts, all simulated with high precision and rich contacts, requiring very precise operation to complete the task.

[0108] 1. Analyze the encoder

[0109] The encoder constructed and trained using the method of this embodiment is analyzed, and it is verified that the method of this embodiment can measure the "state transition distance", that is, the Euclidean distance between two state features can measure the number of states required for the transition between two states. The trajectories of the grasping task and the placing task are used to set different steps to train the TDRP encoders of these two tasks. Then, for these two tasks, 30 trajectories are randomly collected, and the original states in these trajectories and the features encoded by each encoder are visualized, such as Figure 7 and Figure 8 As shown in the figure, the original state and feature space of the grasping task and the placing task are visualized respectively. In each figure, the darker point indicates that the point is at the initial stage of the trajectory, and the lighter the point is, the later its position in the trajectory. The yellow point in the figure indicates the initial state cluster center point of the latter task.

[0110] In the original state space, the distance between any two states has no practical meaning; however, in the feature space formed by the TDRP encoder proposed in this embodiment, the features of each state are arranged in order according to their positions in the trajectory, which proves that the Euclidean distance between any two features can measure their distance in the trajectory, that is, the number of intermediate transition states that one state needs to go through to reach another state. The TDRP encoders with four different hyperparameters of step 10, 30, 50 and 80 were trained respectively, and the feature space they formed is shown in the figure below. It can be seen that the larger the step, the closer the shape of the features on each trajectory in the feature space is to a straight line according to their order, and naturally the "transfer distance" expressed by the Euclidean distance between the two features is more accurate: the greater the Euclidean distance between the two features, the farther they are apart in a trajectory, and the more actions and intermediate transition states are required to reach another state. At the same time, it can be seen that for tasks with shorter state transition sequences (grasping tasks), a smaller step (step = 30) can make the shape of the feature sequence arrangement on the same trajectory approach a straight line; but for tasks with longer state transition sequences (placing tasks), a larger step is required to reduce the curvature of the feature arrangement shape on a trajectory and make it approach the shape of a straight line.

[0111] 2. Reconstruct the reward function of a single reinforcement learning task

[0112] The TDRP encoder constructed by the method of this embodiment is used to reconstruct the reward function of a single reinforcement learning task, which can enhance the efficiency and stability of training in reinforcement learning task training and enhance the training effect. The TDRP encoder trained by the method of this embodiment is used to reconstruct the reward functions of the grasping task and the placing task and retrain them. For these two tasks, several successful trajectories are extracted respectively, and then the terminal state in each trajectory is extracted to form their corresponding terminal state set. The feature set of these states is obtained by using the TDRP encoder, and then the k-means algorithm is executed on each feature set to divide them into several clusters, and the center point of each cluster is obtained as the target feature set. The reward function of the task is reconstructed using the TDRP encoder and this target feature set, and the task is retrained using this reward function. The training results of the two tasks are shown in the figure below. It can be found that in the two tasks, compared with the results of training using the original environment rewards, the training using the reward function reconstructed by the TDRP encoder of this patent has higher efficiency, can converge to the optimal effect faster, and is also more stable: it can be seen that in the later stages of training, the strategy trained by the original environment rewards will crash, resulting in a significant drop in the success rate, but the strategy trained by the reconstructed rewards has been very stable, with no obvious collapse. For the grasping task, the strategy trained using the reward function reconstructed by the TDRP encoder has better results when converging, and the success rate is much higher than the strategy trained by the original environment rewards. Figure 4 and Figure 5 As shown in the figure, the training curves of the grasping task and the placing task are trained based on the reconstructed reward function and the original reward function. The blue curve represents the training curves of the two tasks trained using the reward function reconstructed by the TDRP encoder, and the gray curve represents the training curves of the two tasks trained using the original environment reward function of the task. The horizontal axis represents the number of iterations, and the vertical axis represents the success rate of the task.

[0113] 3. Reconstruction of skill stitching task rewards

[0114] Applying the TDRP encoder constructed by the method of this embodiment to the skill splicing task can enhance the splicing effect and improve the success rate of the overall task after splicing. The grasping task and the placement task are spliced ​​together, and it is expected that the mechanical control device will complete the coherent task of picking up the nut from the table and placing it on the bolt. The initial states of several successful trajectories of the placement task (the latter task) are extracted, and the TDRP encoder is used to encode them to form a feature set. The feature set is clustered, and the center point of each cluster is obtained as the target feature set of the grasping task (the former task). The TDRP encoder and this target feature set are used to reconstruct the reward function of the grasping task, and the reconstructed reward is used to fine-tune the grasping task. As shown in the figure below, this is the success rate of the placement task after fine-tuning, that is, the success rate of the overall task after splicing the grasping task and the placement task. It can be seen that the success rate is gradually increasing, and the reconstructed reward has played a role, such as Figure 6 shown.

[0115] The core idea of ​​this patent is that the distance between the extracted features can represent the difficulty of reaching each other between two states, and at the same time, some low-level interference information in the state that is irrelevant to task execution can be discarded. High-dimensional state information usually includes multiple perceptual signals, many of which are irrelevant to task execution. Directly calculating the Euclidean distance between two high-dimensional original states is usually not able to represent the distance on the trajectory distribution. The goal is to learn a TDRP encoder φ, in which the distance between any two features in the feature space can measure the number of intermediate transition states required to reach the corresponding state, that is, for two states s on the same trajectory i ,s j ,||φ(s i )-φ(s j )|| 2 ∝|ji|. To achieve this effect, the TDRP encoder needs to be able to extract elements related to action decision-making, task execution, and state transition in the state information, while discarding those interference factors that are irrelevant to task control. Any reconstruction-based feature representation method cannot do this, because even information irrelevant to task control is important for state reconstruction. Therefore, it is expected to use contrastive feature learning, an unsupervised feature learning method, to help obtain the desired TDRP encoder.

[0116] In a single reinforcement learning task, the state representation learning method provided by this patent can be used to construct a better reward function for the task. In a continuous control system, if a target state is expected to be reached, the distance to the target state is sometimes added to the reward function, but usually the Euclidean distance in the original high-dimensional state space has no practical meaning. The state representation method proposed in this patent provides a way to measure whether the current state can reach the target state through fewer intermediate state transfers, that is, the target state can be reached faster in a real sense, and the "transfer distance" to the target state is closer. Use the encoding method proposed in this patent to define better reward functions for different reinforcement learning tasks. Specific implementation method 2:

[0118] The electronic device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a mechanical control device state representation learning method with transfer distance metric when executing the computer program.

[0119] The computer device of the present invention may be a device including a processor and a memory, such as a single chip microcomputer including a central processing unit, etc. Moreover, the processor is used to implement the steps of the above-mentioned mechanical control device state representation learning method with transfer distance metric when executing the computer program stored in the memory.

[0120] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), off-the-shelf programmable gate arrays (FPGAs), or a processor.

[0121] Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0122] The memory may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Specific implementation method three:

[0124] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the state representation learning method of a mechanical control device with transfer distance metric.

[0125] The computer-readable storage medium of the present invention can be any form of storage medium that can be read by the processor of a computer device, including but not limited to non-volatile memory, volatile memory, ferroelectric memory, etc. A computer program is stored on the computer-readable storage medium. When the processor of the computer device reads and executes the computer program stored in the memory, the steps of the above-mentioned mechanical control device state representation learning method with transfer distance metric can be implemented.

[0126] The computer program includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0127] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0128] Although the present application has been described above with reference to specific embodiments, various modifications may be made thereto and parts thereof may be replaced with equivalents without departing from the scope of the present application. In particular, as long as there is no structural conflict, the various features in the specific embodiments disclosed in the present application may be used in combination with each other in any manner, and the fact that these combinations are not exhaustively described in this specification is only for the sake of omitting space and saving resources. Therefore, the present application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A method for learning state representation of a mechanical control device with transfer distance metric, characterized in that: The steps include: S1. Set the motion environment to a Markov decision process and collect motion trajectory information data of the mechanical control device; S2. Preprocessing the motion trajectory information data collected in step S1 based on the contrast feature learning method to process it into motion trajectory information positive data and motion trajectory information negative data, and constructing a motion trajectory information training data set using the motion trajectory information data, the motion trajectory information positive data and the motion trajectory information negative data; S3. Construct a deep neural network as the TDRP encoder; Step S3 builds a deep neural network as the TDRP encoder. The deep neural network includes an input layer, three hidden layers and an output layer. Each layer of the network is implemented using a fully connected network. A RELU network layer is included between every two networks as an excitation layer. Based on the state data being the image data, two convolutional network layers are added before the input layer to realize that the input of the TDRP encoder network is the state data, and the output is the eigenvalue of the state data. θ represents the TDRP encoder network, where θ is the TDRP encoder network parameter; S4. Setting a loss function and an optimizer for the TDRP encoder constructed in step S3, inputting the motion trajectory information training data set obtained in step S2 into the TDRP encoder constructed in step S3, and performing training to obtain a TDRP encoder having a transfer distance metric; The specific implementation method of step S4 includes the following steps: S4.

1. Set the loss function Loss for the TDRP encoder constructed in step S3 to train and update the TDRP encoder. The calculation expression is: in, For expectations; S4.

2. Input the motion trajectory information training data set obtained in step S2 into the TDRP encoder network for end-to-end training update, calculate the error using the loss function constructed in step S4.1, and calculate the gradient through the back propagation algorithm. The calculation expression is: Among them, ε(t) is a cross-order function: S4.

3. Set ADAM as the optimizer for the TDRP encoder constructed in step S3, and use the optimizer to update the parameters of the TDRP encoder network to optimize and update the network. The calculation expression for the optimization update of the tth round is: Among them, m t is the first-order momentum of the tth step, β1 is m t Hyperparameters of Among them, v t is the second-order momentum at step t, and β2 is v t Hyperparameters of Among them, η,∈ is a hyperparameter; S5. For the TDRP encoder with transfer distance metric obtained in step S4, reinforcement learning tasks are set based on different application environments, auxiliary rewards are constructed, and then the auxiliary rewards are combined with the environmental rewards of the reinforcement learning tasks to generate the final internal rewards. The internal rewards are used to train the reinforcement learning tasks, and the training results are repeated in steps S1 to S5 to finally obtain a mechanical control device state representation learning method with transfer distance metric.

2. A mechanical control device state representation learning method with transfer distance metric according to claim 1, characterized in that: The specific implementation method of step S1 includes the following steps: S1.

1. Set the motion environment as a Markov decision process, represented by M = (S, A, P, R, y), where S is the state space set, A is the action space set, P is the transition probability set, R is the reward value set, and y∈(0,1) is the discount coefficient of the reward function; S1.2 Set p = (s'|s, a) as the state transition probability from the first state s∈S to the updated state s'∈s. The mechanical control device determines the action a∈A to be performed in the state s according to the strategy π: a~π(s), and updates the state s'~P(s, a). At the same time, it obtains the reward value r = R(s, a), and uses τ i ={(s0,a0)(s1,a1)...(S T ,a T )} represents the i-th trajectory with a length of T, and the trajectory set is τ i ∈τ.

3. A mechanical control device state representation learning method with transfer distance metric according to claim 1 or 2, characterized in that: Step S2 preprocesses the motion trajectory information data collected in step S1 based on the contrast feature learning method, and converts each state data s in the historical trajectory τ into t ∈S, respectively and Represents the positive and negative data sets of motion trajectory information, and the expression is: Among them, step is the set step size, which is used to control the size of the positive and negative example sets.

4. A mechanical control device state representation learning method with transfer distance metric according to claim 3, characterized in that: The specific implementation method of step S5 includes the following steps: S5.

1. Set the application environment to a single reinforcement learning task, then reconstruct the reward function of the single reinforcement learning task, and the calculation expression is: r(s,a,s g )=R(s,a)-||φ(s)-φ(s g )||2; S5.

2. Set the application environment to the skill connection reinforcement learning task, and set the termination state of the previous skill to To the initial state set of the next technique The TDRP encoder φ with transfer distance metric obtained in step S4 is used for encoding, and then the eigenvalues ​​are aggregated into n state clusters, and the center point of the kth feature state cluster is set to Encode the state of the previous skill, and calculate the minimum distance between it and the center point of the n state clusters as part of the reward to form a new reward function. The calculation expression is: The reward function is used to fine-tune the strategy of the previous skill, hoping that the state of this skill can approach the initial state of the next skill, thus achieving the connection between the two skills.

5. An electronic device, characterized in that The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a mechanical control device state representation learning method with transfer distance metric as described in any one of claims 1 to 4 when executing the computer program.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for learning the state representation of a mechanical control device with a transfer distance metric as described in any one of claims 1 to 4 is implemented.

Images