A robot decision-making method based on link prediction relevance

Abstract

The present invention discloses a robot decision-making method based on link prediction relevance, which relates to the field of robot control technology, including: acquiring offline environmental data of robot control tasks to construct an environmental data set; selecting motion trajectories with high cumulative returns to form a high-reward trajectory set; converting each time step of each motion trajectory in the high-reward trajectory set into a triple form to construct a knowledge graph; training a link prediction model with the knowledge graph, and using the model to predict the relevance of entities and relationships in each time step of the training trajectory, and modifying the expected return of each time step based on the relevance to form a new motion trajectory; training a Transformer model with the new motion trajectory, and using the model to generate the optimal strategy for the robot control task. The present invention uses link prediction to enhance the model's preference for high-reward trajectories, improves the utilization efficiency of trajectory data in robot control tasks, and enables the robot to perform path planning and task execution more accurately.

Description

Technical Field

[0001] The present invention relates to the field of robot control technology and also to the field of offline reinforcement learning technology, and specifically to a robot decision-making method based on link prediction relevance. Background Art

[0002] In robotic control tasks, the agent needs to maximize the cumulative reward through a series of decisions. Traditional reinforcement learning methods usually require direct interaction with the environment to learn the optimal strategy. However, in practical applications, such as autonomous driving and industrial automation, the high cost and high risk of online interaction limit the widespread application of these methods. Offline reinforcement learning provides a solution that does not rely on real-time interaction by learning using static datasets.

[0003] In recent years, a series of emerging offline reinforcement learning methods have been proposed and applied to the field of robot control, and have achieved remarkable results. For example, the Q-Transformer model proposed by DeepMind significantly improves the learning efficiency of large-scale data sets (including real-world data) by discretizing each action dimension and representing the Q value as an independent input element; the research team of the University of Berkeley demonstrated the application of asynchronous offline update strategies in robot operation and demonstrated the effectiveness of multiple robot parallel algorithms.

[0004] Although these research results have promoted the development of offline reinforcement learning in the field of robot control to a certain extent, there are still some challenges. When faced with a large amount of low-quality data and complex environments, the improvement of model performance of existing methods is still limited. Therefore, how to effectively use high-quality trajectory data and optimize the strategy learning process is still the focus of current research. Summary of the invention

[0005] In order to solve the above problems, the purpose of the present invention is to provide a robot decision-making method based on link prediction relevance, aiming to improve the robot's decision-making ability and efficiency in complex environments. This method constructs a knowledge graph and uses a link prediction model to optimize the expected return value of the trajectory to improve the robot's learning efficiency and decision-making accuracy.

[0006] In order to achieve the above object, the present invention adopts the following technical solution.

[0007] A robot decision-making method based on link prediction relevance includes the following steps:

[0008] S1. Obtain offline environmental data required for the robot control task to construct an environmental data set; the environmental data set is a collection of motion trajectories, each of which includes parameters of multiple time steps. The parameters of each time step include state, action, immediate reward and expected return. The expected return is the sum of the immediate rewards from the current time step to the final time step.

[0009] S2: Calculate the cumulative reward of each motion trajectory in the environment dataset, select the motion trajectories with high cumulative rewards to form a high-reward trajectory set. The cumulative reward is the sum of the instant rewards of each time step in the motion trajectory;

[0010] S3, convert each time step of each motion trajectory in the high-reward trajectory set into a triple form, and construct a knowledge graph. The triple has the state as the entity and the expected reward or action as the relationship;

[0011] S4, input the knowledge graph into the constructed link prediction model for training, and generate a link prediction model based on high-reward trajectories to predict the relevance of entities and relations in triples;

[0012] S5, arbitrarily select a motion trajectory from the environmental data set as a training trajectory, and use the trained link prediction model to calculate the relevance of entities and relationships in each time step of the training trajectory;

[0013] S6, based on the correlation predicted by the trained link prediction model, the expected return of each time step of the training trajectory is corrected to form a new motion trajectory;

[0014] S7. Repeat steps S5 and S6 to obtain multiple new motion trajectories, use the multiple new motion trajectories to train the Transformer model, and use the trained Transformer model to generate the optimal strategy for the robot control task.

[0015] In the present invention, when selecting motion trajectories with high cumulative returns, all motion trajectories can be arranged in descending order according to the cumulative returns, and the first X trajectories with the highest cumulative rewards are selected as the high-reward trajectory set, where X is a manually set value; a threshold can also be set, and motion trajectories with cumulative returns equal to or higher than the set threshold are selected as the high-reward trajectory set; of course, a selection ratio can also be set to select a predetermined ratio of motion trajectories from all motion trajectories as the high-reward trajectory set.

[0016] As a specific implementation of the present invention, the ComplEx model is selected as the link prediction model, and the scoring function of the ComplEx model is:

[0017]

[0018] in, Represents a triple Ratings; Represents a triple The embedding vector of the head entity, Embedding vectors representing relations in triples, represents the conjugate vector of the embedding vector of the tail entity in the triple, represents the real part, express Perform dot product operation on three vectors;

[0019] The ComplEx model uses a negative log-likelihood loss function as follows:

[0020]

[0021] in, represents the knowledge graph constructed using a set of high-reward trajectories, , are all triplets, among which, the triplets By triple Swapping The relations of other triples are obtained later; Represents the sigmoid function; after the link prediction model is trained based on the high-reward trajectory set, it can be regarded as a link prediction function; through the link prediction function, the correlation between the robot motion trajectory data and the high-reward trajectory set can be obtained.

[0022] As a specific implementation of the present invention, correcting the expected reward of each time step of the training trajectory includes the following steps:

[0023] If actions are used as relations, the modified expected return is calculated as follows:

[0024]

[0025] This approach combines the original expected return and relevance, refines the distribution of reward signals, and enables the model to more accurately capture effective strategies in complex environments or high-quality data;

[0026] If the expected return is used as the relationship, the modified expected return calculation formula is as follows:

[0027]

[0028] This approach always utilizes high-reward signals, which can effectively remove the noise caused by low-quality expected returns and increase robustness. It is very effective in simple and medium-complexity environments where high-reward trajectories are easier to identify.

[0029] in, Represents a triple The corrected value of the expected reward at the corresponding time step; Represents a triple The expected return at the corresponding time step; A triplet representing the prediction of the link prediction model relevance; Represents the sigmoid function.

[0030] The beneficial effects of the present invention are as follows:

[0031] This paper uses the knowledge graph link prediction and the strong internal logic of link prediction in the knowledge graph to enhance the model's preference for high-reward trajectories, thereby improving the efficiency of trajectory data utilization in robot control tasks. By re-labeling expected rewards and optimizing the trajectory splicing process, the robot's decision-making ability in complex environments is enhanced. In addition, the optimal strategy generated using the self-improvement ability of the Transformer model enables the robot to perform path planning and task execution more accurately. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a flowchart of a robot decision-making method based on link prediction relevance provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. The described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0034] A robot decision-making method based on link prediction relevance, such as Figure 1 As shown, the following steps are included:

[0035] S1. Obtain offline environmental data required for the robot control task to construct an environmental data set; the environmental data set is a collection of motion trajectories, each of which includes parameters of multiple time steps. The parameters of each time step include state, action, immediate reward and expected return. The expected return is the sum of the immediate rewards from the current time step to the final time step.

[0036]

[0037]

[0038]

[0039] In the formula, represents the environmental dataset; Indicates Parameter set of motion trajectories; Indicates the total amount of motion trajectory; Indicates Movement track The time step parameter; represents the total number of time steps; Respectively represent Movement track The state, action, and immediate reward of a time step; For the The motion trajectory starts from Time step to The cumulative instant reward of the time step, i.e. Movement track The expected return at a time step;

[0040] S2: Calculate the cumulative reward of each motion trajectory in the environment dataset, select the motion trajectories with high cumulative rewards to form a high-reward trajectory set. The cumulative reward is the sum of the instant rewards of each time step in the motion trajectory;

[0041]

[0042] In the formula, Indicates The cumulative return of the movement trajectories;

[0043] In this embodiment, all motion trajectories are arranged in descending order according to the cumulative rewards and selected from front to back. The specific number of selections is related to the relationship in the triple. When the expected reward is used as the relationship, 80% of the total motion trajectories in the environmental data set are selected as the high-reward trajectory set. When the action is used as the relationship, 20% of the total motion trajectories in the environmental data set are selected as the high-reward trajectory set.

[0044] S3, convert each time step of each motion trajectory in the high-reward trajectory set into a triple form, and construct a knowledge graph. The triple has the state as the entity and the expected reward or action as the relationship;

[0045] When the state is used as the entity and the expected return is used as the relationship, the triple formed is ;

[0046] When the state is used as the entity and the action is used as the relationship, the triple formed is ;

[0047] S4, input the knowledge graph into the constructed link prediction model for training, and generate a link prediction model based on high-reward trajectories to predict the relevance of entities and relations in triples;

[0048] S5, arbitrarily select a motion trajectory from the environmental data set as a training trajectory, and use the trained link prediction model to calculate the relevance of entities and relationships in each time step of the training trajectory;

[0049] In this embodiment, the knowledge graph after the robot motion trajectory is converted is complex and asymmetric. The ComplEx model is selected as the link prediction model, and the scoring function of the ComplEx model is:

[0050]

[0051] in, Represents a triple Ratings; Represents a triple The embedding vector of the head entity, Embedding vectors representing relations in triples, represents the conjugate vector of the embedding vector of the tail entity in the triple, represents the real part, express Perform dot product operation on three vectors;

[0052] The ComplEx model uses a negative log-likelihood loss function as follows:

[0053]

[0054] in, represents the knowledge graph constructed using a set of high-reward trajectories, , are all triplets, among which, the triplets By triple Swapping The relations of other triples are obtained later; Represents the sigmoid function; after the link prediction model is trained based on the high-reward trajectory set, it can be regarded as a link prediction function; through the link prediction function, the correlation between the robot motion trajectory data and the high-reward trajectory set can be obtained.

[0055] Step S6, based on the correlation predicted by the trained link prediction model, the expected return of each time step of the training trajectory is corrected to form a new motion trajectory;

[0056] If actions are used as relations, the modified expected return is calculated as follows:

[0057]

[0058] If the expected return is used as the relationship, the modified expected return calculation formula is as follows:

[0059]

[0060] In the formula, Represents a triple The corrected value of the expected reward at the corresponding time step; Represents a triple The expected return at the corresponding time step; A triplet representing the prediction of the link prediction model relevance; Represents the sigmoid function;

[0061] Step S7, repeating steps S5 and S6 to obtain multiple new motion trajectories, using the multiple new motion trajectories to train the Transformer model, and using the trained Transformer model to generate the optimal strategy for the robot control task;

[0062] In the process of training the Transformer model, the optimization goal is to minimize the loss function, that is, to minimize the difference between the actual action and the predicted action.

[0063] (1) Data description

[0064] This embodiment is experimentally verified in the robot control task experimental environment Gym-MuJoCo. Gym-MuJoCo contains 9 data sets, namely hopper-medium-v2, halfcheetah-medium-v2, walker2d-medium-v2, hopper-medium-replay-v2, halfcheetah-medium-replay-v2, walker2d-medium-replay-v2, hopper-medium-expert-v2, halfcheetah-medium-expert-v2, and walker2d-medium-expert-v2 data sets. Among them, there are three tasks in the Gym-MuJoCo environment: hopper, halfcheetah, and walker2d. These three tasks indicate that the intelligent agent needs to control different types of robots to take stable actions. The Hopper task is a monopedal robot jumping task simulated based on the MuJoCo physics engine. The Hopper robot consists of 3 joints and 4 degrees of freedom. The task goal is to keep the robot balanced and moving forward on flat ground; the HalfCheetah task simulates a two-dimensional robot with a shape similar to half a cheetah. Its goal is to run forward by controlling the movement of multiple joints. This robot consists of 6 joints and 8 degrees of freedom. The task goal is to achieve rapid forward movement by coordinating the actions of each joint of the robot; the Walker2d task simulates a two-dimensional bipedal robot walking task. The robot has 6 joints and 9 degrees of freedom. The goal is to keep the robot upright and walk forward by controlling the movement of both feet. This is a complex task because the robot must learn to balance and coordinate the movement of the legs to achieve stable walking. '-medium', '-medium-replay' and '-medium-expert' represent sub-environments that rely on different strategies to collect robot motion trajectory data; '-medium' includes one million robot motion sample data collected by the medium strategy; '-medium-replay' uses the medium strategy for training and collects all samples placed in the buffer during training; '-medium-expert' mixes equal amounts of data collected by expert strategies and medium strategies.

[0065] (2) Evaluation indicators

[0066] The evaluation metric used in the experimental environment of this embodiment is the standardized score. The standardized score refers to converting the original reward score in the evaluation phase into a standardized metric so that the reward scores in different tasks or different environments can be reasonably compared. Tasks such as Hopper, Walker2d, and HalfCheetah in the Gym-MuJoCo environment usually use a specific standardized method to calculate the score. The specific formula is:

[0067]

[0068] The score of the agent is calculated by the cumulative reward obtained when the agent executes the model strategy in a specific task environment. In this embodiment, it is the cumulative reward of the model strategy for 1000 time steps (that is, the sum of the instant rewards for executing 1000 time steps); the expert strategy score is the performance of the known optimal strategy in the task, and the random strategy score is the score when the agent takes random actions. The higher the standardized score, the better the performance of the robot using this method in this environment.

[0069] (3) Parameter setting

[0070] The specific hyperparameter settings are shown in Table 1. Table 1 shows the hyperparameter settings of the relevant models.

[0071] Table 1 Hyperparameter settings of the model

[0072]

[0073] (4) Experimental results compared with other models

[0074] In order to demonstrate the reliability and effectiveness of the present invention, this example compares the two models of the present invention (KGDT-A and KGDT-R) with two recent transformer-based offline reinforcement learning methods on the same robot control dataset. The experimental results are shown in Table 2.

[0075] DT: Abstracts traditional reinforcement learning tasks into sequence modeling problems. Unlike previous reinforcement learning methods based on value functions or policy gradients, DT directly outputs the optimal action through the Transformer model. Specifically, DT predicts future actions that can achieve the expected reward by training an autoregressive model conditioned on the expected return, past states, and actions. DT converts reinforcement learning tasks into sequence modeling, which enables better capture of the relationship between action sequences and goals when dealing with complex robot control tasks.

[0076] QDT: Combines the CQL algorithm with the conditional sequence modeling framework to relabel expected rewards. Through this relabeling, QDT can more effectively handle unbalanced trajectory data in offline reinforcement learning, thereby achieving better experimental performance. In robotic control tasks, QDT improves policy learning in this way, allowing the model to obtain more stable results when processing data sets with high diversity;

[0077] Table 2 Experimental results of each model in the Gym-MuJoCo environment

[0078]

[0079] (5) Analysis of experimental results

[0080] As can be seen from Table 2, in most robot control tasks, the performance of KGDT-A and KGDT-R is better than the baseline level. Whether taking the expected reward as a relation or the action as a relation, the overall performance of the present invention surpasses other comparison models.

[0081] In the medium environment, the data quality is at a medium level. Both KGDT-A and KGDT-R can optimize model performance, but the improvement is small except for the hopper task. This is because the medium-quality dataset provides relatively balanced exploration and utilization, and the model has limited room for improvement. However, even in this case, KGDT-A and KGDT-R still show certain advantages, especially in tasks with higher data complexity, demonstrating their effective strategy extraction capabilities.

[0082] In the medium-replay sub-environment, the improvement of KGDT-R is particularly significant. Since the expected return can provide more comprehensive reward feedback, the KGDT-R model is better able to extract effective strategies from trajectory data with high diversity and strong exploratory nature while filtering out noisy data. This effect is particularly evident in the halfcheetah and walker2d tasks, indicating that using the expected return as a relation enables the model to perform well in dealing with data noise and complex trajectories.

[0083] In the medium-expert sub-environment, the KGDT-A model performs better. The expert data provides high-quality trajectories. KGDT-A uses actions as relations to more accurately learn the key action sequences in high-reward trajectories, thereby optimizing the overall strategy. This approach is particularly suitable for high-quality datasets, where the impact of each action on the final reward is more critical.

[0084] Specifically, the KGDT-A and KGDT-R models effectively filter out low-quality noisy data and enhance the robustness of the model in different environments by re-labeling the expected return. This approach has shown excellent performance improvement potential in complex robot control tasks, especially when facing diverse offline datasets.

[0085] The above are only preferred specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the embodiments of the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A robot decision-making method based on link prediction relevance, characterized in that: The steps include: S1. Obtain offline environmental data required for the robot control task to construct an environmental data set; the environmental data set is a collection of motion trajectories, each of which includes parameters of multiple time steps, and the parameters of each time step include state, action, immediate reward and expected return, and the expected return is the sum of the immediate rewards from the current time step to the final time step; S2: Calculate the cumulative reward of each motion trajectory in the environmental data set, and select motion trajectories with high cumulative rewards to form a high-reward trajectory set, where the cumulative reward is the sum of the instant rewards of each time step in the motion trajectory; S3, converting each time step of each motion trajectory in the high-reward trajectory set into a triple form, constructing a knowledge graph, where the triple has a state as an entity and an expected reward or action as a relationship; S4, inputting the knowledge graph into the constructed link prediction model for training, generating a link prediction model based on high-reward trajectories to predict the relevance of entities and relationships in triples; S5, arbitrarily selecting a motion trajectory from the environmental data set as a training trajectory, and using the trained link prediction model to calculate the relevance of entities and relationships in each time step of the training trajectory; S6. Correcting the expected return of each time step of the training trajectory based on the correlation predicted by the trained link prediction model to form a new motion trajectory; Correcting the expected return of each time step of the training trajectory includes the following steps: If the action is used as a relationship, the modified expected return calculation formula is: If the expected return is used as the relationship, the modified expected return calculation formula is: in, Represents a triple The corrected value of the expected reward at the corresponding time step; Represents a triple The expected return at the corresponding time step; A triplet representing the prediction of the link prediction model relevance; Represents the sigmoid function; S7, repeating steps S5 and S6 to obtain multiple new motion trajectories, using the multiple new motion trajectories to train the Transformer model, and using the trained Transformer model to generate the optimal strategy for the robot control task; The link prediction model is a ComplEx model; The scoring function of the ComplEx model is: in, Represents a triple Ratings; Represents a triple The embedding vector of the head entity, Embedding vectors representing relations in triples, represents the conjugate vector of the embedding vector of the tail entity in the triple, represents the real part, express Perform dot product operation on three vectors; The ComplEx model uses a negative log-likelihood loss function, which is: in, represents the knowledge graph constructed using a set of high-reward trajectories, , are all triplets, among which, the triplets By triple Swapping The relations of other triples are obtained later; Represents the sigmoid function.

2. A robot decision-making method based on link prediction relevance according to claim 1, characterized in that: When the expected reward is used as the relationship in step S3, 80% of the total motion trajectories in the environmental data set are selected as the high-reward trajectory set in step S2; when the action is used as the relationship in step S3, 20% of the total motion trajectories in the environmental data set are selected as the high-reward trajectory set in step S2.

Images