A Method and System for Mining Diverse Preference Patterns Based on Training Multiple Reward Models

Abstract

This invention discloses a method and system for mining diverse preference patterns based on training multiple reward models. Through deep learning training, corresponding preference patterns are mined from a preference dataset labeled with multiple different unknown preference patterns. Each preference pattern represents a behavioral pattern or task that a human group hopes the robot will follow. At the same time, corresponding reward models are trained for each preference pattern, and numerical values ​​are used to measure the degree to which the robot performs the task to meet human expectations. This enables the trained reward models with multiple preferences to enable different human groups to make the robot complete the expected tasks.

Description

Technical Field

[0001] This invention relates to a method and system for mining diverse preference patterns based on training multiple reward models. Specifically, it relates to a method and system for mining diverse preference patterns in robot control based on training multiple reward models, and particularly to the process of mining multiple different human preference patterns and obtaining corresponding reward models in the process of obtaining reward signals necessary for reinforcement learning tasks using human feedback. Background Technology

[0002] Deep reinforcement learning technology has developed rapidly and has achieved remarkable results in many fields such as Atari games, Go, robot control, and natural language generation, demonstrating broad application potential. However, in some complex tasks, the environment often fails to provide clear, high-quality reward signals, making it difficult to directly apply traditional reinforcement learning methods.

[0003] In recent years, reward learning techniques based on human feedback have gradually emerged. These methods collect comparisons or rankings of behavioral data from human annotators, use the Bradley-Terry model to model the probability of human preference for a specific sample, and then train a neural network as a reward model. This reward model assigns higher reward scores to answers that better align with human preferences, thus transforming abstract, subjective human preferences into numerical reward signals that can be used for reinforcement learning training. This technology enables reinforcement learning systems to learn and optimize behaviors that conform to human preferences without relying on pre-set environmental rewards, providing crucial technical support for the practical application of reinforcement learning.

[0004] While reward learning techniques based on human preferences have made significant progress in various tasks, most current research assumes that human annotators compare and rank data according to a single, fixed criterion, thereby training a single reward model reflecting a single human preference. This assumption ignores the diversity and complexity of human preferences themselves. In reality, different annotators may give different or even conflicting evaluations of the same pair of samples, and even the same annotator may make inconsistent judgments in different contexts or at different times. In robot control tasks, the evaluation of robot actions may involve multiple aspects such as efficiency, stability, and anthropomorphism. However, different people may have different specific concerns about these aspects and varying degrees of attention to each aspect. For example, suppose human annotators are asked to choose the better trajectory between two samples of a robot walking forward, based on the quality of the robot's performance in a "walking" task. Some annotators might simply want the robot to walk as fast as possible, without considering other factors, and therefore choose the faster trajectory. Others might prefer the robot to maintain a safe speed and a more stable and aesthetically pleasing gait, and therefore choose the slower but more stable trajectory. This leads to different groups making different choices when comparing the same two trajectories, resulting in conflicting labels in the dataset. Traditional reward-based learning techniques using human feedback use a single reward model, which can only choose one of the two conflicting labels, failing to simultaneously model the preferences of different groups. Therefore, relying solely on a single reward model is insufficient to accurately reflect the diversity and dynamic changes in human preferences. Summary of the Invention

[0005] Purpose of the Invention: Existing reward learning techniques based on human feedback can only measure the quality of a robot's performance on a specific task under a single criterion, and cannot simultaneously model the preference patterns of different human groups. Current methods cannot extract the different tasks (such as fast running, steady walking, running and jumping, etc.) that different groups expect the robot to perform from mixed datasets containing multiple preference patterns, nor can they train corresponding reward models for each task. To address these problems and shortcomings, this invention provides a diverse preference pattern mining and training method based on training multiple reward models. This method utilizes deep learning technology to mine these preference patterns from mixed datasets containing multiple unknown preference patterns, and trains corresponding reward models for each preference pattern. This provides the necessary reward signals for reinforcement learning training of robots to complete various tasks expected by different human groups.

[0006] Technical Solution: A method for mining diverse preference patterns based on training multiple reward models. Through deep learning training, this invention can mine corresponding preference patterns from a mixed preference dataset labeled with multiple different unknown preference patterns. Each preference pattern represents a behavioral pattern or task that a human group hopes the robot will follow. Simultaneously, corresponding reward models are trained for these preference patterns, and numerical values ​​are used to measure how well the robot performs these tasks to meet human expectations. This provides an accurate reward model that reflects multiple preferences for the subsequent reinforcement learning stage, making it possible to train a robot that can complete the tasks desired by different human groups.

[0007] This invention utilizes the similarity between points in a high-dimensional space to model the human preference labeling process. Both robot trajectory data and human-preferred robot action patterns in the dataset are modeled as points in a multi-dimensional space. The points modeled from robot trajectories are called data points, and the points modeled from human-preferred robot action patterns are called preference points. The closer the data point is to the preference point and the higher the similarity, the more the human group corresponding to that preference point believes the robot's behavior at that data point better aligns with their preferences. The reward model for the robot action pattern corresponding to the preference point will then award a larger reward to that robot trajectory data sample. This method uses the distance between data points and preference points to model the reward value awarded by the robot action pattern to that robot trajectory data sample, thereby constructing a reward model that conforms to the preference pattern.

[0008] The method specifically includes the following steps: Step 1: Prepare a preference dataset labeled by annotators with various different preferences.

[0009] The dataset originates from a Markov decision process. The defined environment can be a real-world scenario or a simulator.

[0010] in, The state space represents the robot's specific state at a given moment, such as the current position of each part of the robot and the angle of each joint. It represents the motion space, which represents the control inputs that can be given to the robot, such as the torques applied to each joint. Let represent the state transition function, which indicates the probability of a robot transitioning to a specific state in the next instant when it performs a certain action in a certain state.

[0011] Step 2: Initialize the data point encoding module and the preference point modeling module.

[0012] This invention uses a neural network to transform the trajectory data of robot actions into a point in a high-dimensional space; this neural network is called the data point encoding module. Furthermore, this invention uses multiple points in the high-dimensional space to represent various different human preference patterns; these points are called preference points, and the collective set of all preference points is called the preference point modeling module. Before training the data point encoding module and the preference point modeling module begins, the number of preference points needs to be preset. and preference point dimension Random initialization indivual A unit vector of dimension is used as a preference point, denoted as . Preference points All meet and .this These preference points are collectively referred to as the preference point modeling module. The individual preference points are collectively referred to as the preference point modeling module. A fully connected neural network is initialized as the data point encoding module, denoted as... Used to transfer trajectory data Transform into dimensions Data points.

[0013] The data point encoding module, combined with any preference point, constitutes a reward model. The preference point modeling module of this invention can model multiple different preference points simultaneously, thus generating multiple reward models. These reward models share the data point encoding module and each has its own preference point. Given a sample in the dataset (containing two trajectories and a human annotation y), the data point encoding module converts the two trajectories into two data points. The cosine similarity between the data points and the preference point can be calculated. This cosine similarity value represents the numerical reward that the trajectory corresponding to the data point can obtain under the preference pattern corresponding to that preference point.

[0014] After preparing the preference dataset, multiple rounds of deep learning training are initiated to mine the various preference patterns contained in the dataset and to train the data point encoding module and the preference point modeling module. In each round, steps 3 through 6 are repeated.

[0015] Step 3: Sample data from the preference dataset.

[0016] At the start of each training round, a batch of data is randomly sampled from the dataset. ,in This refers to the amount of data used in a single training round. This indicates the first [number] in this batch of data. Two trajectories in a sample, and This indicates that human annotators gave the first The preference labels assigned to the two trajectories in a sample can take values ​​of 0, 1, or 0.5. This indicates that human annotators believe Superior , This indicates that human annotators believe Superior , This indicates that human annotators consider the two trajectories to be of roughly equal quality.

[0017] Step 4: For each sample in the batch of data obtained in Step 3 The data point encoding module is used to separate the two trajectories in the sample. The encoding is two data points.

[0018] Step 5: Calculate the reward for each of the two trajectories of each sample in the data sampled in Step 3 and the prediction loss for the preference label of each sample in the data sampled in Step 3 using the preference point modeling module.

[0019] After obtaining the data points, for each preference point included in the preference point modeling module... The cosine similarity between data points and preference points is used as a measure of the cumulative reward for the trajectory. Based on this numerical reward, the Bradley-Terry model is used to calculate the probability that the reward model prefers one trajectory over the other. Then, the cross-entropy loss function is used to calculate the prediction loss of the single reward model consisting of the data point preference module and the single preference point for each sample in the data sampled in step 3.

[0020] Step 6: Integrate the prediction loss of multiple reward models composed of the data point encoding module and multiple preference points using the principle of minimizing the optimal loss, and use the gradient descent method to perform deep learning training to adjust the parameters of multiple preference points in the neural network that serves as the data point encoding module and the preference point modeling module.

[0021] Step 7: Repeat steps 3 through 6 until convergence is reached or the maximum number of iterations is reached. This yields reward models corresponding to different preference patterns. The reward model consists of multiple preference points contained in a shared data point encoding module and a preference point modeling module. These multiple preference points correspond to various preference patterns mined from the dataset. For any trajectory, the data point encoding module can convert it into data points. The distance between the data points and preference points can be used to determine the degree of preference for a given trajectory among different preference patterns.

[0022] A method and system for mining diverse preference patterns based on training multiple reward models, comprising the following modules: The data point encoding module, composed of a fully connected neural network, reads two trajectory segments from the sample data in the preference dataset, representing the robot's attempts to complete certain tasks. These segments are called robot action trajectory data, and the neural network encodes the samples into two data points in a specified multidimensional space. The function of the data point encoding module is to extract the features of the robot's behavior trajectory segments and transform them into data points in multidimensional space as input to the preference point modeling module.

[0023] The preference point modeling module contains multiple randomly initialized preference points, each used to model a human preference pattern. This module receives data points encoded by the data point encoding module as input, and calculates the numerical reward for the preference pattern corresponding to the preference point on the trajectory corresponding to the data point by performing a cosine similarity operation between the preference point and the data point.

[0024] The training module will jointly train the two modules using a method based on minimizing the optimal loss. Its input consists of the preference label for each sample in the preference dataset, and two numerical rewards obtained from the data point encoding module and the preference point modeling module. The training module then calculates the cross-entropy loss using the two rewards and the preference label, and trains the neural network of the data point encoding module and the preference points of the preference point modeling module using gradient descent.

[0025] The implementation process and method of the system are the same and will not be described again.

[0026] In step 1, The state transition function represents the probability that a robot will transition to a specific state in the next time step after performing an action in a given state. This probability is often modeled based on physical dynamics and environmental interactions. Unlike Markov decision processes used in common reinforcement learning, there is no reward function in this scenario. Instead, each sample in the preference dataset contains two trajectories, and a labeler assigns a preference label according to their own preference pattern, indicating which trajectory in the pair is more favored by that labeler. Formalizing this, a sample in the preference dataset can be represented as... ,in They represent lengths of A trajectory, each step containing a state. and an action A pair of trajectories are numbered using 0 and 1 respectively, and This indicates that the annotator is referring to the trajectory pair. and preferences, This indicates that the annotators prefer trajectory 0. , This indicates that the annotator considers the two trajectories to be of similar quality, while This indicates that the annotator prefers trajectory number 1. Trajectories in preference datasets can be collected by human experts or programmed robots running in real-world scenarios or simulators, while preference labels are provided by a group of human annotators based on their respective preference patterns. Due to the large volume of preference data pairs required, multiple human annotators are often needed to label the data in practice. However, different human annotators may have different preference patterns, and even the same person's preference patterns may change at different times and in different states, influenced by emotions and the environment. Therefore, preference datasets often contain a mixture of various and unknown preference patterns, rather than all preference labels following a clear, single standard. For example, given the requirement to "choose the better walking path from two trajectories," different human annotators will have different understandings of "better walking path," thus choosing different trajectories and providing different preference labels.

[0027] In step 2, the two trajectories in the sample and The data is encoded using a data point encoding module and converted into data points. and .

[0028] In step 2, three fully connected neural networks are initialized as data point encoding modules, namely, state encoders. Motion encoder and joint encoder Used to transfer trajectory data Transform into dimensions Data points.

[0029] In step 5, the similarity between the data point and the preference point in the preference point modeling module is used as the cumulative reward for the trajectory corresponding to the preference point. That is to say, the first Preference points The corresponding reward model gives the trajectory The cumulative rewards, among which This indicates the calculation of cosine similarity.

[0030] Given the rewards calculated by different reward models for two trajectories, this invention uses the Bradley-Terry model to calculate the probability that the reward model favors one trajectory over the other, and then uses labels... Calculate the cross-entropy loss for a single reward model. For the ... One preference point, The probability of being preferred is: The corresponding cross-entropy loss is: for The reward model integrates multiple cross-entropy losses obtained by minimizing the optimal loss in the following way: in The temperature coefficient, used to balance overall performance and optimize efficiency, is a pre-specified hyperparameter. (Large) It will treat the cross-entropy loss of different preference points approximately equally, while when As the value approaches zero, the aforementioned integration loss will approach zero. The minimum value among them. For different scenarios, the most suitable value needs to be selected according to the characteristics of the scenario. In the scenario of multi-joint robots, this scheme sets the coefficient to 0.5. By using this ensemble loss for model optimization, this invention enables the reward model that can most correctly predict the preference label of any sample sampled from the preference dataset to obtain the largest gradient, thereby optimizing the preference prediction of that sample. This promotes the formation of differences between different preference points, thus covering the various different preference patterns contained in the preference dataset. As a result, a reward model is trained for the preference patterns contained by all annotators who participated in the dataset annotation, which can measure the robot's performance when completing the tasks expected by these annotators.

[0031] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method for mining diverse preference patterns based on training multiple reward models as described above.

[0032] A computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps of the diverse preference pattern mining method based on training multiple reward models as described above.

[0033] This invention provides a method for mining diverse human preferences and simultaneously training multiple reward models. Compared to previous techniques, the training process takes into account the diversity and complexity of human preferences, enabling the mining of various human preference patterns from mixed preference datasets and the simultaneous training of multiple reward models. It solves the problem of high noise and limited performance in traditional reward learning based on human feedback, where different labelers may give different or even conflicting evaluations to the same pair of samples, thus offering broad application prospects. Through this invention, various tasks that humans actually want robots to perform or action patterns that they want robots to follow can be mined from mixed preference datasets full of conflicting labels, which are inaccessible to traditional reinforcement learning methods based on human feedback. Furthermore, a reward model can be generated for each task to measure the quality of the robot's behavior when performing that task, making it possible to train robots to complete these tasks and meet human expectations through reinforcement learning. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the data point encoding module described in this invention; Figure 2 This is a schematic diagram of the preference point modeling module described in this invention; Figure 3 This is a schematic diagram of the trajectory reward and training loss calculation process described in this invention; Figure 4 This is a flowchart of the diverse preference pattern mining process based on training multiple reward models as described in this invention. Detailed Implementation

[0035] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0036] A method and system for mining diverse preference patterns based on training multiple reward models is presented. This method models the human preference annotation process by using the distance between points in a high-dimensional space. Both sample data and human preference patterns in the dataset are modeled as points in a multi-dimensional space. A data point encoding module and a preference point modeling module are jointly trained using a method that minimizes the optimal loss condition. This allows for the mining of various human preferences from mixed preference datasets and the generation of multiple reward models corresponding to these preference patterns. This solves the problem that traditional reward learning methods based on human feedback cannot explicitly extract the different tasks that different human groups expect robots to perform in the field of robot manipulation, nor can they simultaneously train multiple reward models for these different tasks.

[0037] Figure 4This paper demonstrates the entire process of the method for mining diverse preference patterns based on training multiple reward models, as described in this invention, including the following steps: Step 1: Prepare a preference dataset labeled by human annotators.

[0038] The dataset originates from a Markov decision process. The defined environment can be a real-world scenario or a simulator. Representing the state space, Represents the action space, This represents the state transition function.

[0039] In multi-joint robot control, the state space can be composed of all possible positions, joint angles, velocities, and angular velocities of the robot's various parts. Each state represents the robot's configuration at a given moment. Different robots have varying numbers of joints and ranges of motion, thus their state spaces also differ in dimension. The action space contains all possible control inputs, such as the torques applied to each joint. In this scenario, because the control inputs for each joint can take any real value within a certain range, the action space is typically continuous. The state transition function describes the probability that the robot will transition to a specific new state given the current state and the action taken. In multi-joint robot control, this probabilistic model is usually constructed based on physical dynamics and the interaction with the environment. Because robot motion is affected by physical constraints and external disturbances, state transitions often have uncertainties, thus requiring a probabilistic model for description.

[0040] A single sample in the dataset contains two trajectories obtained from the aforementioned environment, which can be acquired through interaction within that environment by a human expert or a previously trained AI model. In addition, the sample includes a label indicating which of the two trajectories a human annotator prefers. Formally, a sample can be represented as... ,in They represent lengths of A trajectory, each step containing a state. and an action ,and This indicates that the annotator is referring to the trajectory pair. and The given preference labels This indicates that the annotators prefer trajectory 0. , This indicates that the annotator considers the two trajectories to be of similar quality, while This indicates that the annotator prefers trajectory number 1. Different annotators may assign preference labels based on different criteria. In robot control problems, some annotators may focus more on the robot's speed in a specific direction, while others may focus more on the robot's jump height. Still others may focus more on whether the robot follows a certain posture (such as keeping one leg off the ground). In addition, some annotators may assign preference labels based on highly subjective factors such as aesthetics. This can lead to different people assigning different preference labels to the same two trajectories.

[0041] Step 2: Initialize the data point encoding module and the preference point modeling module.

[0042] Before training begins, input the preset number of preference points. and preference point dimension Randomly initialized by the preference point modeling module indivual A unit vector of dimension is used as a preference point, denoted as . Preference points All meet and Simultaneously, three fully connected neural networks are initialized as data point encoding modules, namely the state encoder. Motion encoder and joint encoder Used to transfer trajectory data Transform into dimensions Data points.

[0043] Step 3: Sample data from the preference dataset.

[0044] After preparing the preference dataset, multiple rounds of deep learning training are initiated to uncover various preference patterns contained within the dataset and train a reward model. In each round, this invention randomly samples a batch of data from the dataset in the aforementioned format. ,in The amount of data used in a single training round.

[0045] Step 4: Use the data point encoding module to encode the trajectory into data points.

[0046] The first step in training is to use the data point encoding module to convert the trajectories in the sampled data from step 2 into data points, such as... Figure 1 As shown, for a single trajectory The specific steps are as follows: Step 41: The data point encoding module will first obtain the state code for each time step through the state encoder. ,in From arrive All time steps.

[0047] Step 42: The data point encoding module obtains the motion code for each time step through the motion encoder. ,in From arrive All time steps.

[0048] Step 43: Concatenate the state code and action code for each time step, and obtain the joint code for each time step through a joint encoder and normalization. .in This refers to the standardization operation, which ensures that the joint encoding maintains a modulus of 1, i.e. in Indicates a having A vector of dimension, where Representing vectors The Each component.

[0049] Step 44: By summing the joint codes of all time steps, the code of the entire trajectory, i.e., the data points corresponding to the trajectory, can be obtained. .

[0050] Step 5: Calculate the reward and predicted loss for a single preference using the preference point modeling module.

[0051] After obtaining the data points, this method uses the similarity between the data points and the preference points as a measure of the cumulative reward for the trajectory. Based on this numerical reward, the probability that the reward model favors one trajectory over the other is calculated using the Bradley-Terry model, thus calculating the individual prediction loss for each reward model on the preference pair. For a single sample... For a single preference point The specific steps are as follows: Step 51: For the first Reward Model Based on Preference Patterns Its trajectory The given method for calculating cumulative rewards is as follows: .in This indicates similarity calculation. The cumulative reward calculation process is as follows: Figure 2 As shown. In the robot operating environment, this invention uses cosine similarity for calculation.

[0052] Step 52: Use the Bradley-Terry model to calculate the preference prediction for the trajectory using a single reward model. In the Bradley-Terry model, for the reward model... trajectory The probability of being preferred is: Step 53: For each reward model Calculate the cross-entropy loss for each individual preference prediction of the reward model. The specific formula is as follows: Step 6: Integrate the prediction losses of different reward models using the principle of minimizing the optimal loss, and then perform deep learning training. The complete prediction loss calculation process is as follows: Figure 3 As shown.

[0053] Step 61: After obtaining the prediction loss of a single model, this invention integrates the multiple cross-entropy losses obtained by minimizing the optimal loss in the following way: in The temperature coefficient, which is used to balance overall performance and optimize efficiency, is a pre-specified hyperparameter.

[0054] Step 62: Perform deep learning training on all neural networks of the data point encoding module and the preference points of the preference point modeling module to minimize the above-mentioned integration loss.

[0055] Step 7: Repeat steps 3 through 6 until convergence is reached or the maximum number of iterations is reached. Afterwards, the preference point modeling module includes... Preference points This means that it is obtained from the mixed preference dataset. These are different preference patterns. Combined with the three encoders in the data point encoding module, this yields... Different preference patterns correspond to A reward model. This model considers the robot's state at a given moment and the actions it performs in that state. The joint encoding can be obtained through steps 41 to 43. Then, by calculating the joint encoding and preference points The cosine similarity can be used to obtain the state. Next action At that time, the first The reward model corresponding to the individual group preference pattern will give the robot's decision a reward, that is, the reward for the robot's current decision on completing the first step. The magnitude of the impact of the task that a human group expects to accomplish.

[0056] Through the above steps, this invention can mine different tasks that different human groups hope the robot can complete or different behavioral patterns that can be followed from mixed preference datasets, and generate a reward model for each task that reflects the quality of the robot's task completion, making it possible to train robot policies using reinforcement learning methods so that the robot can meet the preferences of different groups.

[0057] The diverse preference pattern mining method based on training multiple reward models can be used in many real-world environments, including robot manipulation, game strategy, and natural language generation, to perform data mining and generate reward models.

[0058] Obviously, those skilled in the art should understand that the steps of the diverse preference pattern mining method based on multiple reward models or the modules of the diverse preference pattern mining system based on multiple reward models in the above embodiments of the present invention can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the embodiments of the present invention are not limited to any particular hardware and software combination.

Claims

1. A method for mining diverse preference patterns based on training multiple reward models, characterized in that, Through deep learning training, corresponding preference patterns are mined from a preference dataset labeled with multiple different unknown preference patterns. Each preference pattern represents the behavior pattern or task that a human group hopes the robot will follow. At the same time, corresponding reward models are trained for each preference pattern. Numerical values ​​are used to measure the degree to which the robot performs the task to meet human expectations. This enables the trained reward models with multiple preferences to enable different human groups to make the robot complete the expected tasks. The reward model utilizes the similarity between points in a high-dimensional space to model the human annotation process of preferences. During the training of the reward model, both the robot trajectory data and the robot action patterns preferred by humans in the preference dataset are first modeled as points in a multi-dimensional space. The points modeled from the robot trajectory are called data points, and the points modeled from the robot action patterns preferred by humans are called preference points. Then, the distance between the data points and preference points is calculated. The closer the distance and the higher the similarity, the more the human group corresponding to the preference point believes that the robot's behavior at this data point is more in line with their preferences. The reward model for the robot action pattern corresponding to the preference point will give a larger reward to the robot trajectory data sample. The distance between the data points and preference points is used to model the reward value given by the robot action pattern to the robot trajectory data sample, thereby constructing different reward models that conform to different preference patterns.

2. The method for mining diverse preference patterns based on training multiple reward models according to claim 1, characterized in that, The preference dataset is a dataset labeled by annotators with multiple different preferences, derived from a Markov decision process. The defined environment is either a real-world scenario or a simulator, where... The state space represents the robot's specific state at a given moment. This represents the action space, which indicates the control inputs that can be given to the robot. Let represent the state transition function, which indicates the probability that the robot will transition to a certain state in the next time step after performing a certain action in a certain state; a sample in the preference dataset is represented as . ,in They represent lengths of A robot's movement trajectory, each step containing a state. and an action A pair of trajectories The trajectories are numbered using 0 and 1 respectively, and This indicates that the annotator is referring to the trajectory pair. and preferences, This indicates that the annotators prefer trajectory 0. , This indicates that the annotator considers the two trajectories to be of similar quality, while This indicates that the annotator prefers trajectory number 1. .

3. The method for mining diverse preference patterns based on training multiple reward models according to claim 1, characterized in that, The data point encoding module converts the robot's trajectory into data points. Specifically, it uses a neural network as the data point encoding module to convert the robot's trajectory data into a point in a high-dimensional space. The points in the d-dimensional space are used to represent various human preference patterns. These points are called preference points, and the sum of all preference points is called the preference point modeling module. Before training begins for the data point encoding module and the preference point modeling module, the number of preference points is preset. and preference point dimension Random initialization indivual A unit vector of dimension is used as a preference point, denoted as . Preference points All meet and ;this The individual preference points are collectively referred to as the preference point modeling module; the fully connected neural network is initialized as the data point encoding module, denoted as... Used to transfer trajectory data Transform into dimensions Data points.

4. The method for mining diverse preference patterns based on training multiple reward models according to claim 1, characterized in that, The preference point modeling module can model multiple different preference points simultaneously, with each preference point corresponding to a reward model. Multiple reward models share a data point encoding module and have their own preference points. Given a sample in the preference dataset, the data point encoding module will convert the two trajectories of the sample into two data points respectively, calculate the cosine similarity between the data point and the preference point, and the value of the cosine similarity is the numerical reward obtained by the trajectory corresponding to the data point under the preference mode corresponding to the preference point.

5. The method for mining diverse preference patterns based on training multiple reward models according to claim 1, characterized in that, After preparing the preference dataset, multiple rounds of reward model training are performed to mine the various preference patterns contained in the preference dataset and train the data point encoding module and preference point modeling module. In each round, steps 301 to 304 are repeated. Step 301: Sample data from the preference dataset; At the start of each training round, a batch of data is randomly sampled from the dataset. ,in This refers to the amount of data used in a single training round; among which This indicates the first [number] in this batch of data. Two trajectories in a sample, and This indicates that human annotators gave the first Preference labels assigned to two trajectories in a sample; Step 302: For each sample in the batch of data obtained from sampling The data point encoding module is used to separate the two trajectories in the sample. Encoded as two data points; Step 303: Calculate the reward for the two trajectories of each sample in the sampled data and the prediction loss for the preference label of each sample in the sampled data using the preference point modeling module. After obtaining the data points, for each preference point included in the preference point modeling module... The cosine similarity between data points and preference points is used as the cumulative reward metric for the trajectory. Based on the cumulative reward metric, the Bradley-Terry model is used to calculate the probability that the reward model prefers one trajectory to the other. Then, the cross-entropy loss function is used to calculate the prediction loss of each sample in the sampled data by the single reward model consisting of the data point preference module and the single preference point. Step 304: Integrate the prediction loss of multiple reward models composed of data point encoding module and multiple preference points using the principle of minimizing optimal loss, and use gradient descent to perform deep learning training to adjust the parameters of multiple preference points in the neural network that serves as the data point encoding module and the preference point modeling module. Step 305: Repeat steps 301 to 304 until convergence or the maximum number of iterations is reached; the reward models corresponding to different preference patterns are obtained.

6. The method for mining diverse preference patterns based on training multiple reward models according to claim 1, characterized in that, The Bradley-Terry model is used to calculate the probability that the reward model prefers one trajectory to the other, and this is done through labeling. Calculate the cross-entropy loss for a single reward model; for the One preference point, The probability of being preferred is: The corresponding cross-entropy loss is: for The reward model integrates multiple cross-entropy losses obtained by minimizing the optimal loss in the following way: in To balance the temperature coefficient between overall performance and optimization efficiency, model optimization is performed using integrated loss.

7. A method and system for mining diverse preference patterns based on training multiple reward models, characterized in that, It includes the following modules: The data point encoding module, composed of a fully connected neural network, reads sample data of robot movement trajectory data from the preference dataset and encodes the robot movement trajectory data into two data points in a specified multi-dimensional space through the neural network. The function of the data point encoding module is to extract the features of the robot behavior trajectory data and transform the features into data points in multi-dimensional space as input to the preference point modeling module. The preference point modeling module contains multiple randomly initialized preference points, each of which is used to model a human preference pattern. This module receives data points encoded by the data point encoding module as input, and obtains the numerical reward of the preference pattern corresponding to the preference point for the trajectory corresponding to the data point by performing cosine similarity operation between the preference point and the data point. The training module uses a method based on minimizing the optimal loss to jointly train the data point encoding module and the preference point modeling module using deep learning. The training module takes as input the preference label of each sample in the preference dataset, and two numerical rewards obtained from the two trajectory input data point encoding module and the preference point modeling module. Subsequently, the training module calculates the cross-entropy loss based on the two reward and preference labels, and trains the neural network of the data point encoding module and the preference points of the preference point modeling module using gradient descent.

8. A computer device, characterized in that: The computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the diverse preference pattern mining method based on training multiple reward models as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that: When the computer program / instruction is executed by the processor, it implements the steps of the diverse preference pattern mining method based on training multiple reward models as described in any one of claims 1-6.

Images