Personal agent strategy training method and system based on preference reinforcement learning

Abstract

The invention discloses a preference reinforcement learning-based body-equipped agent strategy training method and system, and the method comprises the steps: obtaining and constructing a multi-source non-stationary environment preference data set: collecting historical interaction trajectories from different bottom-layer dynamic environments, and generating trajectory fragment pairs and corresponding human preference labels; carrying out environment dynamic predictor training, and enabling a calculation control unit to grasp dynamic evolution and dynamic change rules of the body-equipped intelligent body in different physical environments in advance; executing decoupling joint optimization based on conditional mutual information and a regret model; extracting a pure task award decoder which is absolutely immune to any dynamic physical disturbance; security underlying action strategy distillation is executed based on expectation regression, and a final action strategy for controlling physical equipment is safely generated by using pure rewards; and after the strategy network is trained and converged, the processor compiles the strategy network into an executable instruction, and directly deploys the executable instruction to a control unit of the target physical equipment to execute intelligent action control.

Description

Technical Field

[0001] This invention relates to the interdisciplinary field of artificial intelligence, embodied intelligence, and data communication interaction technology, specifically to a machine learning model training technique and an embodied agent control method for complex dynamic systems. More specifically, this invention relates to a preference reinforcement learning method that decouples dynamic environmental information from real reward signals through representation learning, thereby enabling the transferability of reward functions under different dynamic environmental conditions. This method can be widely applied to complex control scenarios in embodied intelligent systems (such as autonomous driving, drone swarms, quadruped robots, flexible robotic arms, etc.) where policy learning needs to be performed in dynamically changing physical environments. Background Technology

[0002] In the field of embodied intelligence, reinforcement learning has become a core technological paradigm in complex decision-making scenarios such as autonomous driving and robot control. Its core idea is to enable intelligent agents to learn the optimal strategy that maximizes cumulative rewards through trial and error in interaction with the environment. However, the design of the reward function directly determines the learning effect. Manually defining a precise reward function is often extremely difficult in real-world tasks, and human experts struggle to accurately quantify complex human intentions such as "safety, comfort, and efficiency" using simple numerical formulas. To reduce the cost of reward design, reinforcement learning based on human preferences has emerged: by collecting pairwise comparisons of trajectory segments from humans, and assuming that these preferences are generated by the sum of the cumulative rewards of the trajectories, a neural network is fitted using a contrastive learning framework as the reward function. Finally, this is combined with offline reinforcement learning algorithms to train the decision-making strategy. This method significantly reduces the burden of human annotation and is therefore widely used in practical systems.

[0003] However, existing preference learning control methods based on cumulative rewards suffer from fatal flaws when deployed in the real physical world, severely limiting the operational safety and robustness of embodied agents in non-stationary physical environments. First, traditional reward accumulation models face parameter uncertainty. Preference data only provides relative comparisons between trajectories, while mathematically, there are infinitely many reward functions that can equally well explain the same set of preference labels. This "reward unrecognizable" characteristic makes the learned reward function extremely sensitive to data noise and model assumptions. In practical applications, even small biases in preference labeling or changes in data distribution can cause drastic fluctuations in the fitted reward function, leading to highly unstable subsequent policy training and difficulty in converging to reliable behavior. Second, real physical environments are highly non-stationary, and their dynamic characteristics (such as friction, wind resistance, and load damping) change over time or in different scenarios. Taking autonomous driving, a typical example of embodied intelligence, as an example, the adhesion between the tires and the road surface differs significantly when a vehicle is driving in rain or sunshine. This inevitably causes the dynamic features of the environment to be absorbed into the reward function, meaning that a reward model learned in one dynamic environment will completely fail when transferred to another. It fails to distinguish between "task objectives" and "environmental dynamics," leading to incorrect scores in new environments and a lack of guidance for agent behavior. This makes existing preference learning methods difficult to implement in real-world, non-stationary physical systems. Therefore, there is an urgent need for a reward learning method that can achieve stable learning and generalize across environments, as well as a safe embodied agent decision-making and action control strategy. Summary of the Invention

[0004] Purpose of the invention: To address the problems existing in the preference reinforcement learning of embodied intelligent agents (including autonomous driving, robots, etc.), this invention proposes an embodied intelligent agent policy training method and system based on preference reinforcement learning. By introducing a regret model to replace the traditional cumulative reward assumption, the method alleviates the parameter uncertainty of the traditional cumulative reward model. Combined with a dynamic adversarial decoupling mechanism, the reward signal for safe task execution is accurately extracted, thereby achieving stable and transferable reward learning in non-stationary environments. At the same time, it significantly improves the absolute safety of physical entity operations and cross-scenario generalization ability.

[0005] Technical Solution: An embodied agent policy training method based on preference reinforcement learning. By combining the regret preference model with the decoupling of conditional mutual information features, the agent can extract absolutely pure task reward information from multi-source non-stationary environments. Subsequently, using an offline reinforcement learning algorithm based on expectation regression, the embodied agent can extract control policies with cross-environment adaptability without querying risky actions.

[0006] The method includes the following steps: Step 1: Acquire and construct a multi-source non-stationary environment preference dataset, including collecting historical interaction trajectories from different underlying dynamic environments, generating trajectory segment pairs and corresponding human preference labels, and providing the computational control unit with interaction materials containing different physical environmental perturbations. Specifically, the sensor data acquisition interface receives historical interaction data streams from multiple source environments with differing underlying dynamic parameters. This data is then segmented and formatted by the processor to include state... With action Equal-length trajectory segments After pairing, the binary preference labels are evaluated by human experts or based on preset rules. This label indicates which trajectory best matches the task expectation. Finally, the trajectory pairs with accompanying environmental non-stationary features are combined with the preference labels to construct a multi-source environmental preference dataset in memory. Simultaneously, all single-step transition data are extracted and used to construct an unlabeled experience replay pool. This is for use by subsequent modules.

[0007] In step 1, the embodied intelligent agent collects historical interaction trajectories from different underlying dynamic environments. These historical interaction trajectories include the historical interaction trajectory of autonomous driving, the motion trajectory of a drone / robot, or the spatial motion trajectory of a flexible robotic arm grasping an object.

[0008] Step 2: Train the environmental dynamics predictor to allow the computational control unit to learn in advance the dynamic evolution of the embodied agent under different physical environments, providing a benchmark for subsequent decoupling. During execution, four main deep neural networks are first initialized: state-action feature encoders. Reward Decoder Conditional dynamic predictor and marginal dynamic predictor In this pre-training phase, the network parameters of the feature encoder and reward decoder are first frozen, and state-action pairs are encoded as latent features. And receive the estimated reward Subsequently, from the preference dataset Data is extracted in medium batches, and two predictors are trained independently by maximizing the log-likelihood probability. Specifically, the conditional dynamic predictor is based on complete latent features. The pre-training process learns to predict the probability distribution of the next state, while the marginal dynamic predictor learns to predict the next state solely based on the decoded scalar reward. This pre-training process aims to minimize the following loss function: in, Expressing expectations; These represent the state, action, and next state, respectively. This represents an unlabeled experience replay pool. The parameter is Conditional dynamic predictor; The parameter is Marginal dynamic predictor; This represents the latent features output by the feature encoder after the state and action have passed through it; Indicates the estimated scalar reward; This indicates a conditional dynamic predictor. and marginal dynamic predictor The corresponding parameters and Update.

[0009] Step 3: Perform decoupled joint optimization based on conditional mutual information and the regret model. After pre-training convergence, the computation control unit will unfreeze and fully activate the feature encoder. and reward decoder And the additionally introduced baseline utility network The backpropagation gradient flow is used to freeze the parameters of both dynamic prediction networks. During computation, the preference alignment loss based on the regret model is first calculated, i.e., the advantage function is calculated for each data step: in, Representing time steps The state and actions under these conditions; The parameter is The reward decoder; The parameter is State-action feature encoder; Indicates the time discount factor; The parameter is The baseline utility network acts as an environmental cost absorber. Next, the advantage function of the entire trajectory is accumulated and substituted into the Bradley-Terry probability model to calculate the correlation with the true preference label. Cross-entropy loss: in, Expressing expectations; Indicates binary preference label; These represent samples taken from a multi-source environmental preference dataset. Two trajectories; For the sigmoid function, , These represent the cumulative dominance function of the corresponding trajectory at each time step; This indicates a state-action feature encoder. Reward Decoder and baseline utility network The corresponding parameters An update is performed. Simultaneously, to accurately remove environmental noise, the frozen dynamic prediction network is invoked, and the upper bound of the variational contrast log ratio is calculated, thereby generating the conditional mutual information loss for feature adversarial penalty. in, Expressing expectations; These represent the state, action, and next state, respectively. This represents an unlabeled experience replay pool. The parameter is And the conditional dynamic predictor network has been frozen; The parameter is State-action feature encoder; The parameter is And the marginal dynamic predictor network has been frozen; The parameter is The reward decoder; Indicates the current hidden feature; This indicates a state-action feature encoder. and reward decoder Network parameters Update.

[0010] Finally, the above two losses are applied using penalty coefficients. The total loss is obtained by weighted merging: Backpropagation is then performed to minimize the total loss. Through this joint adversarial optimization, the system forces the latent sensor variables to completely preserve the agent's kinematic features relevant to the core task while eliminating underlying physical dynamic properties (such as friction and load damping). After training, a "pure task reward decoder" that is absolutely immune to any abnormal physical environment is solidified and extracted.

[0011] Step 4: Distill the underlying action policy based on expectation regression, and securely generate the final action policy for controlling the physical device using pure rewards. First, using the pure reward decoder extracted in the previous step, process the offline dataset... Each line of historical data in the dataset is re-inferred forward and assigned a scalar reward that is independent of the environment. This completely removes dynamic noise from the original data. Then, the core stage of anti-extrapolation, expected regression value assessment, begins, where the processor constructs a state-value network in memory. Target Action Value Network and policy network And perform asymmetric expected regression to calculate the loss: in It is an asymmetric expected regression loss function; The parameter is The state-value network outputs... Indicates the current state Value assessment; The parameter is The target action value network, output Indicates a state-action pair Value assessment. This is achieved by setting hyperparameters. The agent can implicitly approximate the upper limit of the value of the optimal action by evaluating only historically known safe actions, without needing to query and infer dangerous off-distribution actions as traditional algorithms do. Subsequently, the system synchronously corrects the parameters of the action value network by minimizing the mean squared error loss. in, Expressing expectations; These represent the state, action, and next state, respectively. The parameter is The target action value network, output Indicates a state-action pair Value assessment; The parameter is The state-value network outputs... Indicates the next state Value assessment; This represents the pure scalar reward relabeled by the trained reward encoder; This indicates a value network for target actions. parameters Update.

[0012] Because the value of the next state in this objective is derived from a value network that prevents extrapolation. Therefore, this step of the Q-value update also inherits the absolutely safe data manifold boundary constraints.

[0013] Finally, in the policy extraction and distribution stage, the processor uses the updated state value network to calculate the temporal difference error to update the Q network, and then guides the policy network to securely mimic historical actions with high-purity reward advantages by calculating the advantage-weighted exponential loss. in, Expressing expectations; These represent state and action, respectively. This represents an unlabeled experience replay pool. The parameter is The target action value network, output Indicates a state-action pair Value assessment; The parameter is The state-value network outputs... Indicates the state Value assessment; The parameter is The policy network outputs the policy in the state. Select action The probability of; The temperature hyperparameter in the advantage-weighted exponential loss; This indicates a policy network. parameters The update process is then performed. Once the network training has converged, the processor compiles the policy network into lightweight executable instructions, which are then directly deployed to the control unit of the target physical device for execution of safety-embodied intelligent action control.

[0014] To implement the above method, the present invention also provides an embodied agent policy training system based on preference reinforcement learning, which logically comprises four core modules that operate in concert: The data acquisition and construction module is used to acquire and construct a multi-source non-stationary environment preference dataset, including collecting historical interaction trajectories from different underlying dynamic environments, generating trajectory fragment pairs and corresponding human preference labels. The environmental dynamic prediction module, which includes a conditional dynamic predictor and a marginal dynamic predictor, is responsible for pre-detecting and learning the physical evolution laws of complex non-stationary environments, thereby providing an accurate mutual information estimation benchmark for subsequent feature decoupling. Reward Modeling Module: This module is used to perform decoupled joint optimization based on conditional mutual information and regret model to constrain the feature encoder and pure reward decoder, and extract pure task rewards that are absolutely immune to any abnormal physical environment; Offline policy training module: This module is used to perform safe underlying action policy distillation based on expectation regression, and to safely generate the final action policy for controlling the physical device using pure rewards; after the policy network training converges, the policy network is compiled into executable instructions and directly deployed to the control unit of the target physical device to execute embodied agent action control.

[0015] The system implementation process and methods are the same, and will not be repeated here.

[0016] Beneficial effects: Compared with existing technologies, this invention accurately filters out sensor noise by introducing regret models and conditional mutual information methods, avoiding pattern collapse of the neural network at the agent's control end. It also achieves cross-environment generalization capability while perfectly preserving core task features, significantly improving the success rate and convergence speed of model training. Simultaneously, it strictly restricts policy optimization to within the historical safe data manifold, completely blocking the risk of device malfunction caused by extrapolating unknown actions from the algorithmic level, providing extremely high physical operation safety guarantees for the deployment of embodied agents in complex physical environments. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the method flow according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the process and architecture for acquiring and constructing multi-source non-stationary environmental data according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the offline policy training process according to an embodiment of the present invention; Figure 4 This is a schematic diagram of the modular structure of the system according to an embodiment of the present invention. Detailed Implementation

[0018] The present invention will be further illustrated below with reference to specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading the present invention, any modifications of the present invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0019] In real life, embodied intelligent agents, such as autonomous vehicles, drone swarms, or flexible robotic arms, often face extremely harsh and unstable environmental challenges during operation. For example, vehicles encounter various weather conditions, such as sunny days, rainy days, and snowy days, causing drastic and unpredictable changes in road friction and wind resistance. When a robotic arm grasps objects of different materials, the load on its end effector and joint damping also undergo constant abrupt changes. Traditional algorithms based on cumulative rewards for preference learning mistakenly absorb and equate environmental physical resistance or assistance as the "good" or "bad" of the task itself. This leads to the complete failure of the original control strategy or even safety accidents once the dynamic parameters of the environment change. Using the system and method proposed in this invention, embodied intelligent agents can not only sensitively separate environmental noise from the core task, ensuring that they adhere to the correct task objectives even when the environment changes dynamically, but also extract a low-level motion control strategy with extremely high adaptability and absolute safety boundaries from historical data without any risky trial and error in actual operation.

[0020] A method for training embodied agent policies based on preference reinforcement learning includes the following steps: Step 1: Obtain and construct a multi-source non-stationary environment preference dataset, including collecting historical interaction trajectories from different underlying dynamic environments, generating trajectory segment pairs and corresponding human preference labels; Step 2: Train the environmental dynamics predictor so that the computational control unit can learn in advance the dynamic evolution law of the embodied intelligent agent under different physical environments; Step 3: Perform decoupled joint optimization based on conditional mutual information and regret model; extract a pure task reward decoder that is absolutely immune to any abnormal physical environment; Step 4: Based on the expectation regression, perform safe underlying action policy distillation, and use pure rewards to safely generate the final action policy for controlling the physical device; after the policy network training converges, the processor compiles the policy network into executable instructions and directly deploys them to the control unit of the target physical device to perform embodied intelligent action control.

[0021] This embodiment uses the continuous motion control task of an autonomous vehicle—a typical representative of embodied intelligent agents—under non-stable road conditions (covering multiple levels of friction changes and wind resistance interference) as a specific application scenario. For example... Figure 1 The entire workflow includes four interconnected processing stages: data construction, dynamic pre-training, feature decoupling and optimization, and security policy distillation.

[0022] In step S101, the underlying data ingestion and structuring engine is activated, specifically targeting multiple source environments with significant shifts in dynamic variables such as friction and wind resistance, to acquire and construct a multi-source non-stationary environment preference dataset. This step aims to provide a training material base containing rich physical perturbation features for subsequent feature decoupling algorithms by combining massive unsupervised trajectory sampling with qualitative preference annotations from human experts. Figure 2 As shown, the specific implementation is as follows: Step 11: Multi-source non-stationary data stream acquisition. First, historical driving records of the vehicles under various dynamic source environments are collected extensively through data acquisition interfaces deployed on the test vehicle fleet or high-fidelity physics simulators. These source environments include different operating conditions such as dry asphalt roads (high friction coefficient), slightly wet roads, and slippery water surfaces (low friction coefficient).

[0023] Step 12: State and Action Space Formatting. In the data preprocessing layer, the current observation information of the embodied agent at any discrete time step is processed into a high-dimensional vector containing the current vehicle speed, yaw rate, throttle opening feedback, and the relative distances to surrounding obstacles detected by the lidar; while the action information is processed into a continuous action command vector representing the steering wheel angle and longitudinal acceleration.

[0024] Step 13: Trajectory Segment Slicing and Preference Label Generation. These continuous time-series data are segmented according to fixed time windows to generate trajectory segment pairs of equal length. These trajectory pairs are then pushed in batches to a human review platform. Human experts (or pre-defined safety rule scripts) rely solely on intuition to assign a binary preference label to each trajectory pair based on factors such as whether the vehicle maintains lane centering and drives smoothly. This is used to indicate which trajectory better matches the mission expectations.

[0025] Step 14: Construction of Structured Dataset and Experience Replay Pool. These trajectory segments, which are mixed with a large amount of road friction noise, are packaged and encapsulated with preference labels to construct a preference dataset for subsequent use by the deep network in the central memory. Simultaneously, the trajectory is broken down, and millions of unlabeled single-step state transition records are extracted to construct an experience replay pool for exploring environmental patterns. .

[0026] In step S102, the environmental dynamic prediction module is activated, and pre-training of the dynamic prediction network is performed. The core purpose of this step is to autonomously discover and quantify the underlying physical slip and dynamic change patterns of the environment through unsupervised learning without touching the task reward, thereby providing accurate standards for removing environmental noise in subsequent steps.

[0027] Step 21: Deep Neural Network Component Initialization. On a cloud server equipped with a high-performance tensor computing unit, instantiate multiple components consisting of deep neural networks. Input layer state-action feature encoder. It is responsible for performing matrix dimensionality reduction on the high-dimensional state-action concatenation input, projecting it into a low-dimensional, compact latent feature. Simultaneously initialize the conditional dynamic predictor. and marginal dynamic predictor .

[0028] Step 22: Feature Encoder Parameter Freeze. In the current pre-training phase, to ensure the absolute objectivity of subsequent evaluation benchmarks, the network parameters of the feature encoder are forcibly frozen, blocking its gradient updates.

[0029] Step 23: Predicting the probability of physical transitions in the environment. The conditional dynamic predictor has a global perspective; it receives latent features containing all environmental information. It attempts to output the probability distribution of the next physical state; while the marginal dynamic predictor has a huge information bottleneck, as it can only receive a one-dimensional reward scalar output from the reward decoder and guess the next physical state based on it.

[0030] Step 24: Maximize Log-Likelihood Iterative Update. By continuously drawing large batches of data from the unlabeled experience replay pool, and aiming to minimize the negative log-likelihood loss, high-frequency gradient descent updates are performed on the two referee networks. The specific dynamic pre-training loss function is as follows: After tens of thousands of iterations of training, these two predictors have thoroughly mastered the slippage patterns of vehicles on road surfaces with different friction levels, forming a mathematical benchmark for measuring information content.

[0031] In step S103, the reward modeling module is activated, formally performing the core feature decoupling joint optimization based on conditional mutual information and the regret model. The value of this step lies in breaking through the "unrecognizable" deadlock caused by traditional cumulative rewards, using the advantage function to absorb environmental non-stationary fluctuations, and using the conditional mutual information penalty of the pre-trained dynamic prediction network to remove environmental dynamic noise in the latent space, ultimately distilling a pure reward model with perfect cross-scene generalization ability.

[0032] Step 31: Fully unfreeze and reconfigure network gradient channels. After confirming that the two dynamic predictors have converged and grasped the environmental patterns, the conditional dynamic predictor... and marginal dynamic predictor The parameters were completely locked. Next, the feature encoder was fully integrated. and reward decoder And the additionally introduced baseline utility network The backpropagation gradient channel enters a dense joint adversarial optimization loop.

[0033] Step 32: Calculation of Preference Alignment Loss Based on Advantage Function. The system abandons the erroneous approach of simply and crudely accumulating trajectory rewards, and instead introduces a regret model to calculate the advantage function at each step. The calculation formula is as follows: in This is the time discount factor. The advantage function cleverly utilizes the time difference predicted by the baseline utility network to forcibly absorb all value changes caused by non-stationary fluctuations. Subsequently, the processor sums the advantage functions for the entire trajectory and substitutes them into the Bradley-Terry probability model to calculate the cross-entropy preference alignment loss with the true preference label. Step 33: Calculation of adversarial penalty based on variational conditional mutual information. To completely eliminate environmental noise in the latent space of deep networks, the processor calls the previously frozen dynamic prediction network in parallel to calculate its expected log difference when predicting the probability of the next state, i.e., the upper bound loss of variational conditional mutual information: This difference is used as an extremely severe penalty loss, which is added to the overall optimization objective to put pressure on the feature encoder.

[0034] Step 34: Solidify the joint gradient descent and pure reward model. Weight and combine the two losses mentioned above to obtain the total loss formula for joint optimization: The network weights are updated using a dynamic annealing strategy (with the penalty coefficient gradually increased as training progresses). This triggers the feature encoder to perform precise information segmentation. When joint training converges, dynamic noise in the latent space is completely eliminated, while core task features such as vehicle displacement are perfectly preserved. Finally, the system extracts and solidifies a thoroughly refined, pure reward decoder as a cross-scenario task evaluator.

[0035] In step S104, the offline policy training module takes over control and executes a highly secure policy distillation based on the pure reward signal, avoiding out-of-distribution (OOD) queries. For example... Figure 3 As shown, this step aims to address the risks of physical collisions and loss of control caused by robots blindly exploring unknown actions in complex control scenarios. By using asymmetric expectation regression technology, the strategy iteration is strictly confined to a historical safe dataset, and the final output can directly obtain a robust control strategy.

[0036] Step 41: Cross-environment reward relabeling. Using the pure reward decoder extracted in the previous step, the massive historical experience replay pool is relabeled. Each frame of data in the process is re-scored through forward inference to generate a pure scalar reward. This process removes the environmental dynamics information attached to historical data, assigning each driving record a correct guidance score after removing the dynamic influences.

[0037] Step 42: Simplify the offline policy network architecture. First, construct the network framework for performing offline policy optimization, which includes a state-value network. Target Action Value Network and the policy network that is ultimately responsible for outputting and executing actions. .

[0038] Step 43: Asymmetric Expectation Regression Value Extrapolation Evaluation. To completely eliminate the catastrophic consequences of traditional reinforcement learning algorithms evaluating unknown actions, a special asymmetric expectation regression operation is performed during the training of the state-value network. This is achieved by setting an expectation quantile hyperparameter. The state-value network is trained using a specific asymmetric mean squared error loss: This ingenious mathematical design allows the network to evaluate state quality without querying any out-of-distribution (OOD) unknown actions, instead implicitly approximating the upper bound of the optimal value within the known safe data manifold. It also utilizes temporal difference errors to optimize the action value network.

[0039] Step 44: Advantage-Weighted Policy Extraction. After accurately evaluating the state value, the processor synchronously corrects the action value network using the updated value network, thus entering the advantage-weighted policy extraction sprint phase. The processor calculates the exponential weight difference between action value and state value, using behavioral cloning constraints to guide the policy network to replicate and imitate safe driving behaviors with extremely high pure reward advantages. Its policy update loss function is as follows: in This is the temperature coefficient.

[0040] Step 45: After multiple rounds of training and convergence, the policy execution network, with its extremely small volume and fast response, is compiled into lightweight instructions and seamlessly deployed to the onboard edge control platform of autonomous vehicles. In real-world applications, when a vehicle suddenly enters an unknown road surface with entirely new dynamic parameters, thanks to the clean feature representation decoupled from the pre-decoupling and the strict action safety constraints provided by subsequent offline algorithms, the vehicle can adaptively complete safe and smooth driving operations after simple fine-tuning.

[0041] Based on the complete workflow of the embodied intelligent action control method described above, this application also provides an embodied intelligent agent preference reinforcement learning system based on conditional mutual information. (See also...) Figure 4 This diagram illustrates the modular structure of the preference reinforcement learning system provided in this application embodiment. In conjunction with the aforementioned specific example of autonomous driving, the system's logical architecture is primarily composed of the following four core modules working closely together: The data acquisition and construction module is responsible for interfacing with the radar and control system, collecting driving data of the vehicle on dry or wet roads, and accepting safety preference annotations from human experts to construct a training dataset containing rich physical disturbances (corresponding to execution step S101).

[0042] The environmental dynamic prediction module has a built-in feature encoder including a multilayer perceptron (MLP) and two dynamic prediction networks. Its function is to learn the physical laws of friction such as vehicle slippage in advance on the cloud server using unlabeled data, so as to establish a standard for subsequent noise filtering (corresponding to execution step S102).

[0043] The reward modeling module receives data and referee benchmarks from the preceding module. Its built-in calculation unit aligns preferences using the regret model formula and accurately removes dynamic noise caused by weather and road friction using conditional mutual information loss. Finally, it extracts and outputs a cross-environment universal "pure reward decoder" (corresponding to execution step S103).

[0044] The offline policy training module receives the pure reward signal and performs asymmetric expectation regression calculations using an architecture composed of a state-value network and a double-Q network. This module eliminates the possibility of dangerous actions such as the vehicle attempting to overstep the boundary by turning the steering wheel from an algorithmic perspective. Finally, the refined high-safety policy is packaged and distributed to the edge computing platform (corresponding to execution step S104).

Claims

1. A method and system for training embodied agent policies based on preference reinforcement learning, characterized in that, Includes the following steps: Step 1: Obtain and construct a multi-source non-stationary environment preference dataset, including collecting historical interaction trajectories from different underlying dynamic environments, generating trajectory segment pairs and corresponding human preference labels; Step 2: Train the environmental dynamic predictor so that the computational control unit can learn in advance the dynamic evolution law of the embodied intelligent agent under different physical interaction media; Step 3: Perform decoupled joint optimization based on conditional mutual information and regret model; extract a pure task reward decoder that is absolutely immune to any physical environment disturbance; Step 4: Based on the expectation regression, perform safe underlying action policy distillation, and use pure rewards to safely generate the final action policy for controlling the physical device; after the policy network training converges, the processor compiles the policy network into executable instructions and directly deploys them to the control unit of the target physical device to execute embodied intelligent agent control.

2. The embodied agent policy training method based on preference reinforcement learning according to claim 1, characterized in that, In step 1, the embodied intelligent agent collects historical interaction trajectories from different underlying dynamic environments. These historical interaction trajectories include the historical interaction trajectory of autonomous driving, the motion trajectory of a drone / robot, or the spatial motion trajectory of a flexible robotic arm grasping an object.

3. The embodied agent policy training method based on preference reinforcement learning according to claim 1, characterized in that, In step 1, historical interaction data streams from multiple source environments with differing underlying dynamic parameters are received via a sensor data acquisition interface; these data streams are then segmented and formatted by the processor to include state. With action long trajectory segments After pairing, the binary human preference labels are evaluated by human experts or based on preset rules. The label is used to indicate which trajectory better matches the task expectation; trajectory segments with accompanying environmental non-stationary features are combined with preference labels to construct a multi-source environmental preference dataset in memory. Simultaneously, all single-step transition data are extracted and used to construct an unlabeled experience replay pool. .

4. The embodied agent policy training method based on preference reinforcement learning according to claim 1, characterized in that, In step 2, during the training of the environmental dynamic predictor, four deep neural networks are first initialized, namely state-action feature encoders. Reward Decoder Conditional dynamic predictor and marginal dynamic predictor During the training phase, the network parameters of the feature encoder and reward decoder are first frozen, and the state-action pairs are encoded as latent features. And receive the estimated reward ; Subsequently, from the preference dataset Data is extracted in medium batches, and two predictors are trained independently by maximizing the log-likelihood probability; specifically, the conditional dynamic predictor is based on complete latent features. The system learns to predict the probability distribution of the next state, while the marginal dynamic predictor learns to predict the next state solely based on the decoded scalar reward; this pre-training process aims to minimize the following loss function: in, Expressing expectations; These represent the state, action, and next state, respectively. This represents an unlabeled experience replay pool. The parameter is Conditional dynamic predictor; The parameter is Marginal dynamic predictor; This represents the latent features output by the feature encoder after the state and action have passed through it; Indicates the estimated scalar reward; This indicates a conditional dynamic predictor. and marginal dynamic predictor The corresponding parameters and Update.

5. The embodied agent policy training method based on preference reinforcement learning according to claim 1, characterized in that, In step 3, after the environmental dynamic predictor training converges, the computation control unit will unfreeze and fully activate the state-feature encoder. and reward decoder and the additionally introduced baseline utility network The backpropagation gradient flow is used to freeze the parameters of both the conditional dynamic predictor and the marginal dynamic predictor networks. During computation, the preference alignment loss based on the regret model is first calculated, i.e., the advantage function is calculated for each data step. in, Representing time steps The state and actions under these conditions; The parameter is The reward decoder; The parameter is State-action feature encoder; Indicates the time discount factor; The parameter is The baseline utility network acts as an environmental cost absorber; then, the advantage function of the entire trajectory is accumulated and substituted into the Bradley-Terry probability model to calculate the result relative to the true preference label. Cross-entropy loss: in, Expressing expectations; Indicates binary preference label; These represent samples taken from a multi-source environmental preference dataset. Two trajectories; It is the sigmoid function; , These represent the cumulative dominance function of the corresponding trajectory at each time step; This indicates a state-action feature encoder. Reward Decoder and baseline utility network The corresponding parameters Update. At the same time, the frozen dynamic prediction network is invoked to calculate the upper bound of the variational contrast log ratio, thereby generating the conditional mutual information loss for feature adversarial penalty: in, Expressing expectations; These represent the state, action, and next state, respectively. This represents an unlabeled experience replay pool. The parameter is And the conditional dynamic predictor network has been frozen; The parameter is State-action feature encoder; The parameter is And the marginal dynamic predictor network has been frozen; The parameter is The reward decoder; Indicates the current hidden feature; This indicates a state-action feature encoder. and reward decoder Network parameters Update; Finally, the above two losses are applied using penalty coefficients. The total loss is obtained by weighted merging: Then perform backpropagation based on minimizing the total loss.

6. The embodied agent policy training method based on preference reinforcement learning according to claim 1, characterized in that, In step 4, firstly, the pure task reward decoder extracted in the previous step is used to process the offline dataset. Each line of historical data in the dataset is re-inferred forward and assigned a scalar reward that is independent of the environment. This completely removes dynamic noise from the original data; subsequently, the expected regression value assessment stage begins, where the processor constructs a state value network in memory. Target Action Value Network and policy network And perform asymmetric expected regression to calculate the loss. By setting hyperparameters, the agent can implicitly approximate the upper limit of the value of the optimal action by only evaluating historically known safe actions. Subsequently, the parameters of the action value network are simultaneously corrected by minimizing the mean squared error loss; Finally, in the policy extraction and distribution stage, the processor uses the updated state value network to calculate the temporal difference error to update the Q network, and then guides the policy network to reliably imitate historical actions with high-purity reward advantages by calculating the advantage-weighted exponential loss.

7. A system for training embodied agent policies based on preference reinforcement learning, characterized in that, Includes the following modules: The data acquisition and construction module is used to acquire and construct a multi-source non-stationary environment preference dataset, including collecting historical interaction trajectories from different underlying dynamic environments, generating trajectory fragment pairs and corresponding human preference labels. The environmental dynamic prediction module, which includes a conditional dynamic predictor and a marginal dynamic predictor, is responsible for enabling the computational control unit to grasp the dynamic evolution of the embodied intelligent agent under different physical environments in advance. Reward Modeling Module: This module is used to perform decoupled joint optimization based on conditional mutual information and regret model to constrain the feature encoder and pure reward decoder, and extract pure task rewards that are absolutely immune to any abnormal physical environment; Offline policy training module: This module is used to distill the underlying action policy based on expectation regression, and to safely generate the final action policy for controlling the physical device using pure rewards. After the policy network training converges, the policy network is compiled into executable instructions and directly deployed to the control unit of the target physical device to perform embodied intelligent action control.

Images