A reinforcement learning cross-strategy evaluation method based on a conditional value function

Abstract

The application provides a reinforcement learning cross-strategy evaluation method based on a conditional value function, and belongs to the technical field of reinforcement learning. A policy feature vector is generated based on a key state set through an action probability; a conditional value function is reconstructed with the policy feature as a condition, and a cross-strategy evaluation model is constructed; the cross-strategy evaluation model comprises a policy network and a conditional value network; the cross-strategy evaluation model is collaboratively pre-trained; based on an experience data set collected according to a historical policy, the cross-strategy evaluation model is trained offline with the objective of minimizing the mean square error of a predicted value and an actual discount reward; and a state value distribution of different policies is fitted through supervised learning. The method breaks through the fixed limitation of the traditional value evaluation and a specific policy by reconstructing the value function into a conditional value function, and fusing three-dimensional inputs of state, time step and policy features to construct the cross-strategy evaluation model.

Description

Technical Field

[0001] This invention relates to the field of reinforcement learning technology, and in particular to a reinforcement learning cross-policy evaluation method based on conditional value functions. Background Technology

[0002] Reinforcement learning, as one of the core application technologies of intelligent decision-making systems, directly affects the adaptability and safety of agents in complex environments through its policy performance. In this regard, reinforcement learning algorithms based on the Actor-Critic framework have become the mainstream solution. Its policy network generates actions, and the value network evaluates the value of states, forming a closed-loop optimization mechanism of "policy generation - value evaluation".

[0003] Before deploying a strategy, its performance needs to be effectively evaluated. Existing methods typically rely on sampling real interactions between the agent and the environment to evaluate the strategy's merits based on accumulated rewards. While intuitive, this method suffers from long verification cycles, high interaction costs, and difficulty in supporting repeated testing, failing to meet the practical needs of rapid strategy iteration. The value network in the AC framework algorithm inherently possesses state value prediction capabilities, theoretically usable for policy performance estimation. However, in practical applications, directly using an existing value network to evaluate other strategies has significant limitations: Strong coupling between the strategy and the value network leads to a lack of evaluation generalization: the trained value network is highly bound to its corresponding strategy, and its output is only valid for estimating the state value of that strategy. When the evaluation object changes to other strategies (e.g., strategies with different training periods, hyperparameter configurations, or structures), the original value network's predictions will show significant deviations, lacking cross-strategy generalization evaluation capabilities. Resource burden from repeated interactions: due to the limitations of existing value networks, performance evaluation of new strategies requires re-interaction with the environment to collect state trajectory data. This process consumes significant time and computational resources in physical simulations or actual deployments. The efficiency of horizontal strategy comparison is low: In multi-strategy comparison scenarios (such as hyperparameter tuning and algorithm selection), existing methods need to perform a complete environmental interaction evaluation for each strategy independently, and its time complexity increases linearly with the number of strategies.

[0004] Therefore, it is urgent to reconstruct value network capabilities and build a generalized strategy value assessment model to achieve efficient performance evaluation across strategies. Summary of the Invention

[0005] In view of this, this invention provides a reinforcement learning cross-policy evaluation method based on conditional value functions. By extracting policy features and reconstructing conditional value functions, a cross-policy evaluation model capable of evaluating the performance of any target policy is constructed, realizing the mapping from the policy feature space to the value space. This method first designs a policy feature extraction mechanism based on the action probability distribution under key states, compressing the high-dimensional policy space into low-dimensional feature vectors. Second, the value function is extended to a conditional value function, explicitly introducing policy features and time steps as input conditions to construct the cross-policy evaluation model and co-training it with the policy network. Finally, the model is optimized through offline training, enabling it to accurately predict the state value of any policy based on environmental state, time step, and policy features. This method decouples the dependency between policy and value evaluation and effectively utilizes historical experience data, providing a new approach for optimizing the security and efficiency of policy deployment.

[0006] Therefore, the present invention provides the following technical solution: A reinforcement learning cross-policy evaluation method based on conditional value functions includes: Generate policy feature vectors based on action probabilities on the key state set; A cross-policy evaluation model is constructed by reconstructing the conditional value function based on policy characteristics; the cross-policy evaluation model includes: a policy network and a conditional value network; Collaborative pre-training of cross-policy evaluation models; Based on empirical datasets collected from historical policies, the cross-policy evaluation model is trained offline with the goal of minimizing the mean squared error between predicted value and actual discounted return; supervised learning is used to fit the state value distribution of different policies.

[0007] Furthermore, the step of generating policy feature vectors based on action probabilities on the key state set includes: The probability distribution of actions of the policy on the key state set is represented as the policy feature vector:

[0008] In the formula, For strategy feature vectors; For strategy; In strategy State-based Take action The probability of; This is a critical state; For the action space dimension.

[0009] Furthermore, the key state points include: Initial state, right critical state, left critical state, and stable state.

[0010] Furthermore, the conditional value function includes:

[0011] in, For state, For time steps, The policy feature vector, i.e., the condition; function Indicates that the feature is Under the strategy, from the current state At time step Initial expected cumulative return.

[0012] Furthermore, the cross-strategy evaluation model includes: The input to the conditional value function evaluation model is: environmental state. Time step Strategy characteristics ; The output of the conditional value function evaluation model is: conditional value prediction. ; Policy Network It is about strategy Neural network approximation, conditional value network It is for the value function The neural network approximation; The environmental state, time step, and policy features are fused using a nonlinear transformation, and the fused feature vector is output as follows:

[0013] In the formula, For feature fusion function; Environmental state characteristics, representing the environmental state The eigenvectors obtained after nonlinear transformation; The time step feature represents the time step. The eigenvectors obtained after nonlinear transformation; For policy characteristics, represent policy parameters The eigenvectors obtained after nonlinear transformation.

[0014] Furthermore, the collaborative pre-training of the cross-policy evaluation model includes: Current state observed Make decisions based on the current policy network: And instruct the agent to perform actions. Simultaneously extract strategy feature vectors ; Rewards from the environment and the state at the next moment ; Conditional value network output value score: and ; Calculate the time-difference objective and time-difference error: and ,in Discount factor; For time-series difference objectives; This refers to timing difference error; Score the value at time t; Value score at time t+1; For policy networks; For conditional value networks; Update conditional value network parameters: ,in The learning rate of the conditional value network; Update policy network parameters: ,in is the learning rate of the policy network.

[0015] Furthermore, the offline training of the cross-policy evaluation model includes:

[0016] In the formula, The number of samples in the training batch. For learning rate, For the conditional value network prediction function, For the first The environmental state of each sample For the first The time step of each sample For the first The strategy feature vector of each sample, For the first The actual discount return for each sample; For parameters before the update, These are the updated parameters.

[0017] Advantages and positive effects of the present invention: This method reconstructs the value function into a conditional value function and integrates three-dimensional inputs—state, time step, and policy features—to construct a cross-policy evaluation model, overcoming the inherent limitations of traditional value evaluation bound to specific policies. A policy feature extraction mechanism is designed to map the high-dimensional policy space to a low-dimensional feature space, providing transferable feature representations for cross-policy evaluation. The cross-policy evaluation model is trained online and continuously improved and optimized in conjunction with policy training. Simultaneously, cross-policy experience data is collected to perform supervised offline training on the reconstructed cross-policy evaluation model, ensuring its fitting effect on different policy value distributions. Compared to traditional methods, this invention overcomes the policy-value binding limitation, significantly improving policy evaluation efficiency and cross-policy comparability. It provides a new, efficient, and reliable intelligent evaluation paradigm for reinforcement learning policy deployment, effectively solving the core problems of high policy verification costs and long cycles in industrial scenarios. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 The flowchart shows a reinforcement learning cross-policy evaluation method based on conditional value functions. Figure 2 This is an architecture diagram for a reinforcement learning cross-policy evaluation method based on conditional value functions. Figure 3 The graph shows the results of the cross-strategy performance evaluation. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] This invention provides a reinforcement learning cross-policy evaluation method based on conditional value functions, comprising: Generate policy feature vectors based on action probabilities on the key state set; Reconstruct the conditional value function based on policy characteristics to build a cross-policy evaluation model; Collaborative pre-training of cross-policy evaluation models; Based on empirical datasets collected from historical policies, the cross-policy evaluation model is trained offline with the goal of minimizing the mean squared error between predicted value and actual discounted return; supervised learning is used to fit the state value distribution of different policies.

[0023] Step A: Generate policy feature vectors based on action probabilities on the key state set; Step A1: Construct the key state set: For different environmental characteristics and task objectives, specific state points that have a decisive impact on strategy decisions in that environment are selected, including initial, critical, and stable states, to construct a key state set. The key state set is represented as follows:

[0024] In the formula, It is the number of critical states.

[0025] Step A2: Generate policy feature vectors based on action probabilities; Given a strategy Its strategy feature vector The action probability distribution over the key state set:

[0026] In the formula, In strategy Below, based on state Take action The probability, This is a critical state. For the action space dimension.

[0027] By constructing a key state set covering key regions of the policy decision space and extracting its action probability distribution to generate policy feature vectors, this method maps the high-dimensional policy space to a low-dimensional feature space, providing a transferable feature representation for cross-policy evaluation.

[0028] Step B: Reconstruct the conditional value function based on policy characteristics to build a cross-policy evaluation model; To achieve cross-policy evaluation, the traditional value function is reconstructed and extended into a conditional value function conditioned on policy features. The mathematical expression of the reconstructed conditional value function is as follows, which depends not only on the state and time step but also on the policy features:

[0029] in, For state, For time steps, For strategy The characteristics, or conditions, of a policy. The specific meaning of the conditional value function is that, for a given policy characteristic... ,function Indicates that the feature is Under the strategy, from the current state At time step Initial expected cumulative return.

[0030] Traditional value function Corresponding to the reconstructed conditional value function under a fixed strategy (thus fixing the strategy characteristics) And ignore the time step Special cases:

[0031] conditional value function We obtain a networked evaluation model based on conditional value functions by performing network approximation. ,in These are network parameters, and their specific structure is as follows: Input layer reconstruction: environment state Time step Strategy characteristics ; Output layer definition: Conditional value prediction Characterized by the feature Under the strategy, from the current state At time step Initial expected cumulative return; The environmental state, time step, and policy features are fused using a nonlinear transformation, and the fused feature vector is output as follows:

[0032] In the formula, For feature fusion function; Environmental state characteristics, representing the environmental state The eigenvectors obtained after nonlinear transformation; The time step feature represents the time step. The eigenvectors obtained after nonlinear transformation; For policy characteristics, represent policy parameters The eigenvectors obtained after nonlinear transformation.

[0033] In traditional reinforcement learning, each policy Each has its corresponding state value function Therefore, evaluating multiple different strategies requires learning multiple value function network evaluation models. This invention introduces a conditional value function, treating the strategy as a conditional variable, resulting in a generalized value function form. This generalizes the evaluation object from "a specific strategy" to "any target strategy," enabling a fixed evaluation model to meet the evaluation needs of different strategies and achieving cross-strategy performance evaluation.

[0034] Step C: Co-train the cross-policy evaluation model; Policy Network It is about strategy Neural network approximation, conditional value network It is for the value function The neural network approximation, i.e., the cross-policy evaluation network model. Let the parameters of the current policy network be... The parameters of a conditional value network are Without affecting policy optimization training, the cross-policy evaluation model is collaboratively updated by performing online training and parameter updates on both models to obtain new network parameters. and The specific implementation steps are as follows; Current state observed Make decisions based on the current policy network: And make the intelligent agent perform actions. Simultaneously, extract the strategy feature vector based on step A. ; Rewards from the environment and new status ; Conditional value network output value score: and ; Calculate the time-difference objective and time-difference error: and ,in Discount factor; Update conditional value network parameters: ,in The learning rate of the conditional value network; Update policy network parameters: ,in is the learning rate of the policy network.

[0035] Step D: Based on the empirical dataset collected from historical strategies, train the cross-strategy evaluation model offline with the goal of minimizing the mean squared error between the predicted value and the actual discounted return. The cross-policy evaluation model is trained offline using an empirical dataset collected from historical policies. Supervised learning is used to accurately fit the state-value distribution of different policies, thereby improving the accuracy and generalization ability of cross-policy evaluation.

[0036] The cross-policy experience dataset is derived from trajectory information during the online training phase of the policy and includes four-dimensional data: environment state. Time step Strategy characteristics Real discounts and rewards This constitutes a quadruple dataset. The actual discount reward comes from the actual discount reward obtained by the agent through interaction with the environment during the online training process; The fine-tuned cross-policy evaluation model maintains the same main architecture as the online training phase, only allowing for optimization of network weight parameters. State, time step, and policy features are used as three-dimensional fusion inputs, and the output is the predicted state value. ; The network parameters are updated with the objective of minimizing the mean squared error between the predicted value and the actual discounted return.

[0037] In the formula, The number of samples in the training batch. For learning rate, For the conditional value network prediction function, For the first The environmental state of each sample For the first The time step of each sample For the first The strategy feature vector of each sample, For the first The actual discount return for each sample.

[0038] Step E: Performance evaluation of the target strategy; Load the cross-policy evaluation model trained offline; Initialize the evaluation environment: Use a fixed initial state as the benchmark evaluation scenario across policies to ensure the consistency of evaluation conditions; Load the strategy model to be evaluated The strategy models to be evaluated are derived from the strategy models saved during the online training process in each round. Construct the key state dataset according to step A. And extract the corresponding strategy feature vector. ; With a fixed initial evaluation environment, the agent interacts with the environment to obtain actual discounted returns as a conditional value function to evaluate the model's predictions for comparative verification. Input the state sequence, time step sequence, and policy feature vector into the cross-policy evaluation model; Output the state-value sequence related to the time step of the corresponding policy: ; Cross-policy performance evaluation is achieved by comparing the value prediction curves of different strategies under the same state sequence.

[0039] Example This embodiment provides a reinforcement learning cross-policy evaluation method based on conditional value functions, combined with... Figures 1-3 As shown, taking the CartPole-v0 environment as an example, CartPole-v0 is a classic reinforcement learning environment whose goal is to control the movement of a car to keep the pole vertical. The environment state space contains 4-dimensional continuous variables: car position, velocity, pole angle, and angular velocity; the action space consists of two discrete actions: moving left (0) or moving right (1). A reward of +1 is obtained for each time step, with a maximum time step threshold of 200. The specific implementation includes: Step A; Strategy Feature Extraction; Step A1: Construct the key state set; For the CartPole-v0 environment, four key state points were selected: Initial state: (Centered and stationary state); Right-hand critical state: (Critical point of rightward tilt of the rod); Left-hand critical state: (Critical point of leftward tilt of the rod); steady state: (The rod is rotating at a constant speed). The critical state set is represented as:

[0040] Step A2: Generate policy feature vectors based on action probabilities; Given the strategy to be evaluated Calculate the action probability distribution for each critical state: , ; , ; , ; , ; Policy feature vector: .

[0041] Step B: Construct an evaluation model for the conditional value function; Reconstruct the value function into a conditional value function, and establish a network evaluation model based on the conditional value function: Policy networks, including: A fully connected layer 1 with dimensions [4, 64] and ReLU activation function; Fully connected layer 2 with dimensions [64,2], and the activation function is Softmax; Conditional value networks include: State feature extraction includes a fully connected layer with ReLU activation function, a fully connected layer 1 with dimensions [4, 64], and a fully connected layer 2 with dimensions [64, 64]. The time-step processing includes a fully connected layer with ReLU activation function, a fully connected layer 1 with dimensions [1, 16], and a fully connected layer 2 with dimensions [16, 16]. Policy feature processing includes fully connected layers with ReLU activation function, fully connected layer 1 with dimensions [1, 32], and fully connected layer 2 with dimensions [32, 32]. Feature fusion includes a fully connected layer with ReLU activation function, a fully connected layer 1 with dimensions [64+16+32,64], and a fully connected layer 2 with dimensions [64,1].

[0042] In time step (For example, with a maximum step size of 200); Environmental conditions: ; Strategy characteristics: .

[0043] Feature fusion and value output: The state path, time step path, and policy path employ fully connected layers to achieve feature fusion. ; Output state value prediction: .

[0044] Step C: Co-training of the cross-policy evaluation model This step achieves co-optimization of the cross-policy evaluation model and the policy network during the online training phase. The goal of co-training is to minimize the TD error of the cross-policy evaluation model and update the policy network to maximize the expected return through policy gradient ascent. The network training parameters are set as shown in Table 1: Table 1

[0045] Step D: Evaluate the offline training of the model The cross-policy experience dataset collected during step C, the online collaborative training phase, is shown in Table 2. The discount factor is used in the discount reward calculation. : Table 2

[0046] Using a single time step data point as an example, this demonstrates the fine-tuning process of the conditional value function evaluation model: enter: , , ; Value prediction output: ; loss: ; After 500 iterations, the average loss can be reduced to around 0.10.

[0047] Step E: Performance evaluation of the target strategy; The policy models of four typical stages in the online training process of step C are selected as the target policies to be evaluated: the policy model of round 5 as the initial policy, the policy model of round 50 as the early policy, the policy model of round 120 as the mid-term policy, and the policy model of round 200 as the late policy. The policy models are evaluated based on the conditional value function.

[0048] Cross-strategy evaluation results as follows Figure 3 As shown, the statistical results are presented in Table 3: Table 3

[0049] Strategy performance analysis: For the initial strategy, the initial value of conditional value prediction is 7.50, the time step is only 8 steps, and the mean square error between the predicted value and the actual value is 0.00068, which indicates that the strategy is underperforming.

[0050] For the early strategy, conditional value prediction started at 25.75, and the trajectory lasted for 36 steps. The mean squared error between the predicted and actual values ​​was 0.0021, indicating an improvement in performance but still with limitations.

[0051] For the mid-term strategy, the initial conditional value prediction reached 42.16, and its trajectory terminated at 92 steps. This represents a significant performance improvement compared to the early stages; however, the stability of the strategy still needs improvement. The mean squared error between the predicted and actual values ​​was 0.0054.

[0052] For the later-stage strategy, the initial conditional value prediction reached 49.16, and its trajectory persisted for a maximum of 200 steps, indicating that the strategy achieved optimal performance. Furthermore, the mean squared error between the predicted and actual values ​​was 0.0015, demonstrating the accuracy of the predicted value output by the cross-strategy evaluation model.

[0053] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A reinforcement learning cross-policy evaluation method based on conditional value functions, characterized in that, include: Generate policy feature vectors based on action probabilities on the key state set; Reconstruct the conditional value function based on policy characteristics to build a cross-policy evaluation model; The cross-policy evaluation model includes: a policy network and a conditional value network; Collaborative pre-training of cross-policy evaluation models; Based on empirical datasets collected from historical policies, the cross-policy evaluation model is trained offline with the goal of minimizing the mean squared error between predicted value and actual discounted return; supervised learning is used to fit the state value distribution of different policies.

2. The method according to claim 1, characterized in that, The generation of policy feature vectors based on action probabilities on the key state set includes: The probability distribution of actions of the policy on the key state set is represented as the policy feature vector: In the formula, For strategy feature vectors; For strategy; In strategy State-based Take action The probability of; This is a critical state; For the action space dimension.

3. The method according to claim 2, characterized in that, The key state points include: Initial state, right critical state, left critical state, and stable state.

4. The method according to claim 1, characterized in that, The conditional value function includes: in, For state, For time steps, The policy feature vector, i.e., the condition; function Indicates that the feature is Under the strategy, from the current state At time step Initial expected cumulative return.

5. The method according to claim 1, characterized in that, The cross-strategy evaluation model includes: The input to the conditional value function evaluation model is: environmental state. Time step Strategy characteristics ; The output of the conditional value function evaluation model is: conditional value prediction. ; Policy Network It is about strategy Neural network approximation, conditional value network It is for the value function The neural network approximation; The environmental state, time step, and policy features are fused using a nonlinear transformation, and the fused feature vector is output as follows: In the formula, For feature fusion function; Environmental state characteristics, representing the environmental state The eigenvectors obtained after nonlinear transformation; The time step feature represents the time step. The eigenvectors obtained after nonlinear transformation; For policy characteristics, represent policy parameters The eigenvectors obtained after nonlinear transformation.

6. The method according to claim 1, characterized in that, The collaborative pre-training of the cross-policy evaluation model includes: Current state observed Make decisions based on the current policy network: And instruct the agent to perform actions. Simultaneously extract strategy feature vectors ; Rewards from the environment and the state at the next moment ; Conditional value network output value score: and ; Calculate the time-difference objective and time-difference error: and ,in Discount factor; For time-series difference objectives; This refers to timing difference error; Score the value at time t; Value score at time t+1; For policy networks; For conditional value networks; Update conditional value network parameters: ,in The learning rate of the conditional value network; Update policy network parameters: ,in is the learning rate of the policy network.

7. The method according to claim 1, characterized in that, The offline training of the cross-policy evaluation model includes: In the formula, The number of samples in the training batch. For learning rate, For the conditional value network prediction function, For the first The environmental state of each sample For the first The time step of each sample For the first The strategy feature vector of each sample, For the first The actual discount return for each sample; For parameters before the update, These are the updated parameters.

Images