A controllable strategy generation method based on a discrete absorption Markov decision model
By constructing a family of single-parameter convex combination policies and absorption chain equations, and combining them with the guardian Newton method, an analytical target policy is generated, which solves the problem of unstable policy generation in the absorption Markov decision-making process and achieves efficient and interpretable success rate adjustment.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- DALIAN MARITIME UNIVERSITY
- Filing Date
- 2026-01-28
- Publication Date
- 2026-06-05
AI Technical Summary
Existing reinforcement learning methods struggle to construct a continuously adjustable policy space in absorbent Markov decision-making processes, lack analytical mappings, resulting in unstable success rate regulation, and rely heavily on interactive training to generate policies that meet requirements.
A continuous policy family is constructed using a single-parameter convex combination. The analytical derivatives of the starting success rate and weight parameters are established based on the absorption chain equation. The guard Newton method is used to solve for the weight parameters of the target success rate within a bounded interval, generating an analytical target policy.
It achieves efficient generation of strategies that meet target success rates without requiring extensive interactive training. It is computationally efficient and has well-defined parameters, making it suitable for interpretable tuning in path planning and high-risk tasks.
Smart Images
Figure CN122154818A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of reinforcement learning technology, and in particular to a method for generating controllable policies based on a discrete absorptive Markov decision model. Background Technology
[0002] Reinforcement learning is an important method for autonomous decision-making by intelligent agents. The quality of the policy directly affects the stability, security, and task completion ability of the system in complex environments. In many tasks with termination outcomes, the absorbent Markov decision process is a commonly used modeling framework. The policy ultimately guides the agent to either a successful or unsuccessful state; therefore, the success rate becomes a core indicator for measuring policy performance. In practical systems, the success rate of the policy is usually controllably adjusted based on factors such as task objectives and risk preferences to achieve interpretable and controllable decision-making behavior.
[0003] Traditional reinforcement learning methods (such as Q-learning and policy gradient) primarily rely on environmental interactions, with cumulative reward as the optimization objective. In absorptive tasks, these methods can only indirectly influence the success rate through reward signals; the learning process essentially optimizes long-term rewards rather than the absorptive probability itself, making it difficult to establish a directly controllable success rate adjustment mechanism. Furthermore, policy performance changes slowly with training iterations and is influenced by hyperparameters, exploration strategies, and network structure, exhibiting discrete and non-continuously adjustable characteristics, making it difficult to accurately generate policies with a specified success rate. In addition, reinforcement learning policies are mostly implicitly represented by neural networks, lacking a clear analytical structure, making it difficult to establish a mathematical relationship between policy parameters and success rate. Due to the lack of an analytically solvable foundation, existing methods struggle to use deterministic numerical algorithms to inversely calculate the target success rate and to quickly generate satisfactory policies without increasing interactive sampling. In industrial control, high-risk task planning, and adaptive systems that emphasize reliability and reproducibility, this approach, relying on extensive interactive training and experience-driven optimization, fails to meet the requirements for controllable policy performance. Therefore, existing reinforcement learning has obvious limitations in terms of success rate control: it lacks a continuously adjustable policy structure, lacks an analytical mapping between success rate and policy, and policy generation depends on training and is difficult to reproduce rigorously.
[0004] There is an urgent need for a mechanism that can construct a continuously adjustable policy space in discrete absorptive Markov decision-making processes and achieve controllable generation of success rates using mathematical models, so as to meet the requirements of intelligent decision-making systems for interpretability and performance controllability. Summary of the Invention
[0005] In view of this, the present invention provides a controllable policy generation method based on a discrete absorptive Markov decision model. In the Markov decision process with absorptive states, two sets of teacher policies are selected, and a continuous policy family characterized by a single weight parameter is constructed using a single-parameter convex combination at each non-termination state, making the policy space range between... The initial success rate is continuously adjustable. Subsequently, based on the absorption chain equation, the starting success rate is expressed as a function of the weight parameters, and its analytical derivative is derived, establishing an equation between the target success rate and the weight parameters. The Guardian Newton method is used to solve for the weights satisfying the given target success rate within a bounded interval, and the corresponding Markov policy is synthesized accordingly. Finally, the obtained policy is validated offline through Monte Carlo simulation, achieving an analytical mapping from the target success rate to the policy space. This facilitates precise, interpretable, and engineering-feasible adjustment of policy performance, thereby addressing the problems in absorption tasks such as the difficulty in directly modeling and continuously adjusting the success rate, the lack of analytical mapping, and the reliance on extensive interactive training for policy generation, which is difficult to rigorously reproduce.
[0006] Therefore, the present invention provides the following technical solution:
[0007] A method for generating controllable policies based on a discrete absorber Markov decision model includes: The task is modeled as an absorbent Markov decision process with success and failure states; For an absorptive Markov decision process, construct a family of continuous, single-parameter strategies between two teacher strategies; based on this family of strategies, express the starting success rate as a single-parameter function. The target weight is obtained by solving the success rate function within a preset interval; a target strategy is generated based on the target weight.
[0008] Furthermore, the absorbent Markov decision process includes: Discrete state space Action set State transition probability For any state-action pair ,have ; The states are divided into a set of non-terminating states, denoted as T, and a set of absorbing states, denoted as S; The set of absorption states includes: successful states denoted as G and failed states denoted as H; For any absorption state and any action , ; Starting state And based on the transition submatrix between non-termination states Calculate the absorption probability matrix : For any given policy Establish the transition probability matrix and absorption vector ; in, This indicates the transition from a non-terminating state under a given policy. Transition to non-termination state The probability, Table of states One-step transition to successful state The probability, Denotes the success rate vector of the non-terminating subspace, which satisfies the absorption chain equation. .
[0009] Furthermore, the construction of a single-parameter continuous policy family between two teacher strategies includes: ,
[0010] in, These are scalar weight parameters; It is in a non-terminating state; As a first-teacher strategy, For the second teacher strategy; Construct a family of single-parameter policies for single-parameter convex combinations of the first and second teacher policies.
[0011] Furthermore, based on the aforementioned strategy family, the starting success rate is represented as a single-parameter function, including:
[0012]
[0013] in, Transition matrix over non-terminating state subspace; Absorption vector.
[0014] Furthermore, the step of solving the success rate function within a preset interval to obtain the target weight includes: Based on the success rate function within the preset interval Establish the objective equation ; The guardian Newton method is used to iteratively solve the objective equation.
[0015] Furthermore, the iterative solution of the objective equation using the guardian Newton method includes: Initially set the squeeze interval Select initial value .
[0016] In the In this iteration, candidate points are calculated based on Newton's formula:
[0017] The candidate points are used to estimate the positions of the zeros of the equation based on the first-order approximation of the current point. If the derivative Existence and , For derivative tolerance, and the candidate points satisfy: ,
[0018] Then accept Newton's step, let ; If the condition is not met, then a two-step approach is used: ,
[0019] That is, return to the midpoint of the interval; After each iteration, update the squeeze interval according to the sign of the objective function, if:
[0020] Then let Otherwise, .
[0021] When satisfied or When the iteration stops, change the current weight. As a weighted solution that satisfies the target success rate constraint; in For function residual tolerance, When the interval length tolerance is used.
[0022] Furthermore, it also includes: Calculate the actual success rate of the target strategy. If it does not meet the preset threshold, recalculate; if it meets the preset threshold, output the target strategy.
[0023] Advantages and positive effects of the present invention: This method constructs a family of single-parameter continuous policies by combining state-by-state single-parameter convex combinations, and establishes a function of the starting success rate with respect to the weight parameters and its analytical derivative based on the absorption chain equation. It then combines the Guardian Newton method to solve for the weight parameters that satisfy the target success rate constraint within the bounded interval, thereby synthesizing the corresponding single Markov policy and avoiding the uncertainty brought about by mixed lottery policies.
[0024] This method only requires solving a system of linear equations with a number of non-terminating states and performing a small number of matrix operations. It does not require neural network training, has high computational efficiency and numerical stability. The obtained strategy parameters have clear meanings and a clear structure, which facilitates analysis and deployment. It is suitable for absorptive task scenarios such as path planning, process pass rate control and adversarial task win rate adjustment.
[0025] Therefore, this method solves the problems of difficulty in generating a single Markov random policy according to the target success rate in the absorption Markov decision-making process, and the instability of success rate adjustment caused by the reliance on random mixing or empirical parameter tuning in traditional methods. Attached Figure Description
[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0027] Figure 1 This is a flowchart illustrating the overall steps of this method; Figure 2 Heatmap of the policy matrix in Frozen Lake with a target of 0.8; Figure 3 For Frozen Lake with a target of 0.8 Anchoring curve. Detailed Implementation
[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0030] This invention provides a method for generating controllable policies based on a discrete absorbable Markov decision model, comprising: S1. Establish an absorption Markov decision model, divide the state into a set of non-terminating states and a set of absorption states, and determine the analytical solution form of the success probability of each state under a fixed policy.
[0031] 1. Define the discrete state space, the set of actions, and the parameters used to describe a given action. Under the action, the system changes from its current state Transition to the next state The state transition probability of the probability is expressed as: Discrete state space Action set State transition probability For any state-action pair ,have .
[0032] in and These represent the number of states and the number of actions, respectively.
[0033] 2. Divide the states into a set of non-terminating states, denoted as T, and a set of absorbing states, denoted as S; The set of absorption states includes: successful states denoted as G and failed states denoted as H; For any absorption state and any action , That is, the absorption state remains unchanged after the state transition.
[0034] 3. Set the starting state And based on the transition submatrix between non-termination states Calculate the absorption probability matrix It is used to characterize the absorption probability distribution starting from each non-termination state, providing a basis for subsequent strategy evaluation and synthesis.
[0035] 4. For any given strategy Establish the transition probability matrix and absorption vector ,in, This indicates the transition from a non-terminating state under a given policy. Transition to non-termination state The probability, Table of states One-step transition to successful state The probability is used to obtain the success rate vector of the non-termination subspace. It satisfies the absorption chain equation Then there is The probability of successful absorption for each non-termination state can be calculated.
[0036] By modeling the task as an absorbing Markov process with successful and failed states, and constructing the transition probability matrix and absorption probability vector in the non-terminating state subspace, the analytical solution form for the successful absorption probability of each state under a fixed policy is determined.
[0037] S2. Construct a state-by-state single-parameter convex combination of strategies. Construct a family of single-parameter convex combination strategies between two sets of teacher strategies to achieve one-dimensional continuity of the strategy space on the interval [0,1].
[0038] 1. For the same absorbent Markov decision, select two sets of teacher strategies. and They are set as the better and worse strategies under the success rate evaluation index, respectively, so that a strategy family covering a wider performance range can be constructed through the two strategies.
[0039] 2. For each non-termination state According to teacher strategy and Constructing a family of single-parameter strategies using single-parameter convex combinations : ,
[0040] in, These are scalar weight parameters.
[0041] 3. When At that time, strategy Degenerate into teacher strategies (Worse strategy); when At that time, strategy Degenerate into teacher strategies (Optimal strategy), intermediate parameters These are candidate strategies between the two teachers' strategies.
[0042] 4. For any candidate Based on the corresponding strategy Construct the transition matrix on the non-terminal state subspace With absorption vector The success rate vector can be calculated using the absorption chain equation. Used for characterization strategies The probability of successful absorption.
[0043] By constructing a family of single-parameter convex combination strategies between two sets of teacher strategies, the strategy space is made one-dimensional continuous on the interval [0,1], which provides a parameterized basis for subsequent parameter solving and controllable strategy generation based on target success rate.
[0044] S3. Express the starting success rate as a single-parameter function and determine the weight parameters. The analytical or numerical derivative calculation method.
[0045] 1. For any fixed weight parameter The strategy constructed through S2 Substituting into the absorption chain equation, we get:
[0046] Obtain the corresponding success rate vector. Starting state. Success rate is defined as .
[0047] Step 2, Command Then we have the analytic derivative:
[0048] And obtain the starting component .
[0049] 3. When the derivative in analytical form When the required data is difficult to obtain or computationally expensive, given weights Numerical approximation or finite difference methods are used at this point. Make an estimate.
[0050] Based on the absorption chain model, due to the transition matrix With absorption vector right For linear combinations, the overall success rate function about As a continuous and differentiable function, this property provides the function value and derivative information for subsequent weight calculation using the guardian Newton method.
[0051] S4. Solve the success rate function within the preset interval to obtain the target weight.
[0052] 1. Set a target success rate If it exceeds the range, it is not within the solution interval; If it is within the range, then it is based on the success rate function. Establish the objective equation .
[0053] 2. Apply the guardian Newton method to the equations within the bounded interval. Perform iterative solutions.
[0054] Initially set the squeeze interval Select initial value In the first In this iteration, candidate points are first calculated using Newton's formula:
[0055] This candidate point is used to estimate the location of the zeros of the equation based on the first-order approximation of the current point. If the derivative Existence and , For derivative tolerance, and the candidate points satisfy: ,
[0056] Then accept Newton's step, let The above conditions guarantee that the step size falls within the current search interval and the objective function residual monotonically decreases; otherwise, a two-step approach is used. ,
[0057] That is, return to the midpoint of the interval to maintain the interval convergence of the algorithm.
[0058] 3. After each iteration, update the squeeze interval according to the sign of the objective function. If:
[0059] Then let Otherwise, The above sign determination ensures that the target is always included in the interval. Inside.
[0060] 4. When satisfied or When the iteration stops, the current weight is set. As a weighted solution that satisfies the target success rate constraint; in For function residual tolerance, For interval length tolerance; This method combines the fast local convergence of Newton's method with the interval convergence of the bisection method, achieving convergence within the interval... The target weight can be stably obtained internally.
[0061] S5, Single Strategy Generation and Verification 1. Optimal weights obtained using S4 For each non-termination state and actions Target synthesis strategy based on single-parameter convex combination :
[0062] 2. Target strategy Output or store the policy in the form of a policy table or policy matrix for subsequent deployment or invocation.
[0063] 3. Adopting a goal-oriented strategy in the original Markov decision process environment. Multiple Monte Carlo simulations were performed to obtain an empirical estimate of the initial success rate. and compare With respect to preset verification tolerance To determine whether the actual success rate of the target strategy meets the target requirements.
[0064] 4. If Then adjust the number of simulations or reset them as needed. , Threshold, repeat S4; otherwise, consider the target policy. The target success rate constraint must be met.
[0065] Example Combination Figure 1-3 This embodiment takes the classic 4×4 sliding Frozen Lake absorptive Markov decision process as the object, and uses a single Markov policy synthesis with state-by-state single-parameter convex combination of policies and the guardian Newton method to achieve the target success rate.
[0066] The environment parameters are set as follows: the state set is The action set is {Left, Down, Right, Up}; The set of failed absorption states (holes) is... Successfully absorbed state is The starting state is The set of non-terminating states is: .
[0067] Step A: Establish an absorbent Markov decision model: Step A1: Press Grid numbering constructs a discrete state set With action set {left, bottom, right, top}, and define the transition probability model. .
[0068] Step A2: Divide the states into a set of non-terminating states. With absorption state set For any absorption state and any action ,make That is, the absorption state remains unchanged after the state transition.
[0069] Step A3: Construct a transition based on "sliding" dynamics: when performing an action in a non-terminating state. At that time, the actual action taken is selected with a 1 / 3 probability from the three options of "left deviation, straight movement, and right deviation"; if it leads to exceeding the boundary, the original state is maintained.
[0070] Step A4: For any given policy (Probability distribution of actions in each state), in the non-terminating subspace Establish the transition probability matrix With absorption vector :in ,
[0071] ,
[0072] The above and By traversal The states, actions, and transition distributions are accumulated and constructed.
[0073] Therefore, based on the absorption chain equation, the success rate vector can be solved in the non-termination subspace. :
[0074] And take the starting component As a strategy The starting success rate.
[0075] Step B: Construct a state-by-state single-parameter convex combination of the policy: Step B1: Construct two teacher strategies on the same AMDP: one is the maximum success rate strategy. The first method is obtained through value iteration; the second is a uniform strategy. For each non-terminating state, the four actions are taken with equal probability.
[0076] Step B2: For each non-termination state Construct a family of single-parameter strategies based on single-parameter convex combinations: ,
[0077] Step B3: When hour, ;when hour, .
[0078] Step B4: For any candidate Each of these can form a corresponding transition matrix. With absorption vector , in order to and further obtain .
[0079] Step C: Calculation of the success rate function and analytical derivative of the absorption chain: Step C1: Define the objective function ,in Depend on The solution is given.
[0080] Step C2: Let Then we have the analytic derivative:
[0081] And obtain the starting component .
[0082] Step C3: When When the numerical value is unavailable, the derivative is set to unavailable, thereby triggering a binary backoff in subsequent iterations.
[0083] Step D: Solving for the target success rate and iterating using the guardian Newton method Step D1: Set the target success rate Establish the objective equation .
[0084] Step D2: In the interval First calculate the endpoints , .like If not constrained by endpoints, the direct output is closer. Endpoint weights; if squeezed, set the squeeze interval. Select initial value In the first In this iteration, candidate points are first calculated using Newton's formula: Calculate Newton candidate points first
[0085] This candidate point is used to estimate the location of the zeros of the equation based on the first-order approximation of the current point. If the derivative Existence and , For derivative tolerance, and the candidate points satisfy: ,
[0086] Then accept Newton's step, let The above conditions guarantee that the step size falls within the current search interval and the objective function residual monotonically decreases; otherwise, a two-step approach is used. ,
[0087] That is, return to the midpoint of the interval to maintain the interval convergence of the algorithm.
[0088] Step D3: After each iteration, update the squeeze interval according to the sign of the objective function. If:
[0089] Then let Otherwise, The above sign determination ensures that the target is always included in the interval. Inside.
[0090] Step D4: When the condition is met or (in For function residual tolerance, For interval length tolerance, When the iteration stops, the maximum number of iterations is 50, and the current weight is reset. As a weighted solution that satisfies the target success rate constraint.
[0091] Step E: Single Policy Generation and Verification Step E1: Utilize the optimal weights obtained in step D For each non-termination state and actions Target synthesis strategy based on single-parameter convex combination :
[0092] Step E2: Set the target strategy Output or store the policy in the form of a policy table or policy matrix for subsequent deployment or invocation.
[0093] Step E3: Set the target success rate in the FrozenLake environment. 0.3, 0.6, and 0.8 were used to generate corresponding strategies, and 10,000 simulation experiments were conducted. The results show that the error between the simulation success rate and the target success rate is less than 1%, verifying the accuracy and stability of the method.
[0094] Step E4: Curve In the interval The curve exhibits a smooth variation, with the target point located inside the curve, demonstrating that the strategy family constructed in this invention possesses continuity and differentiability.
[0095] The experimental results are shown in Table 1. The results show that the method of the present invention can quickly generate a single Markov policy that satisfies the success rate of any target through analytical calculation and a finite number of iterations without retraining. It has the advantages of high computational efficiency, high accuracy, reproducible results and wide application.
[0096] Table 1. Results of Frozen Lake environmental experiments
[0097] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A method for generating controllable strategies based on a discrete absorbent Markov decision model, characterized in that, include: The task is modeled as an absorbent Markov decision process with success and failure states; For an absorptive Markov decision process, construct a family of continuous, single-parameter strategies between two teacher strategies; based on this family of strategies, express the starting success rate as a single-parameter function. The target weight is obtained by solving the success rate function within a preset interval; a target strategy is generated based on the target weight.
2. The method according to claim 1, characterized in that, The absorption Markov decision process includes: Discrete state space Action set State transition probability For any state-action pair ,have ; The states are divided into a set of non-terminating states, denoted as T, and a set of absorbing states, denoted as S; The set of absorption states includes: successful states denoted as G and failed states denoted as H; For any absorption state and any action , ; Starting state And based on the transition submatrix between non-termination states Calculate the absorption probability matrix : For any given policy Establish the transition probability matrix and absorption vector ; in, This indicates the transition from a non-terminating state under a given policy. Transition to non-termination state The probability, Table of states One-step transition to successful state The probability, Denotes the success rate vector of the non-terminating subspace, which satisfies the absorption chain equation. .
3. The method according to claim 1, characterized in that, The construction of a single-parameter continuous policy family between two teacher strategies includes: ， in, These are scalar weight parameters; It is in a non-terminating state; As a first-teacher strategy, For the second teacher strategy; Construct a family of single-parameter policies for single-parameter convex combinations of the first and second teacher policies.
4. The method according to claim 1, characterized in that, Based on the aforementioned strategy family, the starting success rate is represented as a single-parameter function, including: in, Transition matrix over non-terminating state subspace; Absorption vector.
5. The method according to claim 1, characterized in that, The step of solving the success rate function within a preset interval to obtain the target weight includes: Based on the success rate function within the preset interval Establish the objective equation ; The guardian Newton method is used to iteratively solve the objective equation.
6. The method according to claim 5, characterized in that, The method of using the guardian Newton method to iteratively solve the objective equation includes: Initially set the squeeze interval Select initial value . In the In this iteration, candidate points are calculated based on Newton's formula: The candidate points are used to estimate the positions of the zeros of the equation based on the first-order approximation of the current point. If the derivative Existence and , For derivative tolerance, and the candidate points satisfy: ， Then accept Newton's step, let ; If the condition is not met, then a two-step approach is used: ， That is, return to the midpoint of the interval; After each iteration, update the squeeze interval according to the sign of the objective function, if: Then let Otherwise, . When satisfied or When the iteration stops, change the current weight. As a weighted solution that satisfies the target success rate constraint; in For function residual tolerance, When the interval length tolerance is used.
7. The method according to claim 1, characterized in that, Also includes: Calculate the actual success rate of the target strategy; if it does not meet the preset threshold, recalculate. If the preset threshold is met, the target strategy is output.