A structured optimal control method and device for continuous-time two-dimensional Markov jump systems based on model-driven structured policy iteration

Abstract

The application discloses a kind of structured optimal control method and device of continuous time two-dimensional Markov jump system based on model-driven structured policy iteration, belong to automatic control, random system control and optimal control technical field, face two-dimensional Markov jump system described by Roesser model with known system matrix.The method obtains system matrix, input matrix, mode transition rate and weight matrix, establishes two-dimensional model containing horizontal state and vertical state;Infinite time domain performance index and block diagonal positive definite value function are constructed;Two-dimensional coupled Riccati relationship is obtained from HJB equation;Structured Riccati equation is constructed by block diagonal projection and non-diagonal residual extraction, and control gain is obtained by policy evaluation, projection, positive definite check, policy improvement and residual stability check iteration, and state feedback closed-loop control law is output.The method considers two-dimensional coupling information, structure realizability and stability verification.

Description

Technical Field

[0001] This invention relates to the fields of automatic control, stochastic system control, and optimal control, and specifically to a structured optimal control method and apparatus for continuous-time two-dimensional Markov jump systems based on model-driven structured strategy iteration. Background Technology

[0002] Markov transition systems are a class of dynamic systems that simultaneously involve continuous states and random switching of discrete modes. In real-world engineering applications, when affected by changes in the working environment, actuator failures, sudden load changes, sensor switching, variations in operating conditions, or component aging, the system matrix and input matrix often transition between several finite modes. Using Markov chains to describe this transition mechanism allows for the incorporation of random mode switching into stability and performance optimization analyses during controller design.

[0003] Two-dimensional systems are used to characterize dynamic processes evolving along two independent variables. For image processing, heat conduction processes, spatially distributed systems, repetitive processes, two-dimensional filtering, gridded dynamical systems, and networked controlled objects, the system state does not change only along a single time axis, but propagates or accumulates in two directions. The Roesser model divides the two-dimensional state into horizontal and vertical states, corresponding to information flows in two directions respectively. This model form has good physical interpretability and structural decomposability in two-dimensional control theory.

[0004] In a continuous-time two-dimensional Markov jumping system, control design is constrained by three factors: First, the system is two-dimensional, with the horizontal and vertical states corresponding to different information propagation directions; second, the system is continuous-time, involving state derivatives in both directions; and third, the system exhibits stochastic mode switching, with the mode transition rate entering the infinitesimal generator of the value function. Directly applying the algebraic Riccati equations for a one-dimensional Markov jumping linear system typically fails to adequately express the coupled dynamics in the two evolution directions. Conversely, directly employing unstructured two-dimensional optimal control solutions often results in Riccati matrices that easily contain cross-terms between the horizontal and vertical states.

[0005] Existing technologies include schemes focusing on H∞ control, filtering, or asynchronous control of two-dimensional Markov-jumping Roesser systems, as well as Q-learning or reinforcement learning-based optimal control of two-dimensional discrete systems. These schemes demonstrate a certain research foundation for two-dimensional jumping systems and policy iteration algorithms, but their focus is primarily on H∞ performance, finite-domain control, discrete system learning, or unknown model learning, without directly addressing the problem of incompatibility between the standard Riccati full-matrix solution and the block diagonal-valued function structure constraints in continuous-time two-dimensional Markov-jumping Roesser models.

[0006] Furthermore, in the Roesser model, the horizontal and vertical states have their own boundary inputs and propagation paths. To ensure that the value function matches the causal structure of this space, it is generally desirable for the value function matrix to maintain a block-diagonal positive definite form, consisting of both horizontal and vertical energies. However, the standard Riccati equations produce off-diagonal residuals in the full matrix sense when the system matrix contains horizontal-vertical coupling blocks. Forcing all matrix blocks to be simultaneously zero often fails to simultaneously satisfy the block-diagonal structure and optimality conditions; completely abandoning the block-diagonal structure makes it difficult for the resulting value function to correspond to the two directional energies of the Roesser model, resulting in unclear engineering implementation and stability interpretation.

[0007] Therefore, there is a need in this field for a control method that neither simply avoids two-dimensional coupled dynamics nor violates the structural constraints of the Roesser model. This method should combine fully coupled policy evaluation with block diagonal projection, provided the system matrix is ​​known: preserving complete coupling information during the evaluation phase, reconstraining the value function to the block diagonal subspace during the structuring phase, and using off-diagonal residuals instead of ignoring off-diagonal coupling for closed-loop stability verification. This ensures that the control law has both a performance optimization basis and can be implemented in a two-dimensional structural sense. Summary of the Invention

[0008] Technical problems to be solved

[0009] The purpose of this invention is to provide a structured optimal control method for continuous-time two-dimensional Markov jump systems based on model-driven structured policy iteration, which solves the problems of incompatibility between the standard Riccati solution and the diagonal value function constraint of the two-dimensional Roesser model block in existing optimal control schemes, unclear solution process after the two-dimensional mode transition rate enters the optimality condition, insufficient stability verification after projection, and inadequate handling of policy iteration anomalies.

[0010] Technical solution

[0011] To achieve the above objectives, this invention provides a structured optimal control method for continuous-time two-dimensional Markov jump systems based on model-driven structured strategy iteration. The method is used for closed-loop control of a two-dimensional thermal process with two evolution directions. The entire method is executed in the following order: parameter acquisition and verification – two-dimensional model establishment – ​​performance index construction – HJB derivation – structured Riccati equation construction – strategy iterative solution – closed-loop output and stability verification.

[0012] First, obtain the system matrix, input matrix, mode transition rate matrix, weight matrix, and boundary states, and perform a consistency check. This step is to prevent subsequent Riccati solutions and policy iterations from failing due to dimensionality mismatch, non-positive definite weight matrices, invalid transition rates, or unstable initial control gains.

[0013] Secondly, a continuous-time two-dimensional Markov jump Roesser model is established. This model includes horizontal state, vertical state, and control input, with mode variables taking values ​​from a finite set of modes. For each mode, the system matrix and input matrix can be different, and the controller also uses mode-dependent state feedback gain.

[0014] Next, we construct an infinite time-domain double integral performance index and a block diagonal positive definite value function. The performance index is used to measure the accumulation of state energy and control energy within a two-dimensional region, and the block diagonal value function is used to describe the horizontal and vertical state energy separately, making the value function consistent with the two-directional state structure of the Roesser model.

[0015] Then, the standard two-dimensional coupled algebraic Riccati relation is derived based on the two-dimensional Hamilton-Jacobi-Bellman equations. In the derivation, the horizontal state derivative, vertical state derivative, and mode transition term are simultaneously incorporated into infinitesimal generators, and the mode-dependent state feedback law is obtained by minimizing the control input using the Hamiltonian function.

[0016] Subsequently, a block diagonal projection operator and an off-diagonal residual extraction operator are introduced. The block diagonal projection operator preserves the energy balance within the horizontal and vertical subspaces, while the off-diagonal residual extraction operator records the horizontal-vertical cross-coupling residuals that are projected away. The structured two-dimensional coupled algebraic Riccati equations constructed in this way no longer require all blocks of the complete matrix equation to be simultaneously zero; instead, they require the residuals in the block diagonal subspaces to be zero, and the off-diagonal residuals are used as a stability check quantity.

[0017] Finally, a model-driven structured two-dimensional strategy iterative algorithm is used to solve the problem. In each iteration, the fully coupled Lyapunov equations are solved based on the current control gain, and then the evaluation matrix is ​​projected onto the block diagonal subspace. If the positive definiteness is not satisfied after projection, diagonal regularization or backtracking is performed. After the positive definiteness is satisfied, the control gain is updated. When the changes in the structured weight matrix, the changes in the control gain, and the off-diagonal residual norm simultaneously meet the preset conditions, the optimal structured weight matrix and control gain are output, and the closed-loop control law is generated.

[0018] Beneficial effects

[0019] Compared with existing technologies, this invention provides a structured optimal control method and apparatus for continuous-time two-dimensional Markov jump systems based on model-driven structured strategy iteration, which has the following beneficial effects:

[0020] 1. This invention introduces a structured two-dimensional coupled algebraic Riccati equation into the continuous-time two-dimensional Markov jump Roesser model, enabling controller design to simultaneously consider the effects of horizontal state, vertical state, and mode jump, rather than simply reducing the two-dimensional system to a one-dimensional system.

[0021] 2. This invention preserves the structural realizability of the value function through the block diagonal projection operator, enabling the weight matrix to correspond to both horizontal and vertical energy components, thus avoiding the destruction of the spatial causal structure of the Roesser model by the standard full-matrix Riccati solution.

[0022] 3. This invention introduces off-diagonal residual extraction and residual stability verification mechanisms, rather than simply ignoring the horizontal-vertical coupling terms. Off-diagonal residuals are used as verification quantities in closed-loop stability assessment, ensuring that the control law after structural projection has verifiable stability boundaries.

[0023] 4. This invention adopts a model-driven iterative process of "fully coupled strategy evaluation - block diagonal projection - positive definite preservation verification - strategy improvement". When the system matrix is ​​known, it can be solved by deterministic matrix equations. It does not require a large number of online trials, nor does it rely on black-box data training. The engineering implementation path is relatively direct.

[0024] 5. This invention provides processing logic for abnormal situations such as unstable initial gain, non-positive definite matrix after projection, residual exceeding limits, and non-convergence of iteration, so that the control algorithm has good operability and robustness when implemented on an engineering computing platform. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the overall process of the structured optimal control method for continuous-time two-dimensional Markov jump systems of the present invention.

[0026] Figure 2 This is a schematic diagram illustrating the relationship between the horizontal state, vertical state, and mode transition in the Roesser model of this invention.

[0027] Figure 3 This is a schematic diagram of the diagonal projection and off-diagonal residual processing of the block in this invention.

[0028] Figure 4 This is a schematic diagram of the iterative process of the model-driven structured two-dimensional strategy of this invention.

[0029] Figure 5 This is a schematic diagram of the structured optimal control device module composition of the present invention. Detailed Implementation

[0030] Reference Appendix Figure 1The overall process of this invention includes parameter acquisition and verification, two-dimensional model establishment, performance index construction, structured equation construction, strategy iterative solution, and closed-loop output and stability verification. The invention is described below with reference to embodiments. It should be noted that the following embodiments are used to explain the technical solution of this invention and are not intended to limit the scope of protection. Conventional substitutions made by those skilled in the art to the matrix dimension, weight parameters, iteration thresholds, stability criteria, or computing platforms without departing from the concept of this invention should be considered equivalent embodiments of this invention.

[0031] Example 1: Main System Modeling and Parameter Input

[0032] This embodiment addresses a continuous-time two-dimensional Markov transition system with a known system matrix. The physical object of the system can be a two-dimensional thermal process, an image dynamic processing process, a repetitive production process, a two-dimensional networked control object, or other engineering systems with two evolution directions. The system has a finite number of modes, each representing an operating condition, such as a normal operating condition, a load variation operating condition, an actuator efficiency degradation operating condition, or a sensor noise variation operating condition. Mode switching is described by a Markov chain.

[0033] Before building the model, the control device first reads the input data. The input data includes at least the system matrix, input matrix, state weight matrix, control weight matrix, horizontal mode transition rate matrix, vertical mode transition rate matrix, boundary states, initial control gain, maximum number of iterations, and convergence threshold for each mode. (See attached reference.) Figure 2 As shown, the system matrix and input matrix are used to describe the two-dimensional Roesser state space, the mode transition rate is used to describe random transitions, and the boundary state is used to determine the starting point of the state response within the two-dimensional region.

[0034] To avoid unexplained failures in subsequent calculations, this embodiment performs consistency checks before model building. Dimensionality checks include: the number of rows and columns in the system matrix should match the augmented state dimension; the number of rows in the input matrix should match the augmented state dimension; the dimension of the control weight matrix should match the control input dimension; and the dimension of the state weight matrix should match the augmented state dimension. Positive definiteness checks include: the state weight matrix and control weight matrix should be positive definite or at least meet the preset semi-positive definite / positive definite requirements. Mode transition rate checks include: the off-diagonal elements of each transition rate matrix are non-negative, and the sum of the elements in each row is zero.

[0035] In technical applications, continuous-time two-dimensional Markov transition systems require the simultaneous representation of state derivatives in two directions. To address this, the following Roesser model is established:

[0036] (1);

[0037] In equation (1), This is the first-time variable, used to represent the continuous evolution variable in the horizontal direction; This is the second time variable, used to represent the continuous evolution variable in the vertical direction; This is the horizontal state vector; This is the vertical state vector; For pattern variables; , , and This represents the system block matrix under the current mode; and This is the input block matrix in the current mode; To control the input.

[0038] To facilitate subsequent matrix derivation, we define the augmented state and its derivative:

[0039] (2);

[0040] In equation (2), For augmented state vectors; Let be the augmented state derivative vector. The reason for introducing this definition is that subsequent performance indices, value functions, and Riccati equations are all expressed using the augmented state as a unified variable, thereby avoiding repeated switching of signs between the horizontal and vertical states.

[0041] When the pattern variable satisfies At this time, the model can be written in compact form:

[0042] (3);

[0043] In equation (3), Number the current mode; A finite set of patterns; For pattern The augmented system matrix below; For pattern The augmented input matrix is ​​used. This compact form facilitates the unified construction of control laws and policy iteration equations under different modes.

[0044] The state response of a two-dimensional system is also related to the boundary conditions. Let the boundary conditions be:

[0045] (4);

[0046] In equation (4), It is a two-dimensional boundary state set; The horizontal state boundary is given along the vertical direction; The vertical state boundary is given along the horizontal direction. The technical role of introducing the boundary state is to ensure that the closed-loop state response is determined not only by the internal dynamics but also by the two-dimensional region boundary input, which conforms to the basic structure of the Roesser model.

[0047] Example 2: Two-dimensional Markov transition mechanism and state feedback controller

[0048] This embodiment further illustrates the pattern transition mechanism. The pattern variable takes values ​​from a finite set:

[0049] (5);

[0050] In equation (5), This represents the total number of modes. For a continuous-time two-dimensional system, the modes change not only with the first time variable but also with the second time variable. The local transition probabilities in the horizontal and vertical directions are represented by their respective transition rates. To clearly express the random transitions in the two directions, this embodiment introduces horizontal and vertical transition rates.

[0051] The horizontal mode transfer rate satisfies:

[0052] (6);

[0053] In equation (6), The pattern is along the horizontal direction by Jump to The transfer rate; Let be the diagonal elements of the horizontal generator matrix. This condition guarantees that the horizontal transition matrix possesses the fundamental properties of Markov generators.

[0054] The vertical mode transfer rate satisfies:

[0055] (7);

[0056] In equation (7), The pattern is along the vertical direction by Jump to The transfer rate; The diagonal elements of the meta-matrix generated in the vertical direction are defined. This definition allows mode switching to simultaneously reflect stochastic evolution in both directions.

[0057] Since the value function uses an augmented state representation, the transition rates in both directions need to be embedded into the augmented matrix. This embodiment defines the block diagonal mode transition rate matrix:

[0058] (8);

[0059] In equation (8), For pattern To mode The block diagonal transition matrix; An identity matrix with the same dimension as the horizontal state; This is an identity matrix with the same dimension as the vertical state. The reason for using this block diagonal form is that the horizontal mode transition rate acts on the horizontal energy block, and the vertical mode transition rate acts on the vertical energy block, both of which enter the infinitesimal generator of the two-dimensional value function.

[0060] For each mode, set up a mode-related state feedback controller:

[0061] (9);

[0062] In equation (9), For pattern The control gain matrix is ​​used. The advantage of using a mode-dependent controller is that when the system enters different modes due to changes in operating conditions, fault states, or load changes, the control input can be adjusted according to the dynamic characteristics of the corresponding mode, rather than using the same fixed gain to handle all modes.

[0063] Example 3: Performance Indicators and Block Diagonal Value Functions

[0064] In optimal control, performance metrics are used to describe the desired target for the controller to optimize. This invention addresses two-dimensional continuous-time systems, therefore employing an infinite-time-domain dual-integral performance metric instead of a one-dimensional time integral. The technical scenario is that the controller needs to simultaneously suppress state energy and control energy across the entire two-dimensional evolutionary domain, rather than optimizing only on a single time axis.

[0065] This embodiment constructs the following performance indicators:

[0066] (10);

[0067] In equation (10), For pattern The following performance metrics; For mathematical expectation; This is the state weight matrix; To control the weight matrix; For state energy terms; To control the energy term, the resulting control law achieves a balance between stability and control energy by simultaneously penalizing state deviations and controlling energy consumption.

[0068] Based on the structural decomposability of the Roesser model, this embodiment constructs the value function as the sum of the horizontal state energy and the vertical state energy:

[0069] (11);

[0070] In equation (11), For pattern Value functions; The horizontal state energy function; The vertical state energy function; Let be the weight matrix of the value function. This value function is not an arbitrary quadratic form of the entire matrix, but rather a quadratic form that serves the structure of the Roesser model.

[0071] To ensure that the horizontal and vertical energies remain decomposable, the following requirements must be met:

[0072] (12);

[0073] In equation (12), This refers to the positive definite weight block corresponding to the horizontal state; This represents the positive definite weight block corresponding to the vertical state. The value function is explicitly restricted to the positive definite structure diagonally within the block, preventing the cross terms obtained from the standard Riccati solution from disrupting the two-dimensional state structure.

[0074] Without the block diagonal constraint of equation (12), the standard full matrix value function may contain One type of cross-energy term. While this cross-term may mathematically improve unconstrained optimality performance, it weakens the independent interpretation of horizontal and vertical boundary states in the Roesser model and makes subsequent stability proofs difficult to directly correspond to the two evolution directions. Therefore, this invention uses the block diagonal value function as the fundamental constraint for structured optimality.

[0075] Example 4: Two-dimensional HJB derivation and standard coupling Riccati relation

[0076] This example explains why it is necessary to start with the HJB equations. For Markov jump systems, the value function change comes not only from continuous state changes but also from value function switching caused by mode jumps. For two-dimensional systems, the value function change also includes horizontal state transitions. Changes in direction and vertical state along The direction changes. Therefore, the two-dimensional HJB equations need to include both state derivative terms and mode transition terms.

[0077] Constructing infinitesimal generators based on value functions:

[0078] (13);

[0079] In equation (13), For pattern Infinitesimal generators of subvalued functions; The derivative of the horizontal state along the first time step; The derivative of the vertical state along the second time variable; For pattern The value function weight matrix is ​​given. This formula unifies two-dimensional continuous state changes and mode-jumping changes into a single generator, which is the basis for deriving two-dimensional coupled Riccati relations.

[0080] For a quadratic value function, its gradient can be written as:

[0081] (14);

[0082] In equation (14), Let be the gradient of the value function with respect to the augmented state. This formula is used to transform the HJB equation into a matrix algebra equation.

[0083] Construct the Hamiltonian function:

[0084] (15);

[0085] In equation (15), For pattern The Hamiltonian function is defined as follows: the first term represents the state energy, the second term represents the control energy, the third term represents the influence of continuous state dynamics on the value function under the current mode, and the fourth term represents the influence of mode transition on the value function. This formula is used to transform the performance index minimization problem into a point state minimization problem with respect to the control input.

[0086] Setting the first-order partial derivative of the Hamiltonian function with respect to the control input to zero, we obtain the mode-dependent state feedback law:

[0087] (16);

[0088] In equation (16), For pattern The optimal control input is as follows; Let be the feedback gain determined by the current value function weight matrix. This formula shows that the update of the control gain depends on the input matrix, control weights, and structured weight matrix.

[0089] Substituting the state feedback law into the HJB equation and using quadratic form symmetric decomposition, the standard two-dimensional coupled algebraic Riccati relation can be obtained:

[0090] (17);

[0091] In equation (17), For linear terms corresponding to continuous state dynamics; For mode transition coupling terms; This is the feedback term formed after optimizing the control input. This formula reflects the energy balance in the sense of a complete matrix.

[0092] However, there is a structural conflict between equation (17) and equation (12). If Includes horizontally-vertically coupled blocks, i.e. or If it is not zero, then This often results in off-diagonal partitioning. In this case, forcing the complete matrix equation to be all zeros often makes it difficult to preserve... The block diagonal positive definite structure. This contradiction is precisely the key technical problem that this invention aims to solve.

[0093] Example 5: Block diagonal projection and off-diagonal residuals

[0094] This embodiment illustrates the method for constructing structured Riccati equations. (See attached document.) Figure 3 As shown, for any block matrix divided according to horizontal and vertical states, the block diagonal projection operator is first defined:

[0095] (18);

[0096] In equation (18), This is a block matrix divided according to horizontal and vertical states; Divide the horizontal subspace into blocks; Divide the vertical subspace into blocks; and This is a non-diagonal coupled block; This is the result of the block diagonal projection. The technical function of this formula is to preserve the energy balance within the horizontal and vertical subspaces while discarding cross-blocks that do not conform to the value function structure constraints.

[0097] To avoid completely ignoring off-diagonal coupling, this invention also defines an off-diagonal residual extraction operator:

[0098] (19);

[0099] In equation (19), This is a non-diagonal residual matrix. This formula is used to extract residuals generated by the coupling of horizontal and vertical states, but which are not directly incorporated into the block diagonal value function constraint solution. This maintains the feasibility of structured solutions while providing residual information for subsequent closed-loop stability verification.

[0100] Applying equation (18) to the standard two-dimensional coupled Riccati relation, we obtain the structured two-dimensional coupled algebraic Riccati equation:

[0101] (20);

[0102] In equation (20), This is the block diagonal projection operator. The equation does not aim for an unstructured global optimum in the complete matrix space, but rather, under block diagonal value function constraints, makes the Riccati residuals in the horizontal and vertical subspaces zero. The resulting "structured optimum" refers to the optimum under Roesser structural constraints and block diagonal projection.

[0103] Further define the complete residual:

[0104] (twenty one);

[0105] In equation (21), For pattern The complete Riccati residuals are required. The structured equations require... ,and It then participates in the stability assessment as an off-diagonal residual.

[0106] To ensure the projected matrix maintains positive definiteness, this embodiment incorporates a positive definiteness preservation check during policy iteration. If the resulting matrix after a certain round of projection... If the positive definiteness condition is not met, the following regularization method is applied:

[0107] (twenty two);

[0108] In equation (22), The regularization coefficient is used. This is the identity matrix with the same dimension as the augmented state. This formula does not change the control objective, but rather prevents non-positive definite matrices caused by projection, rounding errors, or inappropriate initial gains from entering the next iteration in numerical calculations.

[0109] Example 6: Model-Driven Iteration of Structured Two-Dimensional Strategy

[0110] This embodiment illustrates the specific execution process of the structured two-dimensional strategy iteration. (See attached document.) Figure 4 When the system matrix is ​​known, iterative model-driven strategies are more direct than fully data-driven approaches. Their inputs are explicit, the computational object is a matrix equation, and they do not require extensive online exploration of the controlled object, making them suitable for implementation in engineering controllers or simulation platforms.

[0111] First, initialize the number of iterations and the control gain:

[0112] (twenty three);

[0113] In equation (23), This represents the number of iterations. For pattern Initial control gain; For pattern The initial block diagonal positive definite matrix is ​​used. The initial control gain can come from offline design, an empirical gain library, or the calculation result of a weighted linear quadratic regulator, but it must pass closed-loop stability verification.

[0114] In the In this iteration, the closed-loop matrix is ​​constructed based on the current control gain:

[0115] (twenty four);

[0116] In equation (24), For the first The closed-loop matrix of the next iteration; For the first The control gain for the next iteration. This closed-loop matrix is ​​used in the policy evaluation phase.

[0117] Then, the evaluation weights are constructed:

[0118] (25);

[0119] In equation (25), For the first The evaluation weight matrix for the next iteration. This matrix includes the state weights, the control energy caused by the current control gain, and the mode transition coupling term.

[0120] Solving the fully coupled Lyapunov equations during the strategy evaluation phase:

[0121] (26);

[0122] In equation (26), This is the weight matrix for the next round obtained from the strategy evaluation. The formula uses a complete coupling matrix during the evaluation phase, thus avoiding prematurely severing the dynamic connection between the horizontal and vertical states.

[0123] After completing the strategy evaluation, perform a block diagonal projection on the evaluation matrix:

[0124] (27);

[0125] In equation (27), This is the projected block diagonal weight matrix. This step ensures that the value function returns to the block diagonal subspace required by the Roesser structure after each iteration.

[0126] After projection, implement policy improvements:

[0127] (28);

[0128] In equation (28), This is the updated control gain. This formula is derived directly from the HJB optimality condition, ensuring that policy improvement aligns with the structured value function.

[0129] To avoid instability in the closed loop due to excessively rapid updates, this embodiment can employ step size correction:

[0130] (29);

[0131] In equation (29), For gain update step size, and When the updated control gain fails to meet stability requirements, it can be reduced. This process continues until the closed-loop matrix regains its preset stability. This anomaly handling makes the algorithm more suitable for real-world computing environments.

[0132] Convergence determination uses multiple thresholds:

[0133] (30);

[0134] In equation (30), The threshold for changes in the structured weight matrix; It represents the matrix norm.

[0135] (31);

[0136] In equation (31), This is the threshold for controlling gain variation. This condition is used to determine whether the control strategy is basically stable.

[0137] (32);

[0138] In equation (32), This is the off-diagonal residual threshold. This condition is used to prevent excessive cross-coupling from being discarded during projection, thus ensuring that the structured control law still has an acceptable closed-loop stability boundary.

[0139] When equations (30), (31), and (32) are all satisfied, the output is:

[0140] (33);

[0141] In equation (33), The optimal weight matrix for structured analysis; This is for structured optimal control gain. The output forms a closed-loop control law:

[0142] (34);

[0143] In equation (34), This is the closed-loop control input for the final output mode. This control input is sent to the actuator or control simulation platform to perform closed-loop regulation of the controlled object.

[0144] Example 7: Stability Verification and Anomaly Handling

[0145] This embodiment illustrates closed-loop stability verification. For structured projection methods, if only the block diagonal equations are checked without checking the off-diagonal residuals, situations may arise where the projection appears to satisfy the equations, but the complete closed-loop dynamics are still not sufficiently stable. Therefore, this invention adds residual stability verification before the output control law.

[0146] Define the closed-loop matrix:

[0147] (35);

[0148] In equation (35), For pattern The final closed-loop matrix is ​​obtained by selecting the Lyapunov function:

[0149] (36);

[0150] In equation (36), For the final horizontal weight block; This is the final vertical weight block. If a positive number exists... and , so that:

[0151] (37);

[0152] In equation (37), and These are the upper and lower bound coefficients of the Lyapunov function. This formula demonstrates that the value function can serve as an effective measure of the energy of the augmented state.

[0153] Closed-loop generators satisfy:

[0154] (38);

[0155] In equation (38), Let be the generators of the closed-loop valued function. If the complete closed-loop generator matrix is ​​negative definite, then the system is asymptotically stable. For the structured projection method, this invention first checks the block diagonal projection terms:

[0156] (39);

[0157] In equation (39), less than zero indicates that the matrix is ​​negative definite. This condition indicates that the energy in both the horizontal and vertical subspaces decays under the closed-loop action.

[0158] Further examination of off-diagonal residuals:

[0159] (40);

[0160] In equation (40), This represents the upper bound of the off-diagonal residuals. This formula is used to limit the influence of horizontal-vertical coupling residuals. If the residuals are too large, it indicates that the cross-coupling discarded by the block diagonal projection cannot be ignored. In this case, the controller should not directly output the final control law, but should continue to iterate or adjust the weight parameters.

[0161] In actual implementation, the following abnormal scenarios may occur:

[0162] First, the initial control gain is unstable. In this case, the control weight can be increased. Reduce the initial gain amplitude, or select an initial value from the offline stable gain library.

[0163] Second, the projected matrix is ​​not positive definite. In this case, a diagonal regularization term can be added, or the matrix can be reverted to the stable weight matrix from the previous round.

[0164] Third, the closed-loop stability fails after the gain update. At this time, the step size correction of equation (29) can be used to gradually approach the new gain.

[0165] Fourth, if the iteration reaches the maximum number of iterations but still fails to converge, output a message indicating that the convergence condition is not met, and record the control gain that last satisfied the positive definiteness and stability checks.

[0166] Fifth, the off-diagonal residuals consistently exceed the limit. This indicates that the system's horizontal-vertical coupling is too strong, and the control objectives under the block diagonal structure constraints need to have their weights reset or the residual threshold relaxed.

[0167] Through the above-described anomaly handling, this invention does not merely provide ideal mathematical steps, but rather a complete logic executable in electronic devices. This logic improves the robustness of algorithm deployment and enables those skilled in the art to implement the method according to the specification.

[0168] Example 8: Implementation of Device Modules and Electronic Equipment

[0169] Reference Appendix Figure 5 The present invention can also be implemented through a control device. The control device includes a model and parameter acquisition module, a performance index construction module, a structured equation construction module, a strategy iterative solution module, an anomaly verification and residual judgment module, a control law output module, and a log recording and parameter storage module.

[0170] The model and parameter acquisition module receives the system matrix, input matrix, boundary states, mode transition rate matrix, and iteration parameters. This module also performs dimension consistency checks, transition rate validity checks, and initial control gain stability checks. If the input data does not meet the requirements, this module outputs an error message instead of continuing to the Riccati solution.

[0171] The performance index construction module generates the state weight matrix, control weight matrix, and block diagonal value function. The module takes the user-defined control objective and system state dimension as input and outputs a value that satisfies the positive definite condition. , and initial If the weight matrix entered by the user is not positive definite, the module can prompt for a reset, or add a small diagonal regularization term within the allowed range.

[0172] The structured equation construction module is used to construct standard Riccati residuals, block diagonal projection residuals, and off-diagonal residuals based on the HJB derivation results. This module does not delete off-diagonal residuals; instead, it sends them to the anomaly check and residual judgment module. This avoids hidden cross-coupling issues during the structured solution process.

[0173] The policy iteration solution module is used to perform fully coupled policy evaluation, block diagonal projection, and policy improvement. This module can call a numerical linear algebra solver to solve the Lyapunov equations. For larger-scale systems, a sparse matrix solver or a parallel solver can be used; for controllers with a small number of modes, the control gain can be obtained offline on an industrial computer or embedded processor and then called online.

[0174] The anomaly detection and residual judgment module is used to check the positive definiteness after projection, the stability of the closed-loop matrix, the norm of the off-diagonal residuals, and the convergence conditions of the iteration. The output of this module determines whether the algorithm continues to iterate, backs down for correction, adjusts the step size, or outputs the final control law.

[0175] The control law output module is used to convert the final gain Write to the controller, and based on the current mode. Select the corresponding gain to generate the control input. If the current pattern cannot be reliably identified, the most recent valid pattern, the pattern probability-weighted gain, or the safe default gain can be used as a downgrade process.

[0176] The log recording and parameter storage module is used to save the system matrix version, weight matrix, mode transition rate, number of iterations, convergence threshold, final control gain, residual norm, and stability verification results. This module does not modify the control algorithm, but rather facilitates engineering debugging, version tracking, and troubleshooting.

[0177] The above modules can be executed by industrial computers, embedded controllers, digital signal processors, servers, or simulation platforms. For online control, model solving can be completed offline, with the control input calculated only based on the current mode and state during the online phase. For simulation verification, strategy iteration can be re-executed after each parameter update.

[0178] Supplementary explanation of Example 8: Input, processing logic, output, and technical function of each algorithm unit

[0179] The following section further explains the input, processing logic, output, and technical functions of the main algorithm units in the control device, making the connection between the various algorithm units clearer.

[0180] The inputs to the model and parameter acquisition unit include the number of modes, the system matrix and input matrix for each mode, the horizontal state dimension, the vertical state dimension, the control input dimension, the mode transition rate matrices in both directions, and the two-dimensional boundary states. The processing logic of this unit is to first determine the data dimensions, and then determine the numerical conditions. For example, if... Not a square formation, or The number of rows and If the number of rows is inconsistent, it means that the state equation cannot be established; if If it is not a positive definite matrix, then the following... Unsafe to use; if the sum of any row in the mode transition matrix is ​​not zero, it indicates that the matrix does not satisfy the Markov generator condition. The output of this unit is the validated set of model parameters. The technical function of this unit is to move algorithmic errors forward to the input stage, avoiding unlocatable numerical errors that occur only during the solution of the Riccati equations.

[0181] The input to the 2D model building unit is a validated parameter set. Based on the dimensions of the horizontal and vertical states, this unit reorganizes the original matrix into a Roesser block format, forming an augmented state representation. Its processing logic includes: arranging the state vectors in the order of horizontal states first, followed by vertical states; arranging the four blocks of the system matrix in the same order; dividing the input matrix into horizontal and vertical input blocks; and recording the matrix index for each mode. This unit outputs a unified augmented model. The technical role of this unit is to ensure that subsequent block diagonal projections can accurately identify the horizontal and vertical blocks, preventing projection errors caused by disordered state arrangement.

[0182] The performance metric construction unit takes as input the augmented state dimension, the control input dimension, and user-defined state and control penalties. This unit first generates or reads... and Then check its positive definiteness. For scenarios where it is necessary to distinguish the penalty intensity for horizontal and vertical states, we can... The unit is configured in a block diagonal form, allowing for adjustable horizontal and vertical state weights. It outputs performance index parameters and an initial block diagonal value function matrix. The unit's technical function is to make the control objective interpretable: increasing the horizontal state weight increases the desired faster suppression of the horizontal state; increasing the control weight increases the desired reduction in control actions.

[0183] The input to the structured equation construction unit is the model matrix, weight matrix, mode transition rate matrix, and current value function matrix. This unit first constructs the complete Riccati residual, then calculates the block diagonal projection residual and the off-diagonal residual separately. The processing logic is as follows: the complete residual is used to retain all coupling information; the block diagonal projection residual is used to form structured solution conditions; and the off-diagonal residual is used for stability and anomaly detection. The unit outputs the structured Riccati equation, the projection residual, and the off-diagonal residual. The technical significance of this unit lies in clearly distinguishing between "constraints involved in the solution" and "coupling effects involved in verification," avoiding the misunderstanding that structured projection is simply the deletion of coupling terms.

[0184] The input to the policy iterative solution unit is the current gain, the current value function matrix, and the structured equations. This unit first constructs a closed-loop matrix using the current gain, then solves the coupled Lyapunov equations, followed by block diagonal projection and policy improvement. The output of this unit is a new value function matrix and a new control gain. In this way, each iteration allows the system dynamics to fully participate in the evaluation before projecting the results back to the realizable structure, thus reflecting the horizontal-vertical coupling effects better than solving directly on two independent subsystems.

[0185] The anomaly detection and residual judgment unit's inputs include the projected value function matrix, the new control gain, the closed-loop matrix, and the block diagonal and off-diagonal residuals. This unit first checks for positive definiteness; if positive definiteness fails, regularization or backoff is performed. Then, it checks for negative definiteness in the closed-loop; if negative definiteness fails, the gain update step size is reduced. Finally, it checks the residual norm; if the residual norm exceeds the limit, iteration continues or a prompt to adjust the weights is given. The unit's output can be the acceptable gain, anomaly type, or a stop flag. The technical role of this unit is to extend the algorithm from ideal mathematical derivation to a workable process, providing a clear handling path for actual equipment when encountering numerical anomalies.

[0186] The inputs to the control law output unit are the final control gain, the current system mode, and the current augmented state. This unit selects the corresponding control law output unit according to the current mode. and output If the controlled object is in a simulation environment, this output can be directly input into the state equation; if the controlled object is an actual device, this output can be converted into voltage, current, valve opening, heating power, or other actuator signals. The technical function of this unit is to convert the structured control gain obtained offline or quasi-online into actual closed-loop control actions.

[0187] As can be seen from the above description, each algorithm unit of this invention has clearly defined input, processing logic, output, and technical functions. System modeling provides a unified state representation for subsequent control design; performance indicators provide optimization objectives; HJB and Riccati derivations provide the source of the control law; projection and residual mechanisms resolve structural conflicts; strategy iteration provides a solution method; exception handling ensures feasibility; and the control law output completes closed-loop application.

[0188] In implementation, the aforementioned algorithm unit can be configured as an offline solving mode, an online calling mode, or a semi-online update mode. The offline solving mode is suitable for devices with long-term stable system parameters. The control gain is pre-calculated and embedded in the controller; during runtime, it only needs to look up the table based on the current mode and state to output the control quantity. The online calling mode is suitable for scenarios with rapidly changing operating conditions but sufficient computing resources. The controller can re-execute the strategy iteration after detecting changes in the mode transition rate or weight parameters. The semi-online update mode is suitable for industrial sites. Normally, the verified gain is called; when residual verification or state decay rate is abnormal, a re-solving is triggered. Through these three deployment methods, this invention can cover different application scenarios such as simulation verification, experimental platforms, and engineering controllers.

[0189] Furthermore, this invention does not require all matrices to be manually input. For engineering objects with installed sensors, the system matrix can be derived from offline identification results; for objects with multiple operating conditions, the mode transition rate can be obtained statistically from historical operation records; for objects with high security requirements, the weight matrix can be automatically configured by the host computer according to the security level. As long as the dimension, positive definiteness, and transition rate verification are satisfied before finally entering the strategy iteration, the method of this invention can be executed. This setting can improve the practicality of the solution and avoid limiting this invention to a single numerical example.

[0190] Example 9: Numerical Example

[0191] To illustrate the feasibility of the method of this invention, a two-mode example is given. This example is only used to explain the algorithm steps and is not intended to limit the matrix dimension or parameter range of this invention. Assume the system has two modes, with the parameters for mode 1 being:

[0192] (41);

[0193] In equation (41), This is the system matrix under mode 1; This is the input matrix for mode 1; This is the state weight matrix under mode 1; This refers to the control weights under Mode 1.

[0194] The parameters for mode 2 are:

[0195] (42);

[0196] In equation (42), , , and These are the system matrix, input matrix, state weight matrix, and control weights under mode 2, respectively.

[0197] The mode transition matrices in the horizontal and vertical directions can be taken as follows:

[0198] (43);

[0199] In equation (43), This is the horizontal transfer rate matrix; Let be the vertical transition rate matrix. All off-diagonal elements of both matrices are non-negative, and the sum of the elements in each row is zero, satisfying the Markov generator requirement.

[0200] The boundary conditions are as follows:

[0201] (44);

[0202] In equation (44), The horizontal state boundary; This is a vertical state boundary condition. This boundary condition is used to generate the initial response within the two-dimensional region.

[0203] Using the model-driven structured two-dimensional policy iterative algorithm of this invention, after several rounds of fully coupled policy evaluation, block diagonal projection, and policy improvement, a convergence control gain can be obtained, for example:

[0204] (45);

[0205] In equation (45), The structured optimal control gain under mode 1; The structured optimal control gain is given in Mode 2. When this control law is applied to the closed-loop system, the state trajectory gradually decays within the two-dimensional region, indicating that the structured control law can achieve closed-loop stability.

[0206] In this numerical example, if block diagonal projection is not performed, the resulting weight matrix typically contains off-diagonal cross terms; if only block diagonal projection is performed without checking off-diagonal residuals, it may be difficult to determine whether the projected coupling terms affect stability. This invention performs both projection and residual verification, thus enabling a clearer distinction between "structured constraints used for solving" and "coupling residuals used for stability assessment."

[0207] This invention is applicable to two-dimensional continuous-time Markov jump control objects for which the system matrix can be obtained. The system matrix can be derived from physical modeling, offline identification, experimental calibration, or simulation linearization. As long as the system matrix for each mode can be obtained... and Based on the pattern transfer rates in the horizontal and vertical directions, a structured controller can be constructed according to the method of this invention.

[0208] Regarding the mode transfer rate, if the transfer rates in the horizontal and vertical directions are the same, it can be set... If only the estimated transfer rate is available, the estimated value can be used as input and a conservative margin can be added to the residual threshold. If there is a small range of uncertainty in the model transfer rate, the upper and lower bounds of the transfer rate can be used for conservative judgment during the stability verification stage.

[0209] For the strategy iterative solution, the Lyapunov equations can be solved using direct matrix equations or iterative linear equations. For systems with a large number of modes, the evaluation matrix can be solved in parallel for each mode, and then residual verification can be performed uniformly. For scenarios where the controller's computing power is limited, the gain table can be obtained offline, and only pattern matching and state feedback calculations can be performed online.

[0210] For the anomaly handling section, the diagonal regularization term for positive definiteness preservation is not required in all cases. When the projected matrix already satisfies positive definiteness, no regularization term is needed. Step size adjustment is also not required in every round; it is only initiated when the closed-loop stability check fails after policy improvement. This avoids unnecessary changes to the control gain while ensuring an executable backoff path for the algorithm in anomaly situations.

[0211] In summary, this invention establishes a complete, implementable, and verifiable structured optimal control method through a continuous-time two-dimensional Markov jump-transition Roesser model, block diagonal value functions, structured two-dimensional coupled algebraic Riccati equations, off-diagonal residual stability verification, and model-driven structured policy iteration. This method retains the deterministic solution characteristics of the Riccati equations and policy iteration while specifically addressing the structural constraints of the two-dimensional Roesser model. It can obtain mode-dependent control gains suitable for closed-loop control while maintaining the decomposability of horizontal and vertical state energies.

Claims

1. A structured optimal control method for continuous-time two-dimensional Markov jump systems based on model-driven structured strategy iteration, characterized in that, The method is used for closed-loop control of a two-dimensional thermal process with two evolution directions, and includes the following steps: S1. Obtain the system matrix, input matrix, mode transition rate matrix, performance weight matrix, boundary state and iteration parameters of the two-dimensional thermal process under each mode, and perform consistency verification. S2. Based on the Roesser model, establish a two-dimensional state space model containing horizontal and vertical states, and form augmented states; S3. Construct the infinite time-domain double integral performance index and the block diagonal positive definite function; S4. Construct infinitesimal generators and obtain mode-dependent state feedback control laws based on the two-dimensional Hamilton-Jacobi-Bellman equations. S5. Perform block diagonal projection on the two-dimensional coupled algebraic Riccati relation and extract the off-diagonal residuals; S6. The mode-dependent structured optimal control gain is obtained through fully coupled strategy evaluation, block diagonal projection, positive definite hold-up verification, and strategy improvement iteration. S7. Generate a closed-loop control input based on the current mode, augmented state, and control gain. Convert the closed-loop control input into heating power and send it to the actuator to perform closed-loop regulation of the two-dimensional thermal process.

2. The structured optimal control method for a continuous-time two-dimensional Markov jump system based on model-driven structured strategy iteration as described in claim 1, characterized in that, The consistency check in S1 includes: determining whether the system matrix and input matrix in each mode can match the horizontal state dimension, vertical state dimension, and control input dimension; determining whether the performance weight matrix is ​​a positive definite matrix; determining whether the off-diagonal elements of the mode transition rate matrix in each direction are non-negative and whether the row sum is zero; and determining whether the closed-loop matrix corresponding to the initial control gain meets the preset stability conditions. If it does not meet the conditions, the initial control gain is regenerated by increasing the control weight, adjusting the initial gain, or using a preset stable gain library.

3. The structured optimal control method for a continuous-time two-dimensional Markov jump system based on model-driven structured strategy iteration as described in claim 1, characterized in that, The continuous-time two-dimensional Markov jump system described in S2 includes a horizontal state evolving along a first time variable and a vertical state evolving along a second time variable; the state feedback controller is a mode-dependent controller, and adopts a corresponding control gain matrix in different modes so that the control input can be adjusted as the Markov mode switches.

4. The structured optimal control method for a continuous-time two-dimensional Markov jump system based on model-driven structured strategy iteration as described in claim 1, characterized in that, The weight matrix corresponding to the value function in S3 is constrained to be a block diagonal positive definite matrix, where the horizontal block is used to represent the horizontal state energy and the vertical block is used to represent the vertical state energy. The block diagonal positive definite matrix is ​​used to ensure that the value function matches the spatial causal structure of the Roesser model.

5. The structured optimal control method for a continuous-time two-dimensional Markov jump system based on model-driven structured strategy iteration as described in claim 1, characterized in that, The infinitesimal generator in S4 includes a horizontal state derivative term, a vertical state derivative term, and a mode transition term. The mode transition term is formed by the horizontal mode transition rate and the vertical mode transition rate, and is used to characterize the influence of two-dimensional random jumps on the rate of change of the value function.

6. The structured optimal control method for a continuous-time two-dimensional Markov jump system based on model-driven structured strategy iteration as described in claim 1, characterized in that, The block diagonal projection operator in S5 is used to retain the diagonal blocks after dividing according to the horizontal and vertical states and set the off-diagonal blocks to zero. The off-diagonal residual extraction operator is used to extract the off-diagonal residuals generated by the coupling terms of the horizontal and vertical states. The structured two-dimensional coupled algebraic Riccati equations require that the residuals after the block diagonal projection be zero, and use the off-diagonal residuals as stability check quantities rather than directly solving constraints.

7. The structured optimal control method for a continuous-time two-dimensional Markov jump system based on model-driven structured strategy iteration as described in claim 1, characterized in that, The fully coupled strategy evaluation in S6 includes: constructing a closed-loop matrix under the current control gain; solving the coupled Lyapunov equations based on the known system matrix, input matrix, performance weight matrix, and mode transition rate matrix to obtain the evaluation matrix for this iteration; the block diagonal projection includes projecting the evaluation matrix into a block diagonal matrix; the positive definiteness preservation check includes adding a diagonal regularization term or reverting to the previous round's stable matrix when the projected block diagonal matrix does not meet the positive definiteness condition.

8. The structured optimal control method for a continuous-time two-dimensional Markov jump system based on model-driven structured strategy iteration as described in claim 1, characterized in that, The strategy improvements in S6 include: updating the control gain of the corresponding mode based on the projected block diagonal weight matrix, input matrix, and control weight matrix; if the updated control gain causes the closed-loop stability check to fail, then reducing the gain update step size or using a convex combination of the previous round's stable control gain and the current round's control gain for correction.

9. The structured optimal control method for a continuous-time two-dimensional Markov jump system based on model-driven structured strategy iteration as described in claim 1, characterized in that, The preset conditions in S7 include the change in the structured weight matrix being less than the first threshold, the change in the control gain being less than the second threshold, the off-diagonal residual norm being less than the third threshold, and the block diagonal projection term of the closed-loop generator being negative definite. When the off-diagonal residual norm does not meet the third threshold, the iteration continues or a prompt that the system weight parameters need to be adjusted is output.

10. A structured optimal control device for a continuous-time two-dimensional Markov jump system based on model-driven structured strategy iteration, characterized in that, The system includes a model and parameter acquisition module, a performance index construction module, a structured equation construction module, a strategy iterative solution module, an anomaly verification and residual judgment module, and a control law output module. The model and parameter acquisition module is used to execute S1 and S2 in claim 1; the performance index construction module is used to execute S3 in claim 1; the structured equation construction module is used to execute S4 and S5 in claim 1; the strategy iterative solution module is used to execute S6 in claim 1; the anomaly verification and residual judgment module is used to verify the projective positive definiteness, closed-loop stability, and off-diagonal residuals; and the control law output module is used to execute S7 in claim 1 and output the closed-loop control law to the actuator of the controlled object or the control simulation platform.

Images