Reinforcement learning discrete-continuous hierarchical decision architecture and method for state switching

Abstract

The present invention relates to the field of intelligent control technology and discloses a state-switching reinforcement learning discrete-continuous hierarchical decision-making architecture and method. By constructing a hierarchical discrete-continuous decision-making architecture, a critic network is used to obtain state value functions and discrete action value functions, respectively, separating the continuous action decision and discrete action decision evaluation systems, and reducing the coupling between the two. Discrete actions are then used as enabling conditions for continuous parameters to ensure that related continuous control channels are completely blocked when discrete decisions are not activated, thereby avoiding action instruction conflicts. A variable gating matrix is ​​constructed based on discrete decision quantities, and the gradient propagation path of corresponding neurons is directly cut off through the gating matrix. The specific shielding parameters of discrete actions during two mode switching are distinguished, and then the continuous control quantity decision of mode switching is executed, thereby effectively solving the coupling conflict and efficiency bottleneck problems in hybrid action space processing.

Description

Technical Field

[0001] The present invention relates to the field of intelligent control technology, and discloses a state switching-oriented reinforcement learning discrete-continuous hierarchical decision architecture and method. Background Art

[0002] Currently, deep reinforcement learning faces significant challenges in dealing with state switching problems (switching from state one to state two): this problem belongs to a mixed action space (discrete and continuous actions coexist). Traditional methods such as parameterized action spaces or parallel network architectures fail to establish a logical hierarchy between discrete decision-making and continuous control, resulting in action conflicts. They also have difficulty meeting determinism requirements in safety-sensitive scenarios such as industrial control.

[0003] Specific application scenarios include the mode switching control process of a turboramjet combination engine. The rapid, safe and stable switching between the turbine mode and the ramjet mode is the key difficulty of the engine. The turbine mode requires a Mach number between 0 and 3 Mach, and the ramjet mode requires a Mach number above 3 Mach. When the mode switching occurs under inappropriate working conditions, a "thrust trap" problem will occur due to flow mismatch and shock wave mutation. That is, due to the difference in the working characteristics of the two modes and the matching problem of the intake and exhaust systems, the thrust suddenly drops and may even be unable to cover the flight resistance. Summary of the Invention

[0004] The purpose of the present invention is to provide a discrete-continuous hierarchical decision-making architecture and method for reinforcement learning for state switching, which can effectively solve the coupling conflict and efficiency bottleneck problems in hybrid action space processing.

[0005] In order to achieve the above technical effects, the technical solution adopted by the present invention is:

[0006] A discrete-continuous hierarchical decision-making method based on reinforcement learning for state switching is used to implement control strategy decisions when the controlled object switches between state 1 and state 2. The control strategy decisions include discrete control decisions on whether to execute state switching or to start executing state switching, as well as continuous control variable decisions, including:

[0007] Based on the state quantity of the current state of the controlled object, a critic network analysis is used to obtain a state value function of the current state of the controlled object, a first action value function with a switch flag set to 0, and a second action value function with a switch flag set to 1; wherein the state value function is used to evaluate the value of the current state of the controlled object, and the first action value function and the second action value function are used to evaluate the value of corresponding discrete control decisions, wherein the discrete control decisions include not executing state switching and starting to execute state switching;

[0008] selecting a discrete control decision corresponding to the maximum value function of the first action-value function and the second action-value function as an optimal discrete control decision, wherein when the maximum value function is the first action-value function with a switching flag set to 0, the optimal discrete control decision is not to execute state switching; when the maximum value function is the second action-value function with a switching flag set to 1, the optimal discrete control decision is to start executing state switching;

[0009] The current switching flag value and the state quantity of the controlled object are used as network inputs, and the actor network analysis is used to obtain the continuous action quantity under the preset continuous decision-making strategy. The final continuous control quantity is obtained according to the gating matrix and the continuous action quantity analysis.

[0010] Furthermore, the continuous action amount is based on The analysis obtained for The amount of continuous action at a moment, for The state quantity at the moment, for The switching flag value at the moment, is a preset continuous decision-making strategy represented by a deep network.

[0011] Furthermore, the final continuous control quantity ,in for The amount of continuous action at a moment, is the gating matrix.

[0012] Furthermore, when the switch flag is set to 0 and the current state is state 1, the gate matrix ; When the switch flag is set to 0 and the current state is state 2, the gate matrix ; When the switch flag is set to 1, the gate matrix ,in 、 、 are all Boolean vectors with the same number of elements, Represents a diagonal matrix consisting of elements corresponding to Boolean vectors.

[0013] To achieve the above technical effects, the present invention further provides a state-switching oriented reinforcement learning discrete-continuous hierarchical decision-making architecture, which is used to implement the state-switching oriented reinforcement learning discrete-continuous hierarchical decision-making method, including:

[0014] a value evaluation module, configured to obtain, based on the state quantity of the controlled object's current state, a state value function of the controlled object's current state, a first action value function with a switch flag set to 0, and a second action value function with a switch flag set to 1 using a critic network analysis; wherein the state value function is used to evaluate the value of the controlled object's current state, and the first action value function and the second action value function are used to evaluate the value of corresponding discrete control decisions, wherein the discrete control decisions include not executing state switching and starting to execute state switching;

[0015] a discrete decision module, configured to select a discrete control decision corresponding to a maximum function of the first action-value function and the second action-value function as an optimal discrete control decision, wherein when the maximum function is the first action-value function with a switching flag set to 0, the optimal discrete control decision is not to execute state switching; and when the maximum function is the second action-value function with a switching flag set to 1, the optimal discrete control decision is to start executing state switching;

[0016] The continuous control quantity analysis module is used to use the current switching flag value and the state quantity of the controlled object as network input, use actor network analysis to obtain the continuous action quantity under the preset continuous decision-making strategy, and obtain the final continuous control quantity based on the gating matrix and the continuous action quantity analysis.

[0017] Furthermore, in the continuous control quantity analysis module, the continuous action quantity is calculated based on The analysis obtained for The amount of continuous action at a moment, for The state quantity at the moment, for The switching flag value at the moment, is a preset continuous decision-making strategy represented by a deep network.

[0018] Furthermore, in the continuous control quantity analysis module, the final continuous control quantity ,in for The amount of continuous action at a moment, is the gating matrix.

[0019] Furthermore, in the continuous control quantity analysis module, when the switch flag is set to 0 and the current state is state 1, the gate matrix ; When the switch flag is set to 0 and the current state is state 2, the gate matrix ; When the switch flag is set to 1, the gate matrix ,in 、 、 are all Boolean vectors with the same number of elements, Represents a diagonal matrix consisting of elements corresponding to Boolean vectors.

[0020] Compared with the prior art, the present invention has the following beneficial effects: the present invention utilizes the Critic network to obtain the state value function for evaluating the continuous action quantity and the action value function of the discrete action respectively, separates the continuous action decision and discrete action decision evaluation systems, and reduces the coupling between the two; then uses the discrete action as the enabling condition of the continuous parameter to ensure that the relevant continuous control channel is completely blocked when the discrete decision is not activated, thereby avoiding action instruction conflicts; constructs a variable gating matrix based on the discrete decision quantity, directly cuts off the gradient propagation path of the corresponding neuron through the gating matrix, distinguishes the specific shielding parameters of the discrete action during the two mode switching, and then executes the continuous control quantity decision of the mode switching, thereby effectively solving the coupling conflict and efficiency bottleneck problems in the hybrid action space processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 Schematic diagram of the discrete-continuous hierarchical decision architecture of reinforcement learning for state switching in Example 1;

[0022] Figure 2 This is a flow chart of the discrete-continuous hierarchical decision-making method for state switching using reinforcement learning in Example 1 or 2;

[0023] Figure 3 Schematic diagram of the critic network structure in Example 2;

[0024] Figure 4 This is a schematic diagram of the actor network structure in Example 2;

[0025] Among them, 1. Value assessment module; 2. Discrete decision module; 3. Continuous control quantity analysis module. DETAILED DESCRIPTION

[0026] The present invention will be described in further detail below with reference to the embodiments and accompanying drawings. However, this should not be construed as limiting the scope of the present invention to the following embodiments, as all technologies implemented based on the present invention fall within the scope of the present invention.

[0027] Example 1

[0028] See also Figure 1 and Figure 2 A discrete-continuous hierarchical decision-making method based on reinforcement learning for state switching is used to implement control strategy decisions when the controlled object switches between state 1 and state 2. The control strategy decisions include discrete control decisions of not executing state switching or starting to execute state switching, as well as continuous control variable decisions, including:

[0029] It is used to implement the control strategy decision when the controlled object switches between state 1 and state 2. The control strategy decision includes a discrete control decision of not executing state switching or starting to execute state switching, and a continuous control quantity decision of the switching process after starting to execute state switching, including:

[0030] Based on the state quantity of the current state of the controlled object, a critic network analysis is used to obtain a state value function of the current state of the controlled object, a first action value function with a switch flag set to 0, and a second action value function with a switch flag set to 1; wherein the state value function is used to evaluate the value of the current state of the controlled object, and the first action value function and the second action value function are used to evaluate the value of corresponding discrete control decisions, wherein the discrete control decisions include not executing state switching and starting to execute state switching;

[0031] selecting a discrete control decision corresponding to the maximum value function of the first action-value function and the second action-value function as an optimal discrete control decision, wherein when the maximum value function is the first action-value function with a switching flag set to 0, the optimal discrete control decision is not to execute state switching; when the maximum value function is the second action-value function with a switching flag set to 1, the optimal discrete control decision is to start executing state switching;

[0032] The current switching flag value and the state quantity of the controlled object are used as network inputs, and the actor network analysis is used to obtain the continuous action quantity under the preset continuous decision-making strategy. The final continuous control quantity is obtained according to the gating matrix and the continuous action quantity analysis.

[0033] In this embodiment, through hierarchical discrete-continuous decision-making, the critic network is used to obtain the state value function of the current state of the controlled object and the action value function of the discrete action, respectively, separate the continuous action decision-making and discrete action decision-making evaluation systems, and reduce the coupling between the two; then the discrete action is used as the enabling condition of the continuous parameter to ensure that the relevant continuous control channel is completely blocked when the discrete decision is not activated, thereby avoiding action instruction conflicts; based on the discrete decision quantity, a variable gating matrix is ​​constructed, and the gradient propagation path of the corresponding neuron is directly cut off through the gating matrix, and the specific shielding parameters of the discrete action during the two mode switching are distinguished, and then the continuous control quantity decision of the mode switching is executed, thereby effectively solving the coupling conflict and efficiency bottleneck problems in the hybrid action space processing.

[0034] In this embodiment, the basic framework of actor-critic and the flowchart of the reinforcement learning discrete-continuous hierarchical decision method are as follows: Figure 2 , as shown, where "command signal" refers to the control command. In engine control, the speed command or thrust command is generally used as the command signal; envEnvironmental variables that engine control focuses on generally include altitude and Mach number; r t Refers to the reward function, which is a parameter of reinforcement learning in the basic actor-critic framework. Since reward functions are well known to those skilled in the art, they will not be described here in detail. S t+1 Refers to the state quantity S t Continuous action volume The new state quantity that the controlled object transfers to is ( S t 、 、 r t 、 S t+1 ) constitute a data tuple, which serves as the data for network training.

[0035] Based on the same inventive concept, this embodiment further provides a state-switching oriented reinforcement learning discrete-continuous hierarchical decision-making architecture, which is used to implement the state-switching oriented reinforcement learning discrete-continuous hierarchical decision-making method, including:

[0036] a value evaluation module 1 for obtaining, based on the state quantity of the controlled object's current state, a state value function of the controlled object's current state, a first action value function with a switch flag set to 0, and a second action value function with a switch flag set to 1, using a critic network analysis; wherein the state value function is used to evaluate the value of the controlled object's current state, and the first action value function and the second action value function are used to evaluate the value of corresponding discrete control decisions, wherein the discrete control decisions include not executing state switching and starting executing state switching;

[0037] a discrete decision module 2, configured to select a discrete control decision corresponding to a maximum value function of the first action-value function and the second action-value function as an optimal discrete control decision, wherein when the maximum value function is the first action-value function with a switching flag set to 0, the optimal discrete control decision is not to execute state switching; and when the maximum value function is the second action-value function with a switching flag set to 1, the optimal discrete control decision is to start executing state switching;

[0038] The continuous control quantity analysis module 3 is used to use the current switching flag value and the state quantity of the current state of the controlled object as network input, adopt actor network analysis to obtain the continuous action quantity under the preset continuous decision-making strategy, and obtain the final continuous control quantity based on the gating matrix and the continuous action quantity analysis.

[0039] Example 2

[0040] See also Figures 1-4 This embodiment takes the mode switching control of a certain type of turboramjet combination engine as an example to explain in detail the process of the reinforcement learning discrete-continuous hierarchical decision-making method for state switching of the present invention. State 1 is the turbojet mode, and state 2 is the ramjet mode. The continuous control quantity of the mode switching process includes the fuel flow rate of the turbine combustion chamber. , ramjet fuel flow , intake duct modal switching valve opening , tail nozzle modal switching valve opening , the continuous control quantity is expressed as .

[0041] Decision-making methods include:

[0042] Step 1: Based on the state quantity of the current state of the controlled object, a critic network analysis is used to obtain a state value function of the current state of the controlled object, a first action value function with a switch flag set to 0, and a second action value function with a switch flag set to 1; wherein the state value function is used to evaluate the value of the current state of the controlled object, and the first action value function and the second action value function are used to evaluate the value of corresponding discrete control decisions, wherein the discrete control decisions include not executing state switching and starting to execute state switching;

[0043] In this embodiment, the critic network is used to evaluate the quality of the strategy and to assist in updating the control strategy, such as Figure 3 As shown. The critic network takes the state quantity of the controlled object as the current state As input, the state value function of the current state of the controlled object , the first action value function with the switch flag set to 0 , the action value function of the switch flag set to 1 As output. State value function Used to evaluate the value of the current state of the controlled object, action value function 、 Used to evaluate the value of corresponding discrete control decisions.

[0044] Step 2: Selecting a discrete control decision corresponding to the maximum value function of the first action-value function and the second action-value function as the optimal discrete control decision, wherein when the maximum value function is the first action-value function with a switching flag set to 0, the optimal discrete control decision is not to execute state switching; when the maximum value function is the second action-value function with a switching flag set to 1, the optimal discrete control decision is to start executing state switching;

[0045] In this embodiment, the control decision is first made in discrete decision-making, and then in continuous control quantity decision-making. The optimal discrete control decision is made by comparing the action value function 、 Implementation: Select the discrete control decision corresponding to the maximum value function of the first state value function and the second state value function as the optimal discrete control decision.

[0046] Step 3: Using the current switching flag value and the state quantity of the controlled object as network input, actor network analysis is used to obtain the continuous action quantity under the preset continuous decision-making strategy, and the final continuous control quantity is obtained based on the gating matrix and the continuous action quantity analysis;

[0047] After completing the discrete decision, this embodiment sets the switching flag corresponding to the optimal discrete control decision to the value As the input of the actor network, it determines the amount of continuous action. The hierarchical decision-making architecture actor network structure and input and output are as follows Figure 4 As shown, the actor network is represented by the state quantity , switch flag value As network input, the continuous action amount Output: Continuous action quantity according to The analysis obtained for The amount of continuous action at a moment, for The state quantity at the moment, for The switching flag value at the moment, is a preset continuous decision-making strategy represented by a deep network.

[0048] In this embodiment, when the switch flag is set to In order to avoid the conflict between the continuous decision-making strategy and “not executing state switching”, some control quantities are shielded, that is, the final continuous control quantity is obtained by multiplying the continuous action quantity and the gating matrix: , where For the current The amount of continuous action at a moment, is the gate matrix; when the switch flag is set to 0 and the current state is state 1, the gate matrix ; When the switch flag is set to 0 and the current state is state 2, the gate matrix ; When the switch flag is set to 1, the gate matrix ,in 、 、 are all Boolean vectors with the same number of elements, Represents a diagonal matrix consisting of elements corresponding to Boolean vectors.

[0049] In this embodiment, when the initial state is the turbojet mode (state 1) and the mode conversion is not performed, the control quantity to be shielded is 、 、 ,at this time ; When the initial state is stamping mode (state 2) and no mode conversion is performed, the control quantity to be shielded is 、 、 ,at this time ; When performing mode conversion, the control quantity is not shielded, Therefore, the value logic of the gating matrix is:

[0050] When the switch flag is set to 0 and the current state is state 1, the gate matrix ;

[0051] When the switch flag is set to 0 and the current state is state 2, the gate matrix ;

[0052] When the switch flag is set to 1, the gate matrix .

[0053] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A state-switching reinforcement learning discrete-continuous hierarchical decision-making method is applied to the mode switching control of a turboramjet engine to implement control strategy decisions when the controlled object switches between state 1 and state 2. The control strategy decisions include discrete control decisions on whether to execute state switching or to start executing state switching, as well as continuous control variable decisions. The method is characterized by: include: Based on the state quantity of the current state of the controlled object, a critic network analysis is used to obtain a state value function of the current state of the controlled object, a first action value function with a switch flag set to 0, and a second action value function with a switch flag set to 1; wherein the state value function is used to evaluate the value of the current state of the controlled object, and the first action value function and the second action value function are used to evaluate the value of corresponding discrete control decisions, wherein the discrete control decisions include not executing state switching and starting to execute state switching; selecting a discrete control decision corresponding to the maximum value function of the first action-value function and the second action-value function as an optimal discrete control decision, wherein when the maximum value function is the first action-value function with a switching flag set to 0, the optimal discrete control decision is not to execute state switching; when the maximum value function is the second action-value function with a switching flag set to 1, the optimal discrete control decision is to start executing state switching; The current switching flag value and the state quantity of the controlled object are used as network inputs, and the actor network analysis is used to obtain the continuous action quantity under the preset continuous decision-making strategy. The final continuous control quantity is obtained based on the gating matrix and the continuous action quantity analysis.

2. The discrete-continuous hierarchical decision-making method for state switching based on reinforcement learning according to claim 1, characterized in that: Continuous action amount according to The analysis obtained for The amount of continuous action at a moment, for The state quantity at the moment, for The switch flag setting value at the moment, is a preset continuous decision-making strategy represented by a deep network.

3. The discrete-continuous hierarchical decision-making method for state switching based on reinforcement learning according to claim 1, characterized in that: The final continuous control amount ,in for The amount of continuous action at a moment, is the gating matrix.

4. The discrete-continuous hierarchical decision-making method for state switching based on reinforcement learning according to claim 3, characterized in that: When the switch flag is set to 0 and the current state is state 1, the gate matrix ; When the switch flag is set to 0 and the current state is state 2, the gate matrix ; When the switch flag is set to 1, the gate matrix ,in 、 、 are all Boolean vectors with the same number of elements, Represents a diagonal matrix consisting of elements corresponding to Boolean vectors.

5. A state-switching oriented reinforcement learning discrete-continuous hierarchical decision-making system, used to implement the state-switching oriented reinforcement learning discrete-continuous hierarchical decision-making method according to claim 1, characterized in that: include: a value evaluation module, configured to obtain, based on the state quantity of the controlled object's current state, a state value function of the controlled object's current state, a first action value function with a switch flag set to 0, and a second action value function with a switch flag set to 1 using a critic network analysis; wherein the state value function is used to evaluate the value of the controlled object's current state, and the first action value function and the second action value function are used to evaluate the value of corresponding discrete control decisions, wherein the discrete control decisions include not executing state switching and starting to execute state switching; a discrete decision module, configured to select a discrete control decision corresponding to a maximum function of the first action-value function and the second action-value function as an optimal discrete control decision, wherein when the maximum function is the first action-value function with a switching flag set to 0, the optimal discrete control decision is not to execute state switching; and when the maximum function is the second action-value function with a switching flag set to 1, the optimal discrete control decision is to start executing state switching; The continuous control quantity analysis module is used to use the current switching flag value and the state quantity of the controlled object as network input, adopt actor network analysis to obtain the continuous action quantity under the preset continuous decision-making strategy, and obtain the final continuous control quantity based on the gating matrix and the continuous action quantity analysis.

6. The state switching oriented reinforcement learning discrete-continuous hierarchical decision system according to claim 5, characterized in that: In the continuous control quantity analysis module, the continuous action quantity is based on The analysis obtained for The amount of continuous action at a moment, for The state quantity at the moment, for The switch flag setting value at the moment, is a preset continuous decision-making strategy represented by a deep network.

7. The state switching oriented reinforcement learning discrete-continuous hierarchical decision system according to claim 5, characterized in that: In the continuous control quantity analysis module, the final continuous control quantity ,in for The amount of continuous action at a moment, is the gating matrix.

8. The state switching oriented reinforcement learning discrete-continuous hierarchical decision system according to claim 7, characterized in that: In the continuous control quantity analysis module, when the switch flag is set to 0 and the current state is state 1, the gate matrix ; When the switch flag is set to 0 and the current state is state 2, the gate matrix ; When the switch flag is set to 1, the gate matrix ,in 、 、 are all Boolean vectors with the same number of elements, Represents a diagonal matrix consisting of elements corresponding to Boolean vectors.

Images