Cross-layer elastic control method for networked switching system based on reinforcement learning

Abstract

The invention discloses a cross-layer elastic control method for a networked switching system based on reinforcement learning, and belongs to the technical field of networked system safety control. In a network layer, a confrontation interaction relationship between a defending party and an attacking party is described by constructing a constraint zero-sum game model, and a constraint condition is determined by solvability of an algebraic Riccati equation of a physical layer. Based on the model, a closed-loop explicit solution for constraining Nash equilibrium is deduced, and the equilibrium packet loss rate required by the design of each subsystem controller is determined according to the closed-loop explicit solution. In order to solve the problem of open-loop operation and modal asynchronization caused by denial of service attack, a slow switching mechanism based on residence time is adopted, and the mean square index stability of the system can be ensured without parameter setting. Modeling an optimal switching signal selection problem as a Markov decision process in a hybrid state space; the invention further provides an improved deep Q network algorithm which is used for solving the Markov decision process, and the dynamic performance of the system is effectively improved while the residence time constraint is met.

Description

Technical Field

[0001] This invention relates to the field of networked switching systems and resilient control research technology, and in particular to a cross-layer resilient control method for networked switching systems based on reinforcement learning. Background Technology

[0002] Switching systems consist of several subsystems, typically described by differential or difference equations and scheduled using switching laws. With the development of communication technology, shared networks have become crucial infrastructure for data transmission, driving the widespread application of networked switching systems in fields such as automotive transmissions, robot control, aircraft navigation, and intelligent transportation. However, the openness of networks also makes systems vulnerable to denial-of-service attacks, leading to data loss and consequently performance degradation or even instability. Traditional control methods face the challenge of unknown system matrices in practice, and traditional identification methods introduce modeling errors, not only reducing control performance but also potentially jeopardizing stability. Adaptive dynamic programming provides a model-free control solution. This method constructs an interactive learning mechanism between the evaluation network and the execution network, approximating the optimal control strategy online without requiring an accurate system model. This eliminates dependence on the system matrix and ensures convergence and closed-loop stability through time-difference learning. This characteristic significantly improves engineering applicability and lays the technical foundation for the practical deployment of cross-layer security control frameworks.

[0003] In the face of complex network attacks, especially in dynamic environments with multiple attack and defense nodes, ensuring the secure operation of networked switching systems has become a major research challenge. Game theory, as an effective theoretical tool for studying decision-making behavior in multi-party adversarial environments, provides an important approach to solving such problems. It can simulate the strategy selection process of both attackers and defenders in dynamic confrontations, revealing their optimal behavioral patterns under resource constraints, thus laying a theoretical foundation for building a security protection system. In research on improving the resilience of networked switching systems, adopting a cross-layer game framework not only helps to deeply understand the interaction mechanisms between attackers and defenders but also strengthens the collaborative relationship between the network layer and the physical layer. Defense strategies designed using this method can effectively address the threats posed by denial-of-service attacks. This framework not only focuses on the dynamic response of the system after an attack but also optimizes the allocation of defense resources and decision-making paths through the game process, thereby enhancing the robustness and operational reliability of the system under attack. Therefore, applying game theory to the field of security control, especially in the context of defending against denial-of-service attacks, has become a research direction with significant theoretical and practical value. By constructing and analyzing reasonable game models, a solid theoretical foundation can be provided for developing efficient and practical system protection solutions. Summary of the Invention

[0004] This invention provides a cross-layer elastic control method for networked switching systems based on reinforcement learning, aiming to overcome the limitations of existing methods such as reliance on system matrices, unidirectional cross-layer coupling, and difficulty in guaranteeing stability against packet loss, and to achieve cross-layer safe control and dynamic performance optimization under model-free conditions.

[0005] To achieve the above objectives, the present invention adopts the following technical solution:

[0006] A method for cross-layer elastic control of a networked switching system based on reinforcement learning includes the following steps:

[0007] Step 1: Construct a closed-loop dynamic mathematical model of a discrete-time networked switching linear system. Use Bernoulli distribution to describe the random packet loss phenomenon caused by denial-of-service attacks. Design a corresponding cost function for each subsystem and establish a mathematical relationship between the network layer signal-to-interference-plus-noise ratio and the physical layer packet delivery rate.

[0008] Step 2: Construct a constrained zero-sum game model in the network layer that considers the solvability of the Ricardian equations in the physical layer, including:

[0009] Constraining zero-sum game participants: defenders and attackers;

[0010] Constrained zero-sum game actions: power allocation strategies of the two players on the wireless channel;

[0011] Constrained zero-sum game utility function: Both players simultaneously consider optimizing signal-to-interference-plus-noise ratio and power consumption;

[0012] Step 3: Reconstruct the cost function of each subsystem, estimate the upper bound of the critical packet delivery rate, solve the model-free constrained Nash equilibrium, determine the optimal power allocation, and determine the packet delivery rate of the wireless channel.

[0013] Step 4: Based on the packet delivery rate obtained in Step 3, design a dwell time condition to ensure the mean square stability of the system, and model the switching signal optimization problem as a Markov decision process based on the dwell time condition; use the improved deep Q network method to solve the policy.

[0014] Step 1 includes:

[0015] Step 1.1: At the physical layer, construct a discrete-time networked switching linear system, as shown in equation (1).

[0016] (1)

[0017] in, For system status, To control the input, switch the signal. It is a piecewise constant function whose values ​​are taken from a finite set of indices. ,in The total number of subsystems. When When, it indicates the first Each subsystem at time Activated. and These represent the system matrix and control matrix of corresponding dimensions, respectively. The dimension of the system state vector. This indicates the dimension of the control input vector;

[0018] Step 1.2: Assume that packet loss caused by a denial-of-service attack follows a Bernoulli distribution; to describe the system state and the transmission state of switching signals, two indicator functions are introduced. and . and This indicates that a data packet was lost, and and This indicates successful transmission; the probability of successful reception is... ,in The packet delivery rate is determined by the network layer; the controller uses state feedback. ,in For the controller at time The modality; let the switching time sequence be ,in, Let be the total number of handovers. Then, the closed-loop dynamic mathematical model of the discrete-time networked switching linear system is expressed as follows:

[0019] (2)

[0020] Step 1.3: Under the condition that the closed-loop dynamic mathematical model of the discrete-time networked switched linear system is unknown, design the gain and switching law of the controller based on offline data to ensure the mean square stability of the discrete-time networked switched linear system; define the first... The stage cost of each subsystem is

[0021] (3)

[0022] in and These represent the state weight matrix and the control weight matrix, respectively.

[0023] No. Each subsystem at time The infinite time-domain cost function is defined as

[0024] (4)

[0025] in, The mathematical expectation operator is represented; the optimal value function can be expressed as: ,in ;

[0026] According to the Bellman equation, we obtain the... Algebraic Ricardi equations of subsystems

[0027] (5)

[0028] Suppose for each Subsystem All are sedable, meaning there exists a corresponding state feedback gain matrix. This makes the closed-loop matrix For Shure stability;

[0029] Step 1.4: In the network layer, the defender (smart sensor) and the attacker compete for power allocation on the wireless channel. The shifting power input directly determines the signal-to-interference-plus-noise ratio (SNR) of the channel. This ratio is a key indicator that determines packet delivery rate and measures the quality of wireless transmission.

[0030] (6)

[0031] Among them, the transmission power of the defending side The attacker's jamming power , and These are the power limits for both parties; and Channel attenuation coefficients for the defender and attacker, respectively. The additive white Gaussian noise power in the environment; in wireless communication systems employing quadrature amplitude modulation, the signal-to-interference-plus-noise ratio (SINR) is a key factor determining the packet delivery rate; according to communication theory, the packet delivery rate... The following can be calculated:

[0032] (7)

[0033] in It is a given parameter, and ; Indicates the lower limit of integration. This represents an auxiliary variable.

[0034] In Step 1 of this invention, the mathematical model established for the discrete-time networked switching linear system with unknown model, and the correlation between the signal-to-interference-plus-noise ratio and the packet loss rate, together lay the theoretical foundation for the subsequent construction of a cross-layer game model.

[0035] Step 2 involves constructing a zero-sum game model in the network layer and setting constraints based on the solvability of the algebraic Ricardian equations in the physical layer, specifically including:

[0036] Step 2.1: Determine the participants in the zero-sum game as the defender and the attacker, and have both make decisions simultaneously;

[0037] Step 2.2, Selection of the Defender To ensure data transmission, the attacker chooses... Disrupts data transmission;

[0038] Step 2.3: Determine the utility functions of both sides in the game. The defender's goal is to maximize the packet delivery rate and reduce energy consumption, while the attacker attempts to minimize the packet delivery rate and reduce their own energy consumption. Therefore, the defender's utility function is defined as follows:

[0039] (8)

[0040] in, and These represent the unit power cost for the defender and the attacker, respectively. The signal-to-interference-plus-noise ratio (SIR) is the trade-off coefficient between power consumption; the attacker's utility function is defined as...

[0041] (9)

[0042] Step 2.4: To analyze the impact of the physical layer on the zero-sum game of the network layer, it can be seen from equation (5) that when the first... When a subsystem is unstable, the solution to its algebraic Ricardi equation Depends on packet delivery rate : The equation has a unique solution. There is no feasible solution at this time. Therefore, there exists a critical packet delivery rate. The equation is solvable if and only if . use Signal-to-interference-plus-noise ratio The monotonic relationship can uniquely determine the critical signal-to-interference-plus-noise ratio. ,satisfy Therefore, this zero-sum game must satisfy the following constraints.

[0043] (10)

[0044] The defender aims to maximize Subject to The attacker aims to maximize Subject to .

[0045] Step 3 specifically includes:

[0046] Step 3.1: To achieve model-free control, the cost function of each subsystem is reconstructed to obtain the... Subsystem The function is as follows:

[0047] (11)

[0048] in

[0049] (12)

[0050] No. The optimal controller gain of each subsystem can be expressed as:

[0051] (13)

[0052] The optimal Bellman equation can be expressed as:

[0053] (14)

[0054] Based on this, we can obtain

[0055] (15)

[0056] Step 3.2, adopting an iterative approach and Model-less solution method for unknown situations Package delivery rate This has a crucial impact on the solvability of the algebraic Ricardi equation, and provides a critical value for the packet delivery rate. An upper bound estimate; due to the actual system matrix Unknown, temporarily disregarding the attack-defense game in the network layer, and setting a sufficiently high transmission power for the defender to ensure the solvability of the algebraic Ricardi equation, let the packet delivery rate at this point be... Under this configuration, by combining the iterative convergence results, we can obtain and steady-state solution and And then use the two to estimate The maximum eigenvalue. For networked handover systems, the critical packet delivery rate. The following conditions must be met

[0057] (16)

[0058] And meet the following conditions

[0059] (17)

[0060] in, Then the Ricardi equation has a unique positive definite solution. .

[0061] Furthermore, since the signal-to-interference-plus-noise ratio (SIR) corresponds one-to-one with the packet delivery rate, when the SIR corresponding to the game equilibrium in the network layer is greater than the SIR corresponding to the upper bound... At that time, the physical layer algebraic Ricardi equation is also solvable;

[0062] Step 3.3: In the network layer, define the parameters that satisfy the signal-to-interference-plus-noise ratio (SINR) constraint. The strategy is as follows: by analyzing the utility functions of both sides and introducing the signal-to-interference-plus-noise ratio constraint, the optimal power allocation strategy for the defender and attacker is derived; in the constructed constrained zero-sum game, the constrained Nash equilibrium is determined. As shown below:

[0063] (18)

[0064] in, ,

[0065] ,

[0066] ,

[0067] ,

[0068] ,

[0069] ,

[0070] ,

[0071] ,

[0072] .

[0073] The packet delivery rate of the wireless channel is calculated based on Nash equilibrium and formula (7). .

[0074] Step 4 specifically includes:

[0075] Step 4.1 For discrete-time networked switching linear systems, when the network layer is in constrained Nash equilibrium and the optimal controller gain designed by equation (13) is used, there exists a constant. , , , The following equations must be satisfied:

[0076] (19)

[0077] in, , If the length of stay If the following conditions are met, the mean square exponential stability of the discrete-time networked switching linear system is achieved.

[0078] (20)

[0079] in, It represents the optimal packet delivery rate under the network layer balancing strategy.

[0080] Step 4.2: Even infinite handover sequences that satisfy the average dwell time condition are still susceptible to packet loss caused by denial-of-service attacks, which may lead to performance degradation. Considering the randomness of the transmission environment, the handover signal optimization problem is modeled as a Markov decision process, with the following framework:

[0081] Step 4.2.1, at time The state is defined as ,in, Indicates the duration since the last handover;

[0082] Step 4.2.2, at time action Defined as a switching signal ;

[0083] Step 4.2.3, Reward: Reward Function The cost design is based on the following stages:

[0084] (twenty one)

[0085] in, and Indicates the subsystem mode The state weight matrix and control weight matrix are shown below. Indicates in controller mode The controller gain under the current, Penalty for length of stay: If and ,but ,in, This is a fixed penalty value for violating the stay time requirement; otherwise... The goal is to maximize the expected cumulative reward. ,in, It is a discount factor;

[0086] Step 4.2.4, State Transition: Next State It evolved as follows:

[0087] (twenty two)

[0088] Step 4.3: An improved deep Q-network algorithm is proposed to solve the above Markov decision problem, where the input is: Q-network. ; target network Experience replay buffer Temporary state transition buffer Initial exploration probability Attenuation factor Minimum exploration rate Maximum number of training rounds Target network update rate: Simulation time domain ;

[0089] Step 4.3.1: Randomly initialize the online Q network and target Q network The parameters are repeated. The following training scenarios;

[0090] Step 4.3.2: Initialize the environment state ;

[0091] Step 4.3.3, Regarding the time step arrive It performs action selection, state update, reward calculation, and storage transformation, transferring state transition experience. Store in temporary state transition buffer ;

[0092] Step 4.3.4, regarding the time step arrive Calculate the cumulative discount return at each time point. ;

[0093] Step 4.3.5: Combine all complete experience tuples in this scenario. Store in the experience replay buffer ;

[0094] Step 4.3.6: Perform network training, which includes experience replay, calculation of target value, calculation of loss and gradient descent update;

[0095] Step 4.3.7: Use a soft update strategy to slowly update the parameters of the target network: ;

[0096] Step 4.3.8: Update the exploration probability ,Right now ;

[0097] After the algorithm is trained, it outputs the optimized online Q-network. The optimal control strategy is determined by the network: in each state Choose the action that maximizes the Q value, i.e. .

[0098] The beneficial effects of this invention are:

[0099] This invention proposes a flexible control strategy that integrates a model-free method with a cross-layer bidirectional coupling mechanism, effectively addressing the problems of strong model dependence, strategy infeasibility, and insufficient stability against packet loss faced by networked handover systems under denial-of-service attacks. This method significantly improves the system's anti-attack capability, mean square exponential stability, and dynamic control performance. In practical engineering scenarios with unknown system matrices and random transmission environments, the collaborative design of constrained zero-sum games and improved deep Q-networks ensures both the physical feasibility of attack and defense strategies and the autonomous learning of optimal handover signals, effectively suppressing the risks of open-loop operation and mode mismatch caused by denial-of-service attacks. This method enables networked handover systems to balance stability and control accuracy under complex attack and random transmission environments, significantly reducing the probability of system failure due to attacks or modeling errors. The proposed method exhibits multiple technical advantages in practical applications. Through a model-free adaptive control mechanism, the system can approximate the controller gain online even in the absence of precise model information, effectively adapting to the dynamic requirements of scenarios with model uncertainty, such as automotive transmission and robot control. Furthermore, by incorporating the solvability constraints of the physical layer algebraic Riccati equations into the network layer game framework, the physical realizability of the defense strategy is enhanced, thereby specifically mitigating data packet loss and handover synchronization issues caused by denial-of-service attacks. Simultaneously, this method transforms the handover signal selection problem into a Markov decision process in a chaotic state space, leveraging an improved reinforcement learning algorithm to achieve autonomous learning and optimization of the handover strategy in a stochastic environment, significantly improving the system's resilient control capability under severe attack conditions. This method provides an efficient and feasible technical path for the resilient control of networked handover systems in complex industrial environments. Attached Figure Description

[0100] Figure 1 This is a diagram of the data transmission structure of the networked handover system under a denial-of-service attack according to the present invention.

[0101] Figure 2 This is a schematic diagram of the framework of the cross-layer elastic control method for the networked switching system of the present invention.

[0102] Figure 3 This is a schematic diagram of the constrained Nash equilibrium solution under different critical packet delivery rates of the present invention.

[0103] Figure 4 This is a graph showing the iterative results of the reward value of the present invention as the number of training rounds increases.

[0104] Figure 5 This is a diagram showing the system dynamic response under the optimal switching strategy of this invention. Detailed Implementation

[0105] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be described in detail below. Obviously, the described embodiments are merely some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other implementation methods obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0106] See Figure 1 A discrete-time networked switched linear system subjected to a denial-of-service (DoS) attack connects to the controller via a wireless channel vulnerable to DoS attacks. System state and handover signals are transmitted via independent data packets, with packet loss occurring independently and simultaneously. To conserve network resources, system state is continuously transmitted, while the handover signal is sent only when a handover occurs. If the system state data packet is lost due to an attack, the control input is reset to zero, and the system enters open-loop operation. If the handover signal is lost, it will cause the running subsystems to lose synchronization with the controller; this state will persist until subsequent signals are successfully transmitted.

[0107] See Figure 2 The workflow of this cross-layer architecture is as follows: The network layer solves a constrained zero-sum game to output the optimal packet delivery rate strategy, thereby controlling the open-loop operation and modal asynchrony risks of the physical layer. Simultaneously, the physical layer feeds back the solvability of the algebraic Riccati equation as a fundamental constraint to the network layer, ensuring the realizability of the game equilibrium solution within the physical system. This bidirectional coupling mechanism is the core of achieving cross-layer resilient control, guaranteeing that the defense strategy is both theoretically optimal and engineering-feasible.

[0108] See Figure 3 Based on the obtained constrained Nash equilibrium strategy, this invention further reveals the analytical relationship between the equilibrium solution and the critical packet delivery rate for the solvability of the algebraic Riccadi equation. To obtain the closed-loop expression of the constrained Nash equilibrium, the optimal responses of the defender and attacker need to be calculated as follows, where the optimal response of the defender is as follows.

[0109] For the purpose of maximizing The defending side, in and Under the constraints, given Optimal reaction Determined as

[0110] (twenty three)

[0111] On the other hand, the attacker's optimal response is as follows.

[0112] For the purpose of maximizing The defending side, in and Under the constraints, given Optimal reaction Determined as

[0113] (twenty four)

[0114] in ,

[0115] ,

[0116] ,

[0117] ,

[0118] ,

[0119] .

[0120] See Figure 4 The improved deep Q-network algorithm proposed in this invention shows a convergent trend in reward values ​​as the number of training rounds increases, with the overall system reward steadily rising. In the early stages of training, the reward curve fluctuates significantly, reflecting the instability of the agent during environment exploration and strategy learning. As training progresses, the fluctuation amplitude decreases and stabilizes, indicating that the proposed improved deep Q-network algorithm gradually converges and the policy performance stabilizes. This result verifies that the proposed method can effectively improve the overall control performance and anti-interference capability of networked switching systems through autonomous learning under random transmission environments and denial-of-service attacks.

[0121] See Figure 5 The dynamic performance of the proposed control method in a networked switching system has been effectively verified. Under switching signal scheduling, the system smoothly transitions between state one and state two over time without significant chattering or instability. The control input curve responds rapidly and has a reasonable amplitude throughout the entire operating period, indicating that the controller can respond to system modal changes in a timely manner and generate feasible control commands. Each subsystem maintains stable operation during asynchronous switching, proving that the method can maintain reliable control performance even under denial-of-service attacks and has good engineering applicability.

Claims

1. A method for cross-layer resilience control of a networked switching system based on reinforcement learning, characterized in that, Comprising the following steps: Step 1: Constructing a closed-loop dynamic mathematical model of a discrete-time networked switched linear system, using Bernoulli distribution to describe the random packet loss phenomenon caused by denial-of-service attacks, designing a corresponding cost function for each subsystem, and establishing a mathematical relationship between the network layer signal-to-interference-and-noise ratio and the physical layer packet drop rate; Step 2: Constructing a constrained zero-sum game model in the network layer considering the solvability of the physical layer Riccati equation, including: Constrained zero-sum game participants: the defense and attack sides; Constrained zero-sum game actions: the power allocation strategy of both sides on the wireless channel; Constrained zero-sum game utility function: both sides consider optimizing the signal-to-interference-and-noise ratio and power consumption; Step 3: Reconstructing the cost function of each subsystem, estimating the upper bound of the critical packet drop rate, solving the constrained Nash equilibrium without a model, determining the optimal power allocation, and determining the packet drop rate of the wireless channel; Step 4: Based on the packet drop rate obtained in Step 3, design the dwell time condition to ensure system mean square stability, and based on the dwell time condition, model the switching signal optimization problem as a Markov decision process; use the improved deep Q network method to solve the strategy.

2. The cross-layer resilient control method for networked switching system based on reinforcement learning according to claim 1, wherein, The Step 1 comprises: Step 1.1, in the physical layer, constructing a discrete-time networked switched linear system, as shown in formula (1), (1) ； where is the system state, is the control input, switching signal is a piecewise constant function taking values in a finite index set where is the total number of subsystems; when denotes the subsystem is activated at time ; and denote the system matrix and the control matrix, respectively, of corresponding dimension, denotes the dimension of the system state vector, denotes the dimension of the control input vector; Step 1.2, the data packet loss caused by denial of service attack obeys Bernoulli distribution; To describe the system state and the transmission state of the switching signal, two indicator functions are introduced and ; and represent the data packet loss, and and represent successful transmission; The probability of successful acceptance is , where represents the packet delivery rate determined by the network layer; The controller adopts the form of state feedback , where is the mode of the controller at time ; Let the switching time sequence be , where is the total number of switching times, then the closed-loop dynamic mathematical model of the discrete-time networked switched linear system is expressed as (2) ； Step 1.3, under the condition that the closed-loop dynamic mathematical model of the discrete-time networked switched linear system is unknown, the gain and switching law of the controller are designed based on offline data to ensure the mean square stability of the discrete-time networked switched linear system; the stage cost of the first subsystem is defined as (3) ； wherein and denote the state weight matrix and the control weight matrix, respectively; The first subsystem defines the infinite-time-domain cost function at time t as (4) ； wherein denotes the mathematical expectation operator; the optimal value function can be expressed as wherein ; According to the Bellman equation, the algebraic Riccati equation of the subsystem is obtained ​ (5) ； Let us assume that for each subsystem there exists a corresponding state feedback gain matrix such that the closed loop matrix is Schur stable; Step 1.4, in the network layer, the defense party (intelligent sensor) and the attack party form a power allocation competition relationship on the wireless channel, and the direct power input of this competition determines the signal-to-noise ratio of the channel , and this ratio is a key indicator for determining the packet delivery rate and measuring the quality of wireless transmission; (6) ； Among them, the transmission power of the defending side The attacker's jamming power , and These are the power limits for both parties; and Channel attenuation coefficients for the defender and attacker, respectively. The additive white Gaussian noise power in the environment; in wireless communication systems employing quadrature amplitude modulation, the signal-to-interference-plus-noise ratio (SINR) is a key factor determining the packet delivery rate; according to communication theory, the packet delivery rate... The following can be calculated: (7) ； wherein is a given parameter, and ; denotes the lower integration limit, denotes an auxiliary variable.

3. The cross-layer resilient control method for the networked switching system based on reinforcement learning according to claim 2, wherein, The Step 2 comprises: Step 2.1, determining the members participating in the zero-sum game as the defense and attack sides, both of which make decisions simultaneously; Step 2.2, defender chooses Guarantee data transmission, attacker chooses Destroy data transmission; Step 2.3, determining the utility function of both sides, the defense side aims to maximize the packet drop rate and reduce energy consumption, and the attack side tries to minimize the packet drop rate and reduce its own energy consumption; accordingly, the utility function of the defense side is defined as (8) ； where, and are the unit power cost of the defender and the attacker, respectively, is the trade-off coefficient between SINR and energy consumption; the utility function of the attacker is defined as (9) ； Step 2.4: To analyze the impact of the physical layer on the zero-sum game of the network layer, it can be seen from equation (5) that when the first... When a subsystem is unstable, the solution to its algebraic Ricardi equation Depends on packet delivery rate : The equation has a unique solution. There is no feasible solution at this time; therefore, there exists a critical packet delivery rate. The equation is solvable if and only if ;use Signal-to-interference-plus-noise ratio The monotonic relationship can uniquely determine the critical signal-to-interference-plus-noise ratio. ,satisfy Therefore, this zero-sum game must satisfy the following constraints. (10) ； The defender aims to maximize , subject to while the attacker aims to maximize , subject to .

4. The cross-layer resilient control method for networked switching system based on reinforcement learning according to claim 3, wherein, The Step 3 comprises: Step 3.1, To realize model-free control, the cost function of each subsystem is reconstructed, and the cost function of the first subsystem is obtained as follows: ​​ (11) ； Wherein (12) ； No. The optimal controller gain of each subsystem can be expressed as: (13) ； The optimal Bellman equation can be expressed as: (14) ； Based on this, we can get (15) ； Step 3.2, adopting an iterative approach and Model-less solution method for unknown situations Package delivery rate This has a crucial impact on the solvability of the algebraic Ricardi equation, and provides a critical value for the packet delivery rate. An upper bound estimate; due to the actual system matrix Unknown, temporarily disregarding the attack-defense game in the network layer, and setting a sufficiently high transmission power for the defender to ensure the solvability of the algebraic Ricardi equation, let the packet delivery rate at this point be... Under this configuration, by combining the iterative convergence results, we can obtain and steady-state solution and And then use the two to estimate The maximum eigenvalue; for networked handover systems, the critical packet delivery rate The following conditions must be met (16) ； And satisfy the following conditions (17) ； wherein then the Riccati equation has a unique positive definite solution ; In addition, since the signal-to-interference-and-noise ratio corresponds to the packet delivery rate one-to-one, when the network layer game equilibrium corresponds to the signal-to-interference-and-noise ratio greater than the upper limit corresponding to the signal-to-interference-and-noise ratio , the physical layer algebraic Riccati equation can also be solved. Step 3.3、In the network layer, the strategy of satisfying the signal-to-interference-and-noise ratio constraint is formulated; by analyzing the utility functions of both sides and introducing the signal-to-interference-and-noise ratio constraint, the optimal power allocation strategies of the defense side and the attack side are derived; in the constructed constrained zero-sum game, the constrained Nash equilibrium is as follows: Step 3.3、In the network layer, the strategy of satisfying the signal-to-interference-and-noise ratio constraint is formulated; by analyzing the utility functions of both sides and introducing the signal-to-interference-and-noise ratio constraint, the optimal power allocation strategies of the defense side and the attack side are derived; in the constructed constrained zero-sum game, the constrained Nash equilibrium is as follows: Step 3.3、In the network layer, the strategy of satisfying the signal-to-interference-and-noise ratio constraint is formulated; by analyzing (18) ； wherein , ， ， ， ， ， ， ， ； The packet delivery rate of the wireless channel is calculated according to the Nash equilibrium and equation (7) .

5. The cross-layer resilient control method for the networked switching system based on reinforcement learning according to claim 4, wherein, The Step 4 comprises: Step 4.1, for the discrete-time networked switched linear system, when its network layer is at the constrained Nash equilibrium and the optimal controller gain designed by equation (13) is adopted, there exists a constant , , , satisfying the following equality relationship: (19) ； wherein , , if the residence time satisfies the following condition, then the discrete-time networked switched linear system is mean-square exponentially stable; (20) ； wherein, is the optimal packet delivery rate under network layer balancing policy; Step 4.2, the infinite switching sequence that satisfies the average dwell time condition is still vulnerable to packet loss caused by denial-of-service attacks, which may lead to performance degradation; considering the randomness of the transmission environment, the switching signal optimization problem is modeled as a Markov decision process, and the specific framework is as follows: Step 4.2.1, at time is defined as wherein, denotes the duration since the last switch occurred; Step 4.2.2, at time the action is defined as a switching signal ; Step 4.2.3, Reward: Reward function The stage cost is designed as follows: (21) ； wherein, and denote the state weight matrix and the control weight matrix under the subsystem modality denote the controller gain under the controller modality is the dwell time penalty term: if and then wherein, is a fixed penalty value if the dwell time condition is violated; otherwise ; the objective is to maximize the expected cumulative reward: wherein, is the discount factor;​​ Step 4.2.4, State transition: next state Evolve as follows: (22) ； Step 4.3, the following improved deep Q-network algorithm is proposed to solve the above Markov decision problem, wherein the input is: Q-network ; target network ; experience replay buffer ; temporary state transition buffer ; initial exploration probability ; decay factor ; minimum exploration rate ; maximum number of training rounds ; target network update rate: ; simulation time domain ; Step 4.3.

1. Randomly initialize the online Q-network and the target Q-network parameters, repeat the following training scenario for 1000 times; Step 4.3.2, initialize environment state ; Step 4.3.3, for time step to , perform action selection, state update, reward computation and storage transition, store the state transition experience into the temporary state transition buffer ; Step 4.3.

4. For each time step to , compute the cumulative discounted return at each time step Step 4.3.5, all complete experience tuples in this scenario are stored in the experience replay buffer into the experience replay buffer ; Step 4.3.6, network training, including experience replay, calculating target values, calculating losses, and gradient descent updates; Step 4.3.7, slowly update the parameters of the target network using a soft update strategy: ; Step 4.3.8, updating the exploration probability i.e. ; After the algorithm training is completed, an optimized online Q network is output ; the optimal control strategy is determined by the network: at each state , the action that maximizes the Q value is selected, i.e. .

Images