Optimal redundancy transmission scheduling method and system for industrial control system

Abstract

The invention provides an industrial control system optimal redundancy transmission scheduling method and system, and the method comprises the steps: collecting system parameters and channel parameters, configuring network parameters, and initializing a system environment and a neural network; mDP modeling is carried out based on the established redundancy transmission scheduling model, the state space of the MDP model is initialized, the training step is set to be 0, and training of the first Episode round and the first step is started; a competitive double-depth Q network D3QN is adopted, and based on an MDP model, an optimal action is output, the action is executed, a remote estimator executes channel allocation, a system begins to transmit data, an expected estimation error covariance is calculated after transmission is completed, a reward value is obtained, a next state is obtained after a packet loss condition is observed, and complete state transition is obtained; the weight is updated by using a gradient descent algorithm, and the precision of the neural network is gradually increased along with the continuous updating of the weight; the average cost is output after each round of training is completed, the average cost is gradually reduced and finally converged, and the optimal redundancy transmission scheduling strategy is obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of industrial control system transmission scheduling, and in particular relates to an industrial control system optimal redundant transmission scheduling method and system. Background Art

[0002] Industrial Control System (ICS) effectively improves the efficiency of industrial production, and its safe and stable operation is the basis for ensuring stable industrial production. With the rapid integration of information technology and industrial automation, ICS began to adopt open communication protocols for digitization and use wireless network channels for information transmission. However, data loss under unreliable channels seriously affects the performance of remote state estimation of the system. How to improve the quality of the channel and the estimation performance of the system has become one of the key issues in ICS research. To this end, a redundant transmission strategy is usually adopted to send redundant information to recover when data loss occurs. At the same time, redundant transmission inevitably leads to a waste of network resources. It is necessary to study how to intelligently allocate and schedule these redundant channels to maximize the accuracy of state estimation. In response to this type of problem, an optimal redundant transmission scheduling algorithm for industrial control systems based on reinforcement learning is designed. The purpose is to ensure the system state estimation performance as much as possible while utilizing limited redundant resources.

[0003] Industrial control systems are widely used in the control systems of industrial fields and important infrastructure, including monitoring and data acquisition systems, distributed control systems and other control systems. It integrates embedded technology, multi-standard industrial control network interconnection, wireless technology, etc., and has opened up many emerging cross-disciplinary cooperation fields. It has been widely used in robot control, intelligent traffic optimization management and environmental monitoring. However, precisely because of its large number of applications, the instability of wireless communication channels will cause an impact that cannot be ignored. External noise interference, power limitations and potential security threats may lead to unreliable data transmission, and the loss of key information will cause serious damage. Therefore, how to improve the performance and reliability of remote state estimation in ICS has become a research hotspot.

[0004] Nowadays, people's requirements for system performance are constantly increasing, especially when resources are abundant. At this time, it is necessary to rationally utilize more resources to further improve the performance of remote estimation, and redundant transmission has become an effective and necessary strategy. Redundant transmission provides a recovery mechanism for possible data loss by sending additional redundant information. We use the redundant transmission method, so that the system can use this redundant information to reconstruct the lost data even in poor channel conditions or interference, thereby ensuring the accuracy of remote state estimation and the continuous operation of the system.

[0005] Although the use of technologies such as redundant channels can significantly improve transmission performance, it inevitably leads to waste of network resources such as sensor power consumption, signal interference, bandwidth waste, etc. Summary of the invention

[0006] The object of the present invention is to provide an optimal redundant transmission scheduling method and system for an industrial control system to solve the above-mentioned problems.

[0007] To achieve the above object, the present invention adopts the following technical solutions:

[0008] In a first aspect, the present invention provides an optimal redundant transmission scheduling method for an industrial control system, comprising:

[0009] Collect system parameters, channel parameters and network parameters, initialize the system environment and the neural network;

[0010] Based on the established redundant transmission scheduling model, MDP modeling is performed, the MDP model state space is initialized, the training step steps is set to 0, and the training of the episode round and the steps step is started;

[0011] Based on the MDP model, the optimal action is output and executed. The remote estimator performs channel allocation, and the system starts to transmit data. After the transmission is completed, the expected estimation error covariance is calculated to obtain the reward value, and the packet loss situation is observed to obtain the next state, thus obtaining a complete state transition.

[0012] The competitive dual deep Q network D3QN is adopted, and the weights are updated using the gradient descent algorithm. As the weights are continuously updated, the accuracy of the neural network will gradually increase, and the optimal redundant transmission scheduling strategy will be obtained.

[0013] Furthermore, the system parameters, channel parameters and network parameters are collected to initialize the system environment and the neural network, including:

[0014] Input the number of systems N, the number of channels M, their respective system matrices, noise matrix system parameters, and failure rate p, recovery rate q channel parameters to complete the initial setting of the system environment; input the learning rate alpha, exploration rate ∈ network parameters to initialize the neural network, initialize the experience replay buffer and its maximum capacity, set the number of training rounds episode to 0, and complete the initialization of the algorithm.

[0015] Furthermore, MDP modeling is performed based on the established redundant transmission scheduling model, the MDP model state space is initialized, the training step steps is set to 0, and the training of the episode round and the steps step is started, including:

[0016] Based on the established redundant transmission scheduling model, a four-tuple is constructed Perform MDP modeling;

[0017] State space: Define the state space as Reduce the MDP to the number of consecutive packet losses of each sensor at time k-1, that is And the remote estimator can receive γ i,k i∈{1,...,N} calculates the state variables by iteration;

[0018] Action space: Define the action space as

[0019] State transition probability: Based on the modeling of channel packet loss:

[0020]

[0021] and

[0022]

[0023] Cost function: Based on the state and action, you can get the immediate cost and then transfer to the next state; first-order cost function The definition is as follows:

[0024]

[0025] Indicates state s k Take action a k the immediate cost of acquisition;

[0026] The goal of building an MDP model is to minimize the expected average cost over an infinite time horizon:

[0027]

[0028] Furthermore, based on the optimal action output by the MDP model, the action is executed, the remote estimator performs channel allocation, the system starts to transmit data, and after the transmission is completed, the expected estimation error covariance is calculated, including:

[0029] The remote state estimator calculates the estimated value based on whether it receives the data packet sent by the sensor. The estimation process is described as follows: If the sensor data is successfully received by the estimator, it will be updated synchronously; otherwise, it will be updated based on the previous estimate and the system model, that is:

[0030]

[0031] Among them, the prediction error covariance function h i (·), defined as

[0032] A Markov decision process is modeled to solve this problem:

[0033]

[0034] in, is the strategy of taking a certain action in a certain state, Π is the set of strategies that can be taken, and Tr{·} represents the trace of the matrix; for each system, the calculation formula of the expected estimation error covariance is

[0035]

[0036] Before the sensor transmits data, the scheduler needs to know the estimated error covariance corresponding to each system to complete the channel allocation according to the estimated error size of each sensor; the channel allocation at time k depends on (P 1,k-1 ,...,P N,k-1 );P i,k Always by function It is calculated that the power n is the number of consecutive packet losses for each system; let τ i,k represents the dead time of each system, that is, the number of consecutive transmission failures at time k:

[0037]

[0038] τ i,k With γ i,k The previous relationship was as follows:

[0039]

[0040] Then the error covariance P i,k By τ i,k The calculation results are:

[0041]

[0042] Therefore, the channel allocation at time k will depend on (τ 1,k-1 ,...,τ N,k-1 ).

[0043] Furthermore, a competitive dual-depth Q network D3QN is used to update weights using the gradient descent algorithm. As the weights are continuously updated, the accuracy of the neural network will gradually increase, and the optimal redundant transmission scheduling strategy will be obtained, including:

[0044] The competitive dual deep Q network D3QN algorithm is used to obtain the approximately optimal redundant transmission scheduling strategy; the algorithm is used to minimize the estimation error covariance of multiple systems;

[0045] Using a discount factor δ close to 1 to approximate the average cost problem, the discounted average reward maximization problem is equivalent to the average cost problem:

[0046]

[0047] State-Action Value Function represents the expected future reward of taking action a in state s; optimal value function The following Bellman optimal equation is satisfied:

[0048]

[0049] Among them, the reward function s′ is the next state under the condition of a given state-action pair (s, a); if the optimal function Get an optimal deterministic and stable strategy, the optimal action a for each state s * as follows:

[0050]

[0051] Use D3QN algorithm to obtain the optimal The function uses two independent neural networks - the target network and the training network to train together, one for selecting actions and the other for evaluating the value of the selected actions. At the same time, the weights are updated using the gradient descent algorithm.

[0052] Furthermore, the gradient descent algorithm is used to update the weights, and the corresponding loss function is:

[0053]

[0054] Among them, y D3QN It is calculated by the following formula:

[0055]

[0056] An experience replay mechanism is adopted, which stores past state transitions in the experience replay buffer and randomly samples them during training for model updating.

[0057] Furthermore, the state is transferred Store into experience replay buffer And accumulate the total cost to evaluate the training effect;

[0058] According to the calculation of y D3QN The value of y D3QNThe square loss function gradient descent updates the network weights. As the weights are continuously updated, the accuracy of the neural network will gradually increase for more accurate scheduling. At this time, steps = steps + 1. After every 50 training steps, the parameter values ​​of the training network are synchronized to the target network, and the average cost of the round is calculated to generate a training curve. If the training is not over, episode = epidode + 1, and the next round of training begins after decreasing the exploration rate; otherwise, the training ends.

[0059] In a second aspect, the present invention provides an optimal redundant transmission scheduling system for an industrial control system, comprising:

[0060] The data acquisition module is used to collect system parameters, channel parameters and network parameters, initialize the system environment and the neural network;

[0061] The MDP modeling module is used to perform MDP modeling based on the established redundant transmission scheduling model, initialize the MDP model state space, set the training step steps to 0, and start the training of the episode round and the steps step;

[0062] The expected estimation error covariance calculation module is used to output the optimal action based on the MDP model, execute the action, and the remote estimator performs channel allocation. The system starts to transmit data. After the transmission is completed, the expected estimation error covariance is calculated to obtain the reward value, and the packet loss situation is observed to obtain the next state, thereby obtaining a complete state transfer;

[0063] The output module is used to adopt the competitive dual deep Q network D3QN and update the weights using the gradient descent algorithm. As the weights are continuously updated, the accuracy of the neural network will gradually increase, and the optimal redundant transmission scheduling strategy will be obtained.

[0064] In a third aspect, the present invention provides a computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of an optimal redundant transmission scheduling method for an industrial control system when executing the computer program.

[0065] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the optimal redundant transmission scheduling method for an industrial control system are implemented.

[0066] Compared with the prior art, the present invention has the following technical effects:

[0067] The present invention proposes an optimal redundant transmission scheduling method for an industrial control system based on reinforcement learning, aiming to solve the problem of how to intelligently allocate and schedule redundant channels under unreliable channel conditions to maximize the accuracy of system state estimation.

[0068] By adopting a redundant transmission strategy and optimizing scheduling under the reinforcement learning framework, this method can effectively reduce the impact of data loss caused by channel unreliability on system state estimation. The transmission of redundant information makes data recovery possible, thereby improving the accuracy of remote state estimation.

[0069] This method avoids unnecessary resource waste by intelligently allocating and scheduling redundant channels. Traditional redundant transmission strategies may lead to excessive consumption of network resources, while this method can minimize resource usage, such as sensor power consumption, signal interference, and bandwidth occupancy, while ensuring state estimation performance.

[0070] In the face of channel instability caused by external noise interference, power limitations, and potential security threats, this method continuously adapts and optimizes the scheduling strategy through reinforcement learning to enhance the robustness of the system. This enables the system to maintain stable operation in various complex and uncertain environments.

[0071] By optimizing redundant transmission scheduling, this method not only improves the accuracy of state estimation, but also indirectly improves the overall performance of industrial control systems. Accurate remote state estimation is the basis for effective control and optimization. Therefore, this method helps to improve the efficiency and quality of industrial production.

[0072] This method is based on a reinforcement learning framework and has strong scalability and adaptability. As the system scale and complexity increase, or channel conditions change, this method can continuously learn and adapt, adjust and optimize the scheduling strategy to meet the needs of different scenarios.

[0073] By intelligently allocating and scheduling redundant channels, this approach helps reduce system maintenance and repair costs caused by unreliable data transmission. Accurate remote status estimation can detect and predict potential problems in a timely manner, so that measures can be taken in advance for prevention and repair.

[0074] In summary, this technical solution realizes the intelligent management of optimal redundant transmission scheduling of industrial control systems by combining redundant transmission strategy and reinforcement learning algorithm. This not only improves the accuracy of state estimation and the overall performance of the system, but also optimizes resource utilization, enhances system robustness, and reduces maintenance costs. Therefore, this technical solution has broad application prospects and important practical significance in the field of industrial control systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] Figure 1 Remote state estimation model for industrial control systems.

[0076] Figure 2 Diagram of the Gilbert-Elliott model.

[0077] Figure 3 Comparison of average costs under different strategies.

[0078] Figure 4 Smoothed D3QN reward value changes.

[0079] Figure 5 Smoothed DQN reward value changes

[0080] Figure 6 Graph of the optimal redundant transmission scheduling algorithm for industrial control systems based on reinforcement learning.

[0081] Figure 7 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0082] The present invention is further described below in conjunction with the accompanying drawings:

[0083] Example 1, please refer to Figure 7 The present invention provides an optimal redundant transmission scheduling method for an industrial control system, comprising:

[0084] Collect system parameters, channel parameters and network parameters, initialize the system environment and the neural network;

[0085] Based on the established redundant transmission scheduling model, MDP modeling is performed, the MDP model state space is initialized, the training step steps is set to 0, and the training of the episode round and the steps step is started;

[0086] Based on the MDP model, the optimal action is output and executed. The remote estimator performs channel allocation, and the system starts to transmit data. After the transmission is completed, the expected estimation error covariance is calculated to obtain the reward value, and the packet loss situation is observed to obtain the next state, thus obtaining a complete state transition.

[0087] The competitive dual deep Q network D3QN is adopted, and the weights are updated using the gradient descent algorithm. As the weights are continuously updated, the accuracy of the neural network will gradually increase, and the optimal redundant transmission scheduling strategy will be obtained.

[0088] Embodiment 2, the present invention provides an optimal redundant transmission scheduling method for an industrial control system, specifically comprising:

[0089] Consider Figure 1 The remote state estimation model of the industrial control system shown in Figure 1 is shown in Figure 2. i,kis the state vector of system i at time k, and each sensor i runs a Kalman filter to calculate the local state estimate is the remote state estimator's estimate of system i at time k.

[0090] Consider N discrete linear time-invariant systems, observed by N sensors. The system dynamics equation is

[0091]

[0092] y i,k is the measured value of system i at time k, ω i,k and i,k They are Gaussian distributions The process noise and the Gaussian distribution The measurement noise. i is the system matrix, C i is the measurement matrix. For each system i, the initial state x i,0 Obey Gaussian distribution P i,0 , Q i is a positive definite matrix, P i,0 is the estimated error covariance matrix.

[0093] Assume that the number of wireless channels is M (M>N), and they are independent of each other. Data loss may occur during wireless transmission due to noise interference, routing problems, network attacks, etc. We can use the Bernoulli random variable ξ m,k =1 means that the data of channel m is successfully transmitted to the remote state estimator; ξ m,k =0 means that data transmission on channel m fails, i.e. packet loss occurs.

[0094] Before transmitting data, M wireless channels need to be allocated to N sensors. Each channel is modeled as a Gilbert-Elliott model:

[0095]

[0096] p and q are failure rate and recovery rate respectively. The schematic diagram of Gilbert-Elliott model is as follows: Figure 2-3 shown.

[0097] Definition i,k is the number of channels that successfully transmit at time among the channels assigned to the sensor. In addition, a Bernoulli random variable is used to indicate that the sensor data is successfully transmitted to the remote state estimator; it indicates that the data transmission fails, that is, packet loss occurs. According to the Gilbert-Elliott model, the probability of packet loss of the sensor at time is:

[0098]

[0099] The remote state estimator calculates the estimated value based on whether it receives the data packet sent by the sensor. The estimation process is described as follows: If the sensor data is successfully received by the estimator, it will be updated synchronously; otherwise, it will be updated based on the previous estimate and the system model, that is:

[0100]

[0101] Among them, the prediction error covariance function h i (·), defined as

[0102] As data packets continue to be lost, the error covariance P i,k will continue to increase. We hope to find a scheduling strategy that can allocate channels in a multi-system multi-sensor environment to minimize the trace of the expected average estimation error covariance. Therefore, we will perform Markov decision process modeling to solve this problem:

[0103]

[0104] in, is the strategy of taking a certain action in a certain state, Π is the set of strategies that can be taken, and Tr{·} represents the trace of the matrix. For each system, the calculation formula for the expected estimation error covariance is

[0105]

[0106] Before the sensor transmits data, the scheduler needs to know the estimated error covariance corresponding to each system in order to complete the channel allocation according to the estimated error size of each sensor. In other words, the channel allocation at time k depends on (P 1,k-1 ,...,P N,k-1 ). It can be observed from formula (4) that P k,k Always by function It is calculated that the power n is the number of consecutive packet losses in each system. Let τ k,k represents the dead time of each system, that is, the number of consecutive transmission failures at time k:

[0107]

[0108] τ i,k With γ i,k The previous relationship was as follows:

[0109]

[0110] Then the error covariance P i,kIt can also be expressed by τ k,k The calculation results are:

[0111]

[0112] Therefore, the channel allocation at time k will depend on (τ 1,k-1 ,...,τ N,k-1 ). Since τ i,k is a scalar, which can realize the matrix P i,k Next, we model the problem of minimizing the expected mean estimation error covariance (5) as a Markov decision process (MDP). Based on the established redundant transmission scheduling model, we can construct a four-tuple Perform MDP modeling.

[0113] State space: Define the state space as According to the previous description, the MDP problem can be reduced to the number of consecutive packet losses of each sensor at time k-1, that is, And the remote estimator can receive γ i,k i∈{1,...,N} calculates the state variables through the iteration formula (3-2).

[0114] Action space: Define the action space as c i,k Satisfies conditional expression (2-5).

[0115] State transition probability: Based on the modeling of channel packet loss, we can get:

[0116]

[0117] and

[0118]

[0119] Cost function: Based on the state and action, you can get the immediate cost and then transfer to the next state. First-order cost function The definition is as follows:

[0120]

[0121] Indicates state s k Take action a k Immediate cost obtained.

[0122] Therefore, the goal of constructing an MDP model is to minimize the expected average cost over an infinite time horizon:

[0123]

[0124] We use the Dueling doubledeep Q-network (D3QN) algorithm to obtain the approximately optimal redundant transmission scheduling strategy. This algorithm combines double Q-lerning and dueling network, and has obvious advantages in reducing overestimation, convergence stability, and convergence efficiency. We will use this algorithm to flexibly and effectively solve the MDP problem of optimal redundant transmission scheduling to minimize the estimation error covariance of multiple systems.

[0125] We use a discount factor δ close to 1 to approximately solve the average cost problem (see Equation 12). Therefore, consider the discounted average reward maximization problem equivalent to Equation (12):

[0126]

[0127] State-Action Value Function represents the expected future reward of taking action a in state s. The optimal value function The following Bellman optimal equation is satisfied:

[0128]

[0129] Among them, the reward function s′ is the next state under the condition of a given state-action pair (s, a). If the optimal function Then we can get an optimal deterministic and stable strategy, the optimal action a for each state s * as follows:

[0130]

[0131] We use the D3QN algorithm to obtain the optimal Function. Considering the continuous accumulation of positive errors during training, two independent neural networks, the target network and the training network (evaluation network), are used for joint training, one for selecting actions and the other for evaluating the value of the selected actions. At the same time, the weights are updated using the gradient descent algorithm, and the corresponding loss function is:

[0132]

[0133] Among them, y D33N It is calculated by the following formula:

[0134]

[0135] In addition, to improve learning efficiency, we adopt an experience replay mechanism, which stores past state transitions in an experience replay buffer and randomly samples them for model updates during training. This helps the network learn from past experience and reduces variance during training.

[0136] The D3QN algorithm in reinforcement learning is introduced to solve the optimal redundant transmission scheduling MDP problem under remote state estimation. We consider the case of N = 2 sensors and M = 3 channels, so the state space size is 2, and the process parameter A i , C i , Q i , R i The settings are as follows:

[0137]

[0138] c1=c2=[1 1],

[0139] R1=R2=[1].

[0140] The failure rate p and recovery rate q of the channel are set to 0.2 and 0.6 respectively, and the action space The size is 20.

[0141] For the D3QN network, the input is a combination of the number of consecutive packet losses of N systems, so the input layer has N = 2 nodes. In the network architecture, two hidden layers are set, each containing 128 nodes. The discount factor is set to δ = 0.98, and the learning rate alpha = 0.001. The exploration rate ∈ is set to range from 1 to 0.01, and is set to 5e -4 This setting ensures that the algorithm can fully explore the state space in the early stage to fully understand the environment, and gradually converge to the optimal strategy as the training progresses. Experience replay buffer Maximum capacity The batch size for each sampling is 64.

[0142] We will train for 500 rounds, with the number of steps per round set to 50. To better observe the quality of training in each round, the process environment will be reset after 50 training steps. In addition, we recorded and analyzed the average cost under different strategies. By comparing the performance of different strategies under the same parameter settings, we can highlight the effectiveness of the D3QN algorithm in learning the optimal redundant transmission scheduling strategy. The average cost under different strategies is shown in Figure 2. Figure 3 shown.

[0143] After about 175 rounds, the average cost converges to about 1.29. We compare this convergence effect with the following algorithms:

[0144] (1) Periodic strategy: From the action space Select actions sequentially in .

[0145] (2) Greedy holding times: For each time step k, select a holding time τ i,k-1 The largest sensor performs redundant transmission, that is, a redundant channel is allocated to the sensor.

[0146] (3) Greedy error covariance strategy: For each time step k, select Tr{P i,k}The largest sensor performs redundant transmission, that is, an extra channel is allocated to the sensor.

[0147] (4) DQN algorithm: Using the same hyperparameters, the training results are after 500 rounds.

[0148] The average costs of the periodic strategy, the greedy strategy for dead time, and the greedy strategy for error covariance are approximately 2.07, 1.48, and 1.43, respectively.

[0149] from Figure 3 It can be observed that at the beginning, the performance of the D3QN algorithm is roughly the same as the periodic strategy. This is because in the initial stage of training, the agent will randomly select actions with a very high probability, and the average cost of randomly selecting actions is often very high. As the algorithm continues to run, the agent gradually becomes familiar with the environment and will choose the optimal action with a greater probability. After about 120-130 rounds, the performance of the proposed D3QN algorithm exceeds the above strategy and has been better than the above strategy in subsequent rounds. Compared with the DQN algorithm, the convergence speed of the D3QN algorithm is slightly faster than that of the DQN algorithm, and after 200 rounds, the convergence effect of the DQN algorithm fluctuates significantly, while the convergence effect of the D3QN algorithm is significantly more stable. In order to more intuitively observe the convergence effects of the two deep learning algorithms, the comparison of the expected returns after sliding window smoothing is shown in the figure below. Figure 4 , Figure 5 shown.

[0150] from Figure 4 , Figure 5It can be seen that although the maximum reward values ​​of the two are basically the same, the D3QN algorithm shows better stability during the training process, the reward value obtained fluctuates less, the reward is more stable, and the maximum reward value is obtained slightly faster than the DQN algorithm. After 200 rounds, the reward value obtained by the D3QN algorithm is basically higher than that of the DQN algorithm, which shows that the D3QN algorithm has significantly improved the convergence stability and convergence speed. As the action space increases, the advantage of D3QN will become more obvious.

[0151] Embodiment 3, Figure 6 A flowchart of an optimal redundant transmission scheduling algorithm for an industrial control system based on reinforcement learning is given. The method is implemented by the following steps:

[0152] Step 1: Input the number of systems N, the number of channels M, their respective system matrices, noise matrices and other system parameters, as well as channel parameters such as failure rate p and recovery rate q, to ​​complete the initial setting of the system environment; input network parameters such as learning rate alpha and exploration rate ∈ to initialize the neural network, initialize the experience playback buffer and its maximum capacity, set the number of training rounds episode to 0, and complete the initialization of the algorithm;

[0153] Step 2: Initialize the MDP model state space s0, set the training step steps to 0, and start the training of the episode round and the steps step;

[0154] Step 3: From the action space generated from N and M In the example, an action a is randomly selected with probability of exploration rate ∈ t ; On the contrary, according to formula (15), the optimal action a is selected with a probability of 1-∈ t ;

[0155] Step 4: Perform action a t , the remote estimator (or scheduler) performs channel allocation, and the system starts to transmit data; after the transmission is completed, the expected estimation error covariance is calculated, and the reward value is obtained And observe the packet loss to get the next state s t+1 , so as to obtain a complete state transfer;

[0156] Step 5: Transfer the state Store into experience replay buffer And accumulate the total cost to evaluate the training effect;

[0157] Step 6: Calculate y according to formula (17) D3QN The value of y D3QNThe squared loss function (see formula 16) updates the network weights by gradient descent. As the weights are continuously updated, the accuracy of the neural network will gradually increase, allowing for more accurate scheduling. At this point, let steps = steps + 1, and repeat step 3 to start the next step of training;

[0158] Step 7: After every 50 training steps, synchronize the parameter values ​​of the training network to the target network, calculate the average cost of this round, and generate a training curve. If the training is not finished, then episode = epidode + 1, after decreasing the exploration rate, return to step 2 and start the next round of training; otherwise, the training ends.

[0159] In yet another embodiment of the present invention, an optimal redundant transmission scheduling system for an industrial control system is provided, which can be used to implement the above-mentioned optimal redundant transmission scheduling method for an industrial control system. Specifically, the system includes:

[0160] The data acquisition module is used to collect system parameters, channel parameters and network parameters, initialize the system environment and the neural network;

[0161] The MDP modeling module is used to perform MDP modeling based on the established redundant transmission scheduling model, initialize the MDP model state space, set the training step steps to 0, and start the training of the episode round and the steps step;

[0162] The expected estimation error covariance calculation module is used to output the optimal action based on the MDP model, execute the action, and the remote estimator performs channel allocation. The system starts to transmit data. After the transmission is completed, the expected estimation error covariance is calculated to obtain the reward value, and the packet loss situation is observed to obtain the next state, thereby obtaining a complete state transfer;

[0163] The output module is used to adopt the competitive dual deep Q network D3QN and update the weights using the gradient descent algorithm. As the weights are continuously updated, the accuracy of the neural network will gradually increase, and the optimal redundant transmission scheduling strategy will be obtained.

[0164] The division of modules in the embodiments of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present invention may be integrated into one processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0165] In another embodiment of the present invention, a computer device is provided, the computer device including a processor and a memory, the memory is used to store a computer program, the computer program includes program instructions, and the processor is used to execute the program instructions stored in the computer storage medium. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc., which are the computing core and control core of the terminal, which are suitable for implementing one or more instructions, and are specifically suitable for loading and executing one or more instructions in the computer storage medium to implement the corresponding method flow or corresponding function; the processor described in the embodiment of the present invention can be used for the operation of an optimal redundant transmission scheduling method for an industrial control system.

[0166] In another embodiment of the present invention, the present invention further provides a storage medium, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space, and these instructions can be one or more computer programs (including program codes). It should be noted that the computer-readable storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk memory. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the optimal redundant transmission scheduling method for an industrial control system in the above embodiment.

[0167] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0168] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0169] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0170] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0171] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. An optimal redundant transmission scheduling method for an industrial control system, characterized in that: include: Collect system parameters, channel parameters and configure network parameters, initialize the system environment and the neural network; Based on the established redundant transmission scheduling model, MDP modeling is performed, the MDP model state space is initialized, the training step steps is set to 0, and the training of the episode round and the steps step is started; The competitive dual-depth Q network D3QN is used, and the optimal action is output based on the MDP model. The action is executed, and the remote estimator performs channel allocation. The system starts to transmit data. After the transmission is completed, the expected estimation error covariance is calculated to obtain the reward value, and the packet loss situation is observed to obtain the next state, and a complete state transfer is obtained; The weights are updated using the gradient descent algorithm. As the weights are continuously updated, the accuracy of the neural network will gradually increase, and the optimal redundant transmission scheduling strategy will be obtained.

2. According to the optimal redundant transmission scheduling method of an industrial control system described in claim 1, it is characterized in that: Collect system parameters, channel parameters and network parameters, initialize the system environment and the neural network, including: Input the number of systems N, the number of channels M, their respective system matrices, noise matrix system parameters, and failure rate p, recovery rate q channel parameters to complete the initial setting of the system environment; input the learning rate alpha, exploration rate ∈ network parameters to initialize the neural network, initialize the experience replay buffer and its maximum capacity, set the number of training rounds episode to 0, and complete the initialization of the algorithm.

3. According to the optimal redundant transmission scheduling method of an industrial control system described in claim 1, it is characterized in that: Based on the established redundant transmission scheduling model, MDP modeling is performed, the MDP model state space is initialized, the training step steps is set to 0, and the training of the episode round and the steps step is started, including: Based on the established redundant transmission scheduling model, a four-tuple is constructed Perform MDP modeling; State space: Define the state space as Reduce the MDP to the number of consecutive packet losses of each sensor at time k-1, that is And the remote estimator can receive γ i,k i∈{1,...,N} calculates the state variables by iteration; Action space: Define the action space as State transition probability: Based on the modeling of channel packet loss: and Cost function: Based on the state and action, you can get the immediate cost and then transfer to the next state; first-order cost function The definition is as follows: Indicates state s k Take action a k the immediate cost of acquisition; The goal of building an MDP model is to minimize the expected average cost over an infinite time horizon:

4. According to the optimal redundant transmission scheduling method of an industrial control system described in claim 1, it is characterized in that: Based on the MDP model output, the optimal action is executed, the remote estimator performs channel allocation, the system starts to transmit data, and after the transmission is completed, the expected estimation error covariance is calculated, including: The remote state estimator calculates the estimated value based on whether it receives the data packet sent by the sensor. The estimation process is described as follows: If the sensor data is successfully received by the estimator, it will be updated synchronously; otherwise, it will be updated based on the previous estimate and the system model, that is: Among them, the prediction error covariance function h i (·), defined as A Markov decision process is modeled to solve this problem: in, is the strategy of taking a certain action in a certain state, Π is the set of strategies that can be taken, and Tr{·} represents the trace of the matrix; for each system, the calculation formula of the expected estimation error covariance is Before the sensor transmits data, the scheduler needs to know the estimated error covariance corresponding to each system to complete the channel allocation according to the estimated error size of each sensor; the channel allocation at time k depends on (P 1,k-1 ,...,P N,k-1 );P i,k Always by function It is calculated that the power n is the number of consecutive packet losses for each system; let τ i,k represents the dead time of each system, that is, the number of consecutive transmission failures at time k: τ i,k With γ i,k The previous relationship was as follows: Then the error covariance P i,k By τ i,k The calculation results are: Therefore, the channel allocation at time k will depend on (τ 1,k-1 ,...,τ N,k-1 ).

5. According to the optimal redundant transmission scheduling method of an industrial control system described in claim 1 of the group, it is characterized in that: The competitive dual-depth Q network D3QN is used, and the weights are updated using the gradient descent algorithm. As the weights are continuously updated, the accuracy of the neural network will gradually increase, and the optimal redundant transmission scheduling strategy will be obtained, including: The competitive dual deep Q network D3QN algorithm is used to obtain the approximately optimal redundant transmission scheduling strategy; the algorithm is used to minimize the estimation error covariance of multiple systems; Using a discount factor δ close to 1 to approximate the average cost problem, the discounted average reward maximization problem is equivalent to the average cost problem: State-Action Value Function represents the expected future reward of taking action a in state s; optimal value function The following Bellman optimal equation is satisfied: Among them, the reward function s′ is the next state under the condition of a given state-action pair (s, a); if the optimal Q function can be calculated Get an optimal deterministic and stable strategy, the optimal action a for each state s * as follows: Use D3QN algorithm to obtain the optimal The function uses two independent neural networks - the target network and the training network to train together, one for selecting actions and the other for evaluating the value of the selected actions. At the same time, the weights are updated using the gradient descent algorithm.

6. According to the optimal redundant transmission scheduling method of an industrial control system described in claim 5 of the group, it is characterized in that: The weights are updated using the gradient descent algorithm, and the corresponding loss function is: Among them, y D3QN It is calculated by the following formula: An experience replay mechanism is adopted, which stores past state transitions in the experience replay buffer and randomly samples them during training for model updating.

7. According to the optimal redundant transmission scheduling method of an industrial control system described in claim 6 of the group, it is characterized in that: Transferring the state Store into experience replay buffer And accumulate the total cost to evaluate the training effect; According to the calculation of y D3QN The value of y Q3QN The squared loss function gradient descent updates the network weights. As the weights are continuously updated, the accuracy of the neural network will gradually increase, allowing for more accurate scheduling. At this time, steps = steps + 1. After every 50 training steps, the parameter values ​​of the training network are synchronized to the target network, and the average cost of the round is calculated to generate a training curve. If the training is not over, then episode = epidode + 1, and after decreasing the exploration rate, the next round of training begins; otherwise, the training ends.

8. An optimal redundant transmission scheduling system for an industrial control system, characterized in that: include: The data acquisition module is used to collect system parameters, channel parameters and network parameters, initialize the system environment and the neural network; The MDP modeling module is used to perform MDP modeling based on the established redundant transmission scheduling model, initialize the MDP model state space, set the training step steps to 0, and start the training of the episode round and the steps step; The expected estimation error covariance calculation module is used to output the optimal action based on the MDP model, execute the action, and the remote estimator performs channel allocation. The system starts to transmit data. After the transmission is completed, the expected estimation error covariance is calculated to obtain the reward value, and the packet loss situation is observed to obtain the next state, thereby obtaining a complete state transfer; The output module is used to adopt the competitive dual deep Q network D3QN and update the weights using the gradient descent algorithm. As the weights are continuously updated, the accuracy of the neural network will gradually increase, and the optimal redundant transmission scheduling strategy will be obtained.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the optimal redundant transmission scheduling method for an industrial control system as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of an optimal redundant transmission scheduling method for an industrial control system as described in any one of claims 1 to 7 are implemented.

Images