Perception and control performance driven intelligent transmission scheduling method for industrial network system

Abstract

The invention discloses a perception and control performance driven intelligent transmission scheduling method for an industrial network system, which relates to the field of industrial network systems and comprises the following steps of: 1, constructing an optimization problem and pre-learning a perception model; 2, transmission resources are reserved; 3, dynamic transmission scheduling based on reinforcement learning; and 4, perception and control performance evaluation. The method provided by the invention mainly meets the actual requirements of balancing system performance and limited transmission resources in complex industrial production, and the adopted intelligent transmission scheduling method is matched with the continuously expanded industrial production scale and potential unknown system parameters at present. According to the method, the transmission scheduling method can be conveniently and quickly deployed under the condition of lacking accurate prior information of a system model, various performances and resource weights are adjusted according to actual production requirements so as to achieve an ideal production effect, and the method can be effectively popularized to various different industrial application scenes.

Description

Technical Field

[0001] The present invention relates to the field of industrial network systems, and in particular to an intelligent transmission scheduling method for industrial network systems driven by perception and control performance. Background Art

[0002] With the continuous evolution of industry, a new generation of industrial network systems that integrate key processes such as ubiquitous perception, transmission, control and learning are enabling the development of industrial automation. Based on advanced edge computing and data transmission technologies, edge computing units can efficiently integrate multi-sensor information on site and send control instructions to actuators in a timely manner, thereby achieving the final production control goal. Better perception and control performance depends on more adequate data interaction, but this is at the expense of scarce resources such as bandwidth, time slots, and energy on industrial sites. Therefore, it is urgent to design a joint scheduling method for uplink (sensor to edge computing unit) and downlink (edge ​​computing unit to actuator) transmission to improve the overall performance of the system. In addition, the ever-expanding scale of industrial network systems and some unknown system models have also made intelligent scheduling methods based on reinforcement learning increasingly concerned. How to combine system performance indicators such as observability and controllability with the design of intelligent scheduling methods remains to be further explored.

[0003] In most existing transmission scheduling methods for industrial network systems, there is no joint scheduling of uplink and downlink data transmission for perception and control performance; there is also a lack of analysis between the adopted intelligent scheduling methods and the comprehensive performance of the system, and the degree of matching between the two still needs to be further improved. For example, the "Wireless Networked Control System Resource Scheduling Method Based on IEEE802.11ax" with domestic application number 202310602949.8 proposed a multi-subsystem transmission scheduling method for control performance, but has not yet considered the situation where a single system needs to integrate multi-sensor information to support control decisions; the "A Method for On-Demand Transmission of Perception Information in Industrial Internet of Things" with domestic application number 202211394440.0 and the "A Method for Industrial Edge Perception with Observability Guarantee" with domestic application number 202110453745.3 only focus on the uplink. The transmission scheduling of perception data does not take into account the on-demand transmission requirements of downlink control instructions; the "A method for intelligent scheduling of cache and communication resources of a single relay in the Internet of Things" with domestic application number 202110824751.5 and the "A method for intelligent scheduling of hybrid business flows in air-based networks based on deep reinforcement learning" with domestic application number 202311674058.X, although they have introduced intelligent reinforcement learning methods to obtain the optimal transmission mechanism, have not yet been combined with specific industrial objects to characterize the perception or control performance of the system. Therefore, the relationship between intelligent scheduling design and system control performance still needs to be further explored.

[0004] Therefore, technicians in this field are committed to developing an intelligent transmission scheduling method for industrial network systems driven by perception and control performance. Summary of the invention

[0005] In view of the above-mentioned defects of the prior art, the technical problems to be solved by the present invention are: to reasonably reserve the required overall transmission resources for perception and control performance, and guide the design of dynamic transmission scheduling based on intelligent learning methods; for actual complex and unknown industrial systems, consider the dynamic transmission scheduling scenarios that require the fusion of multi-sensor data for system status perception and require multiple actuators to jointly complete the control objectives; and to finely design the intelligent scheduling method in combination with perception and control performance to ensure the comprehensive performance of the intelligent scheduling method.

[0006] To achieve the above object, the present invention provides a method for intelligent transmission scheduling of an industrial network system driven by perception and control performance, characterized in that the method comprises the following steps:

[0007] Step S1: construct a first model to characterize the dynamics of the industrial object system, construct a first optimization problem for the industrial network system to optimize the scheduling of uplink and downlink data transmission for perception and control, construct a perception model characterized by parameters to estimate the state of the industrial system, and iteratively update the perception model parameters based on available historical data;

[0008] Step S2: taking the observability index and controllability index of the system in the selected multiple time periods as constraints, by solving the second optimization problem, determining the total number of uplink and downlink transmission time slots reserved for each period in the subsequent dynamic transmission scheduling process;

[0009] Step S3: Model the dynamic transmission scheduling process of the industrial network system as a Markov decision process, construct a policy network represented by parameters, and use a policy gradient reinforcement learning method to iteratively update the parameters;

[0010] Step S4: After multiple rounds of reinforcement learning training, if the control performance and perception performance meet the production accuracy requirements of the industrial network system, the trained policy network and perception model parameters are saved; if the control or perception performance does not meet the required production accuracy requirements, it is necessary to readjust the values ​​of the relevant weights or constraints to make the system tend to have more sufficient uplink and downlink data interaction, and re-perform the reinforcement learning training process until the performance meets the requirements.

[0011] Furthermore, the first model in step S1 is:

[0012] x k+1 =f(x k ,Δ k u k )+w k ,k∈{1,…,T}

[0013] y k =Γ k (h(x k )+v k ).

[0014] In the formula, T is the time range, x k ,y k and u k are the system state, measurement value and control input of k period respectively, and w k and v k are k-cycle system noise and measurement noise respectively; f is the unknown system model; h is the measurement model; Γ k and Δ k They are k-period uplink and downlink transmission success indication matrices, respectively, and are determined by the number of uplink and downlink data transmissions and the single transmission success rate of the dynamic transmission scheduling.

[0015] Further,

[0016] The first optimization problem in step S1 is:

[0017]

[0018]

[0019] Where γ is the discount factor, and the inequality constraint indicates that the sum of the number of uplink and downlink data transmissions in each cycle must be less than the total number of reserved time slots b; c k The cost function is:

[0020] Q and R are the weights of control performance and control dosage, respectively. u and κ d are the uplink and downlink transmission cost weights, N and L are the number of sensors and actuators, respectively. is the number of transmissions of sensor j in cycle k during the dynamic scheduling process, is the number of transmissions from the edge computing unit to executor i within the k cycle during the dynamic scheduling process.

[0021] Furthermore, the iterative updating of the perception model parameters based on the available historical data in step S1 is as follows:

[0022]

[0023] In the formula, l θ is the learning rate for θ update; in is the mean of the initial value of the system state, θ is the characterization parameter of the system perception model, g θ is the perception model, is the system state estimate, is the gradient operator with respect to θ.

[0024] Furthermore, the second optimization problem is:

[0025] System observability indicators over multiple time periods and controllability indicators As constraints, solve:

[0026]

[0027] ω o and ω c are the constraint values ​​of observability and controllability indicators respectively; is the number of transmissions of sensor j in each cycle of the reservation process, is the number of transmissions from the edge computing unit to the actuator i in each cycle of the reservation process, affecting and result.

[0028] Furthermore, the intelligent transmission scheduling problem of the industrial network system in step S3 is modeled as a Markov decision process:

[0029] Define decision states k as follows:

[0030]

[0031] Define decision action a k as follows:

[0032]

[0033] In the formula, p k is the overall transmission scheduling ratio, and are the transmission scheduling weights of sensor j and actuator i respectively, then the number of transmissions from sensor j and edge computing unit to actuator i within k cycles is determined as follows:

[0034]

[0035] Match the optimization target and define the k-cycle return r k For r k =-c k .

[0036] Furthermore, the policy network in step S3 is characterized by parameter β, which represents the decision state s k Next, take action a kThe probability of edge computing unit is that after each round of interaction, the edge computing unit collects trajectories τ={s1,a1,r1,…,s T ,a T ,r T ,s T+1}, the parameter β is iteratively updated as follows:

[0037]

[0038] An operator is expected.

[0039] Furthermore, the step S3 also includes: after each round of interaction, the edge computing unit calculates the Learning and updating the system perception model parameters θ:

[0040]

[0041] Furthermore, the control performance in step S4 is

[0042] Furthermore, the weights in step S4 include Q, R, κ u , κ d , the constraints include ω o ,ω c .

[0043] The present invention proposes an intelligent transmission scheduling method for industrial network systems driven by perception and control performance, which is aimed at the actual needs of balancing system performance and limited transmission resources in complex industrial production. The intelligent transmission scheduling method adopted matches the current expanding industrial production scale and potential unknown system parameters. The transmission scheduling method proposed by the present invention can be deployed conveniently and quickly in the absence of precise prior information on the system model, and the weights of various performance and resources can be adjusted according to actual production needs to achieve ideal production effects. The present invention closely connects advanced intelligent methods with industrial production perception and control needs, and can be effectively promoted to a variety of different industrial application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 An industrial network system architecture diagram supported by edge computing in an embodiment of an industrial network system intelligent transmission scheduling method driven by perception and control performance of the present invention;

[0045] Figure 2 A schematic diagram of the framework composition of data transmission flow of the intelligent transmission scheduling method for an industrial network system driven by perception and control performance of the present invention;

[0046] Figure 3A schematic diagram of a training process for a specific embodiment of the method for intelligent transmission scheduling of an industrial network system driven by perception and control performance of the present invention;

[0047] Figure 4 The present invention is a flowchart of a specific embodiment of the method for intelligent transmission scheduling of an industrial network system driven by perception and control performance. DETAILED DESCRIPTION

[0048] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0049] In the drawings, components with the same structure are indicated by the same numerical reference numerals, and components with similar structures or functions are indicated by similar numerical reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component. In order to make the illustration clearer, the thickness of the components is appropriately exaggerated in some places in the drawings.

[0050] The perception in the embodiment of the present invention mainly includes data collection and data uplink transmission performed by the hardware components of the sensors in the industrial network system, and obtaining system state estimation after performing information analysis on the above data; the control in the embodiment of the present invention mainly includes obtaining system feedback control parameters after performing information analysis on the above data after the sensors in the industrial network system collect data, and sending the control parameters to the actuator.

[0051] Taking a typical industrial network system industrial hot rolling laminar cooling process as an example, the architecture of the above system consists of two parts: the field equipment layer and the edge computing layer. The field equipment layer mainly includes a sensor network composed of sensors such as high-temperature guns and thermal imagers. The above-mentioned sensing elements upload the collected data to the edge computing layer. The field equipment layer also includes actuators such as sprinklers to receive the control parameters issued by the edge computing unit to achieve cooling control. The system architecture is as follows Figure 1 As shown in the figure, the mechanical properties of steel are directly affected by the temperature cooling curve, so the temperature control of the strip is crucial in this process. A large number of thermal imagers, high-temperature guns and other sensors deployed at the industrial site transmit the measured strip temperature data to the edge computing unit wirelessly, and the edge computing unit sends control instructions to each cooling water valve to control its opening. Therefore, it is urgent to balance the perception and control performance of the strip temperature with the limited data transmission resources at the hot rolling site.

[0052] like Figure 2As shown, it is a schematic diagram of the framework composition of the data transmission flow of the intelligent transmission scheduling method of the industrial network system driven by the perception and control performance of the present invention. In the above data flow graphic example, a total of steps need to be experienced, including data collection, uplink transmission, intelligent perception, feedback control, downlink transmission, and the update system after control execution, and then a new round of data collection in the new state is carried out. Among the several main steps of the above data control flow, in each process link, some involve data uploading, and some need to generate control instructions, and in the process of uplink and downlink transmission of data, observable-controllable performance traction and overall transmission quantity constraints are involved.

[0053] Based on the above Figure 2 The overall data transmission flow process shown in FIG. 1 is a schematic diagram of the training process of the perception and control performance driven industrial network system intelligent transmission scheduling method of the present invention. Figure 3 As shown, the problems solved in each step are:

[0054] Step 1: Optimize question construction and perception model pre-learning;

[0055] Step 2: Reservation of transmission resources;

[0056] Step 3: Dynamic transmission scheduling based on reinforcement learning;

[0057] Step 4: Perception and control performance evaluation.

[0058] Further, according to the perception and control performance driven intelligent transmission scheduling method of industrial network system implemented by the present invention, Figure 4 As shown in , the steps include:

[0059] Step S1: construct a first model to characterize the dynamics of the industrial object system, construct a first optimization problem for the industrial network system to optimize the scheduling of uplink and downlink data transmission for perception and control, construct a perception model characterized by parameters to estimate the state of the industrial system, and iteratively update the perception model parameters based on available historical data;

[0060] Step S2: taking the observability index and controllability index of the system in the selected multiple time periods as constraints, by solving the second optimization problem, determining the total number of uplink and downlink transmission time slots reserved for each period in the subsequent dynamic transmission scheduling process;

[0061] Step S3: Model the dynamic transmission scheduling process of the industrial network system as a Markov decision process, construct a policy network represented by parameters, and use a policy gradient reinforcement learning method to iteratively update the parameters;

[0062] Step S4: After multiple rounds of reinforcement learning training, if the control performance and perception performance meet the production accuracy requirements of the industrial network system, the trained policy network and perception model parameters are saved; if the control or perception performance does not meet the required production accuracy requirements, it is necessary to readjust the values ​​of the relevant weights or constraints to make the system tend to have more sufficient uplink and downlink data interaction, and re-perform the reinforcement learning training process until the performance meets the requirements.

[0063] Specifically, step S1 also includes:

[0064] Step S1.1: According to the heat conduction equation, the following system model is constructed to describe the temperature evolution process of the strip steel:

[0065] x k+1 =f(x k ,Δ k u k )+w k ,k∈{1,…,T}

[0066] y k =Γ k (h(x k )+v k ).

[0067] Where T is the time range, x k ,y k and u k are the system state, measurement value and control input of k period respectively, and w k and v k are k-cycle system noise and measurement noise respectively; f is the unknown system model, which is determined by the processes of heat conduction, water cooling and air cooling of the strip; h is the measurement model, which is determined by the layout of various temperature sensors in the industrial site; Γ k and Δ k They are k-period uplink and downlink transmission success indication matrices, respectively, and are determined by the number of uplink and downlink data transmissions and the single transmission success rate of the dynamic transmission scheduling.

[0068] Step S1.2: Construct the first optimization problem as follows:

[0069]

[0070] Where γ is the discount factor, and the inequality constraint indicates that the sum of the number of uplink and downlink data transmissions in each cycle must be less than the total number of reserved time slots b; c k is the cost function:

[0071] Q and R are the weights of control performance and control dosage, respectively. u and κ dare the uplink and downlink transmission cost weights, N and L are the number of sensors and actuators, respectively. is the number of transmissions of sensor j in cycle k during the dynamic scheduling process, is the number of transmissions from the edge computing unit to actuator i within the k cycle during the dynamic scheduling process. In order to achieve the overall optimization goal, in addition to the uplink and downlink transmission scheduling, the control input design is also involved in the optimization problem.

[0072] Step S1.3: To obtain the system state estimate Construct a perception model g based on a multi-layer neural network represented by parameter θ θ , based on the available historical data, the perception model parameters are iteratively updated as follows:

[0073]

[0074] In the formula, l θ is the learning rate for updating the parameter θ; in is the mean of the initial values ​​of the system state, is the gradient operator with respect to θ.

[0075] Furthermore, step S2 further includes:

[0076] The observability index of the system in the selected d time periods and controllability indicators As a constraint condition, the total number of uplink and downlink transmission time slots b reserved for each period in the subsequent dynamic transmission scheduling process is determined by solving the following second optimization problem:

[0077]

[0078] In the formula, and are respectively taken as the expectation of the trace of the observability matrix and the controllability matrix of the system, ω o and ω c are the constraint values ​​of observability and controllability indicators respectively; is the number of transmissions of sensor j in each cycle of the reservation process, is the number of transmissions from the edge computing unit to the actuator i in each cycle of the reservation process, which respectively affect and result.

[0079] Furthermore, step S3 also includes:

[0080] Step S3.1: Model the intelligent transmission scheduling problem of the industrial network system as a Markov decision process and define the decision state s k as follows:

[0081]

[0082] Define decision action a k as follows:

[0083]

[0084] In the formula, p k is the overall transmission scheduling ratio, and are the transmission scheduling weights of sensor j and actuator i respectively, then the number of transmissions from sensor j and edge computing unit to actuator i within k cycles is determined as follows:

[0085]

[0086] Match the optimization target and define the k-cycle return r k For r k =-c k .

[0087] Step S3.2: Construct a policy network π represented by parameter β β (a k |s k ), which represents the decision state s k Next, take action a k The probability of edge computing unit is that after each round of interaction, the edge computing unit collects trajectories τ={s1,a1,r1,…,s T ,a T ,r T ,s T+1}, the policy gradient reinforcement learning method is used to iteratively update the parameter β as follows:

[0088]

[0089] In the formula, l β is the learning rate for parameter β update, is the gradient operator with respect to β, E τ is the expected operation symbol about τ. In this step, the value network can also be introduced to estimate the state value function of the reinforcement learning process, and the gradient descent method can be used to update the parameters of the value network.

[0090] Step S3.3: After each round of interaction, the edge computing unit collects data Learning and updating the system perception model parameters θ:

[0091]

[0092] Furthermore, step S4 further includes:

[0093] After multiple rounds of reinforcement learning training, it is determined whether the control performance and perceived performance ε k If the required strip production accuracy is met, the trained policy network and perception model parameters are saved for online dynamic transmission scheduling after method deployment; if the control or perception performance does not meet the required production accuracy requirements, the relevant weights including Q, R, κ need to be readjusted u , κ d , the constraints include ω o ,ω c , so that the system tends to have more sufficient uplink and downlink data interaction, and re-performs the reinforcement learning training process until the performance meets the requirements.

[0094] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A perception and control performance driven intelligent transmission scheduling method for industrial network systems, characterized in that: The above method comprises the following steps: Step S1: construct a first model to characterize the dynamics of the industrial object system, construct a first optimization problem for the industrial network system to optimize the scheduling of uplink and downlink data transmission for perception and control, construct a perception model characterized by parameters to estimate the state of the industrial system, and iteratively update the perception model parameters based on available historical data; Step S2: taking the observability index and controllability index of the system in the selected multiple time periods as constraints, by solving the second optimization problem, determining the total number of uplink and downlink transmission time slots reserved for each period in the subsequent dynamic transmission scheduling process; Step S3: Model the dynamic transmission scheduling process of the industrial network system as a Markov decision process, construct a policy network represented by parameters, and use a policy gradient reinforcement learning method to iteratively update the parameters; Step S4: After multiple rounds of reinforcement learning training, if the control performance and perception performance meet the production accuracy requirements of the industrial network system, the trained policy network and perception model parameters are saved; if the control or perception performance does not meet the required production accuracy requirements, it is necessary to readjust the values ​​of the relevant weights or constraints to make the system tend to have more sufficient uplink and downlink data interaction, and re-perform the reinforcement learning training process until the performance meets the requirements.

2. The perception and control performance driven industrial network system intelligent transmission scheduling method according to claim 1, characterized in that: The first model in step S1 is: x k+1 =f(x k ,Δ k u k )+w k ,k∈{1,…,T} y k =Γ k (h(x k )+v k ). Where T is the time range, x k ,y k and u k are the system state, measurement value and control input of k period respectively, and w k and v k are k-cycle system noise and measurement noise respectively; f is the unknown system model; h is the measurement model; Γ k and Δ k They are k-period uplink and downlink transmission success indication matrices, respectively, and are determined by the number of uplink and downlink data transmissions and the single transmission success rate of the dynamic transmission scheduling.

3. The perception and control performance driven industrial network system intelligent transmission scheduling method according to claim 2, characterized in that: The first optimization problem in step S1 is: Where γ is the discount factor, and the inequality constraint indicates that the sum of the number of uplink and downlink data transmissions in each cycle must be less than the total number of reserved time slots b; c k is the cost function: In the formula, z k is the k-period ideal system state, and its trajectory equation is z k =f z (z k-1 ), is the system state estimation; Q and R are the control performance and control usage weights respectively, κ u and κ d are the uplink and downlink transmission cost weights, N and L are the number of sensors and actuators, respectively. is the number of transmissions of sensor j in cycle k during the dynamic scheduling process, is the number of transmissions from the edge computing unit to executor i within the k cycle during the dynamic scheduling process.

4. The perception and control performance driven industrial network system intelligent transmission scheduling method according to claim 3, characterized in that: The iterative update of the perception model parameters based on the available historical data in step S1 is as follows: In the formula, l θ is the learning rate for θ update; in is the mean of the initial value of the system state, θ is the characterization parameter of the system perception model, g θ is the perception model, is the system state estimate, is the gradient operation symbol about θ.

5. The perception and control performance driven industrial network system intelligent transmission scheduling method according to claim 4, characterized in that: The second optimization problem is: System observability indicators over multiple time periods and controllability indicators As constraints, solve: ω o and ω c are the constraint values ​​of observability and controllability indicators respectively; is the number of transmissions of sensor j in each cycle of the reservation process, is the number of transmissions from the edge computing unit to the actuator i in each cycle of the reservation process, affecting and result.

6. The perception and control performance driven industrial network system intelligent transmission scheduling method according to claim 5, characterized in that: The intelligent transmission scheduling problem of the industrial network system in step S3 is modeled as a Markov decision process: Define decision states k as follows: Define decision action a k as follows: In the formula, p k is the overall transmission scheduling ratio, and are the transmission scheduling weights of sensor j and actuator i respectively, then the number of transmissions from sensor j and edge computing unit to actuator i within k cycles is determined as follows: Match the optimization target and define the k-cycle return r k For r k =-c k .

7. The perception and control performance driven intelligent transmission scheduling method for industrial network systems according to claim 6, characterized in that: The policy network in step S3 is characterized by parameter β, which represents the decision state s k Next, take action a k The probability of edge computing unit is that after each round of interaction, the edge computing unit collects trajectories τ={s1,a1,r1,…,s T ,a T ,r T ,s T+1 }, the parameter β is iteratively updated as follows: In the formula, l β is the learning rate for parameter β update, is the gradient operator with respect to β, E τ is the expected operation sign about τ.

8. The perception and control performance driven industrial network system intelligent transmission scheduling method according to claim 1, characterized in that: The step S3 also includes: after each round of interaction, the edge computing unit calculates the Learning and updating the system perception model parameters θ:

9. The perception and control performance driven industrial network system intelligent transmission scheduling method according to claim 1, characterized in that: The control performance in step S4 is:

10. The perception and control performance driven industrial network system intelligent transmission scheduling method according to claim 1, characterized in that: The weights in step S4 include Q, R, κ u , κ d , the constraints include ω o ,ω c .

Images