Sensor sampling scheduling method in industrial wireless sensor networks based on reinforcement learning

Abstract

The present invention discloses a sensor sampling scheduling method in an industrial wireless sensor network based on reinforcement learning. The method is for scheduling sensor sampling tasks in a time-series industrial wireless sensor network and edge computing, including defining the indicator Age of Date Processing (AoDP) as freshness to measure the system state. The present invention selects access modes between decision sensors and large and small base stations and the sampling frequency of each sensor. Under the premise of satisfying the bandwidth of large and small base stations and optimizing the allocation of computing resources, the system seeks to minimize the system state freshness (AoDP) of the receiving terminal. A joint optimization based on reinforcement learning and dynamic programming is constructed to optimize the joint decision of uplink transmission and waiting time of various types of sensors under sampling tasks in wireless sensor networks. The present invention can provide a reference for the reasonable decision of minimizing system freshness and sampling equipment for time-sensitive applications in wireless sensor networks and industrial Internet of Things.

Description

Technical Field

[0001] The present invention belongs to a network resource scheduling optimization technology in a time-series industrial wireless sensor, and specifically relates to a sensor sampling scheduling method in an industrial wireless sensor network based on reinforcement learning. Background Art

[0002] In order to improve processing efficiency and achieve higher economic benefits for enterprises, the use of industrial wireless sensor networks is increasing. At present, industrial wireless sensor networks have promoted the development of many applications, such as fire detection in sensor networks, fault detection and intelligent driving in intelligent transportation networks. In order to better realize the intelligent service of sensors, its effectiveness, robustness and real-time performance are becoming more and more important. Among them, the most important indicator is the measurement of system freshness, that is, real-time performance. Although there are many indicators for measuring freshness in the existing technology that have been defined and applied to actual production environments, there are still many problems that have not been effectively solved. The problems mainly include two aspects:

[0003] 1) The traditional definition of freshness is the process from when data is sampled to when it is sent to the receiving end. However, with the increasing complexity of applications, the sampled raw data cannot be used directly. In most cases, data cleaning and analysis are required, such as neural network and big data analysis. Therefore, it is very necessary to consider the processing process in the definition of information degree.

[0004] 2) In addition, many definitions of information are based on single-type data. Current tasks require multi-dimensional information. For example, in the task of fire detection, it is necessary to sample temperature, smoke concentration, carbon dioxide content and other information. The sampling of single-type information is far from meeting the ultimate goal.

[0005] However, if all the above factors are taken into account, the realization of the industrial wireless sensor network system with edge computing capabilities will encounter certain difficulties. The reasons include the following three points:

[0006] a. Considering that the processing process of sampling information is very complicated, how to choose a suitable base station with edge computing function to optimize the processing process of sampling data, and how to better utilize system resources to achieve global optimization under the premise of fixing the overall system processing resources.

[0007] b. For multiple sensors of different types, how to reasonably schedule their transmission order to minimize the final system freshness deserves further consideration. Moreover, for a single sensor, the end time of this round of transmission will have a great impact on the sampling time of the next round of sensor equipment.

[0008] c. For multiple sensor devices, it is difficult to reasonably schedule their sampling sequence. How to achieve the optimal sampling sequence is worth considering. Summary of the invention

[0009] Purpose of the invention: In view of the problems of lack of freshness measurement standards for industrial wireless sensor network equipment and insufficient resource optimization in the above-mentioned prior art, the present invention provides a sensor sampling scheduling method in an industrial wireless sensor network based on reinforcement learning.

[0010] Technical solution: A sensor sampling scheduling method in an industrial wireless sensor network based on reinforcement learning, comprising the following steps:

[0011] (1) Establish an industrial wireless sensor network device group with time-series task sampling and a large and small base station group including edge servers, build an industrial wireless sensor network with edge computing capabilities, and establish a model for task sampling, uplink mode selection, and sampling frequency decision based on the industrial wireless sensor network;

[0012] (2) Each task in the task set K is transmitted to the target receiver, which transmits the sampling requirements of the task to the corresponding set S of terminal sensors through the macro base station MBS and the small base station SBS. Each sensor starts sampling the corresponding different types of data.

[0013] (3) After the sensor samples the data, it performs uplink transmission mode I s,b ∈{0,1} selection and waiting window Choice;

[0014] (4) After each sensor makes a decision, the corresponding base station starts to allocate bandwidth to the sampled data. Calculate the corresponding uplink transmission speed based on Shannon's theorem After the base station receives the sampled data, it allocates CPU calculation frequency to each task.

[0015] (5) If the uploaded object is an SBS, the data continues to be uploaded to the macro base station MBS, and the MBS waits for the transmission of other types of data and collects these results;

[0016] (6) Repeat steps (2) to (5), and the MBS receives and integrates the calculation and analysis results of all sampled data, and transmits the final analysis results to the receiving end;

[0017] (7) After task set k is processed, the next task set k+1 is processed, and so on.

[0018] Furthermore, each task arriving in step (1) requires multiple types of sampling data as the basis for analyzing the final system state. The time when this sampling ends The start time of the next task sampling And, due to the integrity of a task, the start time of the kth task and end time It will be affected by the completion time of all sensing devices.

[0019] Step (2) is the process from the generation of task k to its sending to the sensor, wherein the transmission of the task from the receiving end to the macro base station MBS is an exclusive wireless transmission, the transmission from MBS to the SBS is a wired backhaul transmission, and the transmission from SBS to the terminal sensor device is a wireless cellular network transmission.

[0020] In step (3), after each sensor device completes sampling data, the system manager needs to perform uplink transmission mode I for each sensor. s,s ∈{0,1} selection and waiting window choice.

[0021] For the selection in step (3), the freshness index AoDP is defined according to the conditions that the sensor frequency should meet, and then the selection of the uplink mode of the sensor device and the sampling window constraint problem are considered. Transformed into a constrained Markov problem Finally, it is transformed into a Markov problem To solve;

[0022] Among them, the expression of the defined freshness index AoDP is as follows:

[0023]

[0024]

[0025] △ k (T) indicates how long it has been since the latest data information was received. AoDP The freshness over a long period of time is described by integration.

[0026] The selection of uplink mode and sampling window constraints of sensor devices The mathematical form is as follows:

[0027]

[0028] st,

[0029]

[0030]

[0031]

[0032]

[0033]

[0034] Constrained Markov Problem The mathematical expression is as follows:

[0035]

[0036] st,

[0037] Among them, r(X k ,φ k ) is in the current state X k and the current decision φ k The reward value that can be obtained under this condition, γ is the exploration factor, and the constraint y s (π) is to optimize the waiting window under the constraint of strategy π, and then optimize the sampling frequency of the sensor device;

[0038] Markov Problem The expression is as follows:

[0039] [P3]:

[0040]

[0041]

[0042] In the formula, π L is the strategy of adding Lagrange multipliers. The corresponding variable plus the superscript L indicates the variable after adding the Lagrange multiplier. γ is the exploration factor, which indicates the utilization degree of the reward plus the one-time reward in the current state. is the time when the kth task is completed, is the time when the kth task sampling starts.

[0043] In step (4), once the uplink transmission mode of each sensor device is determined, the corresponding macro base station or small base station receiving terminal will start to allocate transmission bandwidth to the sensor. and CPU calculation frequency Then, combining the distance and Rayleigh fading model, the channel gain of the corresponding channel can be obtained Then we can further get the corresponding channel ratio The speed stated The calculation expression is as follows:

[0044]

[0045] Among them, the channel gain of the corresponding channel is calculated according to the distance and Rayleigh fading model: The corresponding channel ratio is:

[0046] Furthermore, in the method, the sizes of different types of sampled data are expressed as Indicates that the delay expression of uplink transmission is as follows:

[0047]

[0048] After the base station allocates computing resources to the corresponding transmission data, according to the formula To calculate the corresponding processing time.

[0049] In the method, after the sampled data is processed by the base station, the final result cannot be obtained directly. Different processing results need to be jointly analyzed to finally obtain the analysis result of the data. The macro base station must wait until all results are collected before performing the final analysis, integration and processing.

[0050] The calculation results of the macro base station integration analysis are as follows:

[0051]

[0052]

[0053] In the formula, is the time when the kth task is completed, that is, the time when the final processing result is transmitted to the receiving terminal; is the time when the kth task sampling starts; is the uplink delay experienced by the sensor device s of the selected MBS base station during the kth uplink transmission; is the data processing delay experienced by the sensor device s of the selected MBS base station during the kth uplink transmission; similarly, and is the uplink delay and data processing delay experienced by the sensor device s of the small base station b during the kth uplink transmission; t backhaul It is the backhaul delay experienced by data when it is transmitted from the small base station to the MBS after the small base station has completed the processing; finally, It is the time MBS takes to collect, process and analyze data.

[0054] Furthermore, in the method, each sensor has a sampling end time. The start time of the next task sampling The start time of the kth task in the calculation process of the macro base station MBS is affected. and end time It will be affected by the completion time of all sensing devices.

[0055] Beneficial effects: Compared with the prior art, the substantial progress and significant effects of the present invention are as follows:

[0056] 1) The present invention takes into account the characteristics of industrial wireless sensor networks and targets the most influential indicator, namely, adds the timing characteristics of the sampling process of multiple types of data and the processing of the original sampled data to the definition of freshness, which can effectively cope with complex tasks and intelligent scenarios in industrial applications.

[0057] 2) Based on the original problem definition, the method of the present invention transforms the problem form and establishes the Markov decision process, and finally achieves the uplink transmission mode selection, the sequence of sequential sampling and the determination of the waiting window of the joint sensor sampling of the sequential industrial wireless sensor network application based on reinforcement learning. Through comparative experiments, the final effect shows that the present invention can better schedule the sampling sequence and sampling frequency of the sensor equipment, and finally optimize the freshness of the system status information arriving at the receiving end.

[0058] 3) The algorithm constructed based on the present invention continuously adjusts the strategy of system decision-making through self-learning and continuous interaction with the environment. This intelligent adjustment strategy can adapt to complex and changeable network environments, and is superior to strategies with fixed premises. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A schematic diagram of the topological structure of the sensor network in the embodiment;

[0060] Figure 2 A schematic diagram of the new freshness indicator AoDP according to the present invention;

[0061] Figure 3 The figure shows a comparison between the embodiment and the prior art in terms of the long-term average AoDP at different numbers of small base stations;

[0062] Figure 4 Shown is a comparison of the long-term average AoDP of different numbers of sensors in a unit SBS area in the embodiment and the prior art. DETAILED DESCRIPTION

[0063] In order to explain the technical solution disclosed in the present invention in detail, further elaboration is made below in conjunction with the accompanying drawings and specific embodiments of the specification.

[0064] In view of the serious network congestion caused by forwarding a large number of different types of sampled data to a remote base station for processing, and the further reduction of the system freshness, the time-sensitive industrial wireless sensor network equipment application cannot be transmitted, processed and minimized in freshness within the effective time, and the system cannot achieve the optimization of freshness, the present invention provides a sensor sampling scheduling method in an industrial wireless sensor network based on reinforcement learning, which is specifically based on reinforcement learning in the sampling task mode selection and constraint sampling frequency calculation processing of the joint sensor in the time-series industrial wireless sensor network application and edge computing, including defining an indicator AoDP for measuring the freshness of the system state. The method is based on the edge service base station and the terminal sensor device in the industrial wireless sensor network, including establishing an industrial wireless sensor network platform with edge computing capabilities, with the system seeking the minimization of the total long-term average AoDP as the goal, while satisfying the edge base station bandwidth and computing resources, the completion time of the time-series sampling sequence and the sampling frequency of the sensor device, jointly determining the optimal uplink transmission mode selection and the determination of the waiting window.

[0065] The present invention considers the processing mode of the time-series task flow of various types of sampled data in industrial wireless sensor networks in edge computing, and also considers the coordination and precedence relationship between different types of sampled data and the resource allocation of edge service base stations. In order to consider the goal of minimizing AoDP in the system, a joint optimization scheme based on reinforcement learning is constructed to solve the goals of mode selection of sensor uplink transmission and sampling frequency optimization.

[0066] On the basis of the above technical solutions and the prior art that should be familiar to those skilled in the art, further, combined with Figure 1 The implementation process of the method described in the present invention is described in detail below.

[0067] Step 1: Establish an industrial wireless sensor network system with edge computing capabilities and build a function model of the system.

[0068] This step mathematically analyzes the practical problems solved by the joint task offloading and virtual machine placement method in edge computing for time-series industrial wireless sensor network applications.

[0069] According to the actual application scenario, the industrial wireless sensor network system with edge computing capability consists of an industrial wireless sensor network sampling device layer, an edge service base station layer, and a receiving end. The industrial wireless sensor network sampling device layer consists of a set of sensors that can use environmental information data. The cardinality of the set is represented by the set S, |S|. The edge service base station layer has a set of edge base stations with heterogeneous processing power and storage capacity. The cardinality of the small base station set is represented by the set {B∪MBS}, |B|.

[0070] Step 2: When the task reaches the receiving end, the receiving terminal will pass the task down to the macro base station MBS, and then the MBS continues through the SBS, and finally transmits the sampling needs to the terminal's sensor, and the sensor starts sampling data.

[0071] Step 3: After the sensor device receives the sampling task, the system manager will select the uplink mode for each sensor device from a global perspective. s,b and sampling window of determination.

[0072] Step 4: The corresponding macro base station or small base station receiving terminal will start to allocate transmission bandwidth to the sensor and CPU calculation frequency Then, combining the distance and Rayleigh fading model, the channel gain of the corresponding channel can be obtained Then we can further get the corresponding channel ratio Finally, the corresponding uplink transmission speed can be obtained The sizes of different types of sampled data can be used Then the uplink transmission delay can be expressed as After the base station allocates computing resources to the corresponding transmission data, according to the formula To calculate the corresponding processing time. After the sampled data is processed by the base station, the final result cannot be obtained directly, because different processing results need to be analyzed together to finally get the data analysis result. Therefore, the macro base station must wait until all results are collected before the final analysis, integration and processing can be performed. After the above analysis, the final processing result of this task can be expressed as:

[0073]

[0074]

[0075] Step 5: The macro base station collects and analyzes the received data, and finally integrates and analyzes the data, and then further transmits the data to the receiving end to complete the overall task process.

[0076] Then each sensor further determines the start time of the next task based on its own waiting time, and this process is repeated continuously.

[0077] Corresponding to the decision-making process in step 3, more specifically, the system controller needs to meet the sampling frequency requirements of each sensor device. Each sensor device needs to meet these conditions:

[0078]

[0079]

[0080]

[0081] That is, each transmission of the sensing device needs to meet the sampling frequency requirement.

[0082] According to the expansion of the above freshness indicators, the final form of AoDP can be obtained:

[0083]

[0084]

[0085] Next, consider the problem of minimizing the system AoDP state information. Finally, we can construct a problem of selecting the uplink mode and sampling window constraints for sensor devices in edge computing for time-series industrial wireless sensor network applications:

[0086]

[0087] st,

[0088]

[0089]

[0090]

[0091]

[0092]

[0093] Through problem conversion, the original problem is transformed into a Markov decision problem, and finally the problem is solved through reinforcement learning. First, according to the definition of AoDP and combined with Figure 2 , original question It can correspond to the integral form in the figure, that is:

[0094]

[0095] Through reasonable approximate transformation of the formal problem, we can get the approximate form of the second line to facilitate the definition and processing of the subsequent Markov problem. Next, we begin to define the basic elements of the Markov problem:

[0096] 1) System status in is the channel condition of uplink transmission of sensor data in the system at the kth task, is the set of all possible channels.

[0097] 2) Behavior collection: Among them, I k and Wk They are the uplink transmission mode decision and waiting window size decision of all sensors in the kth task.

[0098] 3) Reward function: This reward function is abstracted from the original problem. The goal of this invention is to minimize AoDP, that is, to minimize the cumulative reward size. Define strategy π: That is, based on the known state information, the optimal scheduling decision under the current state is obtained through the strategy, and the Markov problem is obtained:

[0099]

[0100] st, Among them, the constraint y s (π) is the constraint under the condition of strategy π, optimizing the waiting window, and then optimizing the sampling frequency of the sensor device. To further optimize the above problem, the constraints are further optimized to the target through the Lagrange multiplier method. In this way, the final Markov problem is obtained from the original constrained Markov problem

[0101] [P3]:

[0102]

[0103]

[0104] Next, the present invention adopts a two-part problem solving approach.

[0105] First, randomly initialize the Lagrange multipliers of all sensors. Then, continuously iterate to find the optimal Lagrange multiplier of each sensor. Specifically, fix the Lagrange multiplier of the first sensor device, and then obtain the current optimal strategy based on the optimized and improved actor-critic reinforcement learning algorithm, and then update the Lagrange multiplier of the corresponding sensor device based on the gradient updated by this strategy. After all the multipliers in the multiplier set are updated, the current optimal strategy is finally solved again, and this optimal strategy is the optimal scheduling strategy. Through the strategy of reinforcement learning, the decision-making choices of the system can change dynamically with the changes in the system environment, so that the method described in the present invention has better adaptability.

[0106] Combination Figure 3 , Figure 4 The experimental results verify the superior performance of the reinforcement learning strategy in this invention.

Claims

1. A sensor sampling scheduling method in an industrial wireless sensor network based on reinforcement learning, characterized in that: The steps include: (1) Establish an industrial wireless sensor network device group with time-series task sampling and a large and small base station group including edge servers, build an industrial wireless sensor network with edge computing capabilities, and establish a model for task sampling, uplink mode selection, and sampling frequency decision based on the industrial wireless sensor network; (2) Each task in the task set K is transmitted to the target receiver, which transmits the sampling requirements of the task to the corresponding set S of terminal sensors through the macro base station MBS and the small base station SBS. Each sensor starts sampling the corresponding different types of data. (3) After the sensor samples the data, it performs uplink transmission mode I s,b ∈{0,1} selection and waiting window Choice; Specifically, the freshness index AoDP is defined according to the conditions that the sensor frequency should meet, and then the selection of the uplink mode of the sensor device and the sampling window constraint are considered. Transformed into a constrained Markov problem Finally, it is transformed into a Markov problem To solve; Among them, the expression of the defined freshness index AoDP is as follows: The selection of uplink mode and sampling window constraints of sensor devices The mathematical form is as follows: st: Constrained Markov Problem The mathematical expression is as follows: s.t. Among them, the constraint y s (π) is the optimal waiting window under the constraint of strategy π, and the sampling frequency of the sensor device is optimized accordingly; Markov Problem The mathematical expression is as follows: In the formula, π L It indicates that the strategy of adding Lagrange multipliers is taken into account. The corresponding variable plus the superscript L indicates the variable after adding the Lagrange multiplier. γ is the exploration factor, which indicates the utilization degree of the reward plus the one-time reward in the current state. is the time when the kth task is completed, is the time when the kth task sampling starts; (4) After each sensor makes a decision, the corresponding base station starts to allocate bandwidth to the sampled data. Calculate the corresponding uplink transmission speed based on Shannon's theorem After the base station receives the sampled data, it allocates CPU calculation frequency to each task. The speed stated The calculation expression is as follows: Among them, the channel gain of the corresponding channel is calculated according to the distance and Rayleigh fading model: The corresponding channel ratio is: (5) If the object to be uploaded is SBS, the processed sampled data will continue to be uploaded to the macro base station MBS, and the MBS will wait for the transmission of other types of data and collect all processed sampling results; (6) Repeat steps (2) to (5), the MBS receives and integrates the calculation and analysis results of all sampled data, and transmits the final analysis results to the receiving terminal; (7) After task set k is processed, the next task set k+1 is processed, and so on.

2. The sensor sampling scheduling method in an industrial wireless sensor network based on reinforcement learning according to claim 1 is characterized in that: In the method, the sizes of different types of sampled data are expressed as Indicates that the delay expression of uplink transmission is as follows: After the base station allocates computing resources to the corresponding transmission data, according to the formula To calculate the corresponding processing time.

3. The sensor sampling scheduling method in an industrial wireless sensor network based on reinforcement learning according to claim 1 is characterized in that: The calculation end time of the macro base station integrated analysis is as follows: In the formula, is the time when the kth total task is completed, indicating the time when the final result is transmitted to the receiving end; is the time when the kth task sampling starts; is the uplink delay experienced by the sensor device s of the selected MBS base station during the kth uplink transmission; is the data processing delay experienced by the sensor device s of the selected MBS base station during the kth uplink transmission; and is the uplink delay and data processing delay experienced by the sensor device s of the selected small base station b during the kth uplink transmission; t backhaul It is the backhaul delay experienced when data is transmitted from the small cell to the MBS after the small cell processing is completed; It is the time MBS takes to collect, process and analyze data.

4. The sensor sampling scheduling method in an industrial wireless sensor network based on reinforcement learning according to claim 1 is characterized in that: In the method, each sensor has a sampling time, and the sampling time ends at The start time of the next task sampling Therefore, the macro base station MBS needs to consider the start time of the kth task during the calculation process. and end time It will affect the order in which the sampling of k+1 tasks ends.

Images