Sensor sampling scheduling method based on reinforcement learning

Abstract

The present application relates to the field of industrial internet of things, and discloses a sensor sampling scheduling method and device based on reinforcement learning, equipment and medium, comprising: inputting the state and action of each first sensor into the first agent corresponding to each first sensor, and the first agent outputs the first action value corresponding to the first sensor action;The state and action of each second sensor are sequentially input into the second agent, and the second agent sequentially outputs the second action value corresponding to each second sensor action;Wherein, the state of the current second sensor includes the action of each other second sensor whose sequential order in the second agent is before the current second sensor;According to the first action value and the second action value, the optimal action value of the sensor to be scheduled is obtained;According to the optimal action value, the sampling scheduling method of the sensor to be scheduled is determined. The freshness effect of sensor information collection is improved.

Description

Technical Field

[0001] This invention relates to the field of industrial Internet of Things (IoT) technology, and in particular to a sensor sampling scheduling method, apparatus, device, and medium based on reinforcement learning. Background Technology

[0002] As the deployment of 5G commercial networks continues to expand, the development of the Internet of Things (IoT) has entered a new stage. According to a Cisco report, by 2023, there will be 14.7 billion internet-connected devices globally, averaging 1.8 sensors per person. Ericsson's mobile report predicts that by 2026, there will be 26.9 billion internet-connected devices.

[0003] Industrial Internet of Things (IIoT) devices constitute a significant portion of the IoT landscape and represent one of its most important development directions. IIoT is considered one of the biggest paradigm shifts in industrial systems, potentially revolutionizing existing industrial production. Due to ubiquitous connectivity and computing, IIoT transforms from an end-to-end communication network into a comprehensive system capable of information collection, processing, and analysis. Because the system is information-centric, its main components serve information, such as sensors that capture information, cache units that store information, and decision-makers that utilize information. In continuously operating industrial systems, service quality is most significantly affected by the timeliness of information delivery. For example, in automated industrial production lines, controllers need to monitor machine information in real time to make decisions that ensure smooth system operation.

[0004] In summary, how to schedule information in IIoT so that control services can obtain up-to-date system information is an important issue. Summary of the Invention

[0005] This invention provides a sensor sampling scheduling method, device, computer equipment, and medium based on reinforcement learning. By rationally scheduling information in the Industrial Internet of Things, the control service can obtain fresh system information, thereby achieving better real-time scheduling of industrial production, further promoting the implementation of the technology, and optimizing industrial production efficiency.

[0006] Firstly, a sensor sampling scheduling method based on reinforcement learning is provided, including:

[0007] A multi-agent decision-making model is constructed based on the classification of sensors to be scheduled. The sensors to be scheduled include a first sensor and a second sensor. The multi-agent model includes a first agent and a second agent. The first agent corresponds one-to-one with the first sensor, and the second agent corresponds to multiple second sensors.

[0008] The state and action of each first sensor are input to the first intelligent agent corresponding to each first sensor, and the first intelligent agent outputs the first action value corresponding to the action of the first sensor.

[0009] The states and actions of each second sensor are sequentially input into the second intelligent agent, and the second intelligent agent sequentially outputs the second action value corresponding to each second sensor action; wherein, the current state of the second sensor includes the actions of each other second sensor that are input into the second intelligent agent in the order of the current second sensor; the current second type of sensor is the second type of sensor that is currently using the second intelligent agent to calculate the second action value;

[0010] Based on the first action value and the second action value, the optimal action value of the sensor to be scheduled is obtained;

[0011] The sampling and scheduling method for the sensors to be scheduled is determined based on the value of the preferred actions.

[0012] In some embodiments, the sensor to be scheduled includes a first sensor and a second sensor, including:

[0013] The sensors to be scheduled include a first sensor that supports agent operations and a second sensor that does not support agent operations.

[0014] In some embodiments, a multi-agent decision-making model is constructed based on the classification of the sensors to be scheduled, including:

[0015] The multi-agent decision-making model is constructed as a Markov game process, and its mathematical representation is {N, S, {ai}. i＝1，2，...N ,П,Γ,{ri} i＝1，2...N};

[0016] Where N represents the number of agents;

[0017] S represents the set of environment states shared by all agents, S = {S} i} i＝1，2，3，...N The shared status information for each sensor includes the AOI value of the sensor information in the AP information pool and the battery information of each sensor itself. The information corresponds to the application relationship, and the specific expression is as follows:

[0018]

[0019] In the formula, a i This represents the action set of the i-th agent, and the agent's action set is represented by the agent's information update strategy:

[0020] a i (t)=α i (t)={x i (t), q i (t), pi (t)}

[0021] Π represents the set of action selection strategies for an agent, Π = {π1, π2, ... π} N}, each π i It represents S i →a i The probability of;

[0022] Γ represents the state transition probability, expressed as S×a1×a2×...×a N →S' represents.

[0023] r represents the immediate reward for multi-agent cooperative behavior; where the immediate reward is composed of the weighted sum of the absolute value of the applied AoCI and the relative value of the applied AoCI.

[0024] The agent acquires -r when the sensor runs out of battery power and / or at the end of a preset computation cycle. b The penalty is used as the final reward; after the sensor successfully completes the information acquisition task within the cycle, r will be obtained based on the peak AoCI during the operation. f (Δ) additional reward.

[0025] In some embodiments, the states and actions of each second sensor are sequentially input into a second intelligent agent, and the second intelligent agent sequentially outputs the second action value corresponding to each second sensor action, including:

[0026] The second sensor is sequentially numbered, where the sequential number is used to indicate the order in which the second sensor inputs to the second intelligent agent;

[0027] The states and actions of each second sensor are input into the second agent sequentially according to their numbering. Each decision is a sub-step, and the reward obtained in each sub-step is discounted by a factor for updating the second agent, specifically expressed as follows:

[0028] G ss =r ss+1 +γ2r ss+2 +...=∑γ2 i r ss+i+1

[0029] In the formula, γ2 is the discount factor, and 0 ≤ γ2 ≤ 1. The reward value obtained by the strategy adopted in the ss-th sub-step is R. ss The subsequent reward is R. ss+1 R ss+2 Value function V π (s) is the expected value of the cumulative reward at state s when using policy π, specifically expressed as:

[0030] V π (s)=E{G ss |S ss =s}=E π {∑γ2 n r ss+n+1 |S ss =s}

[0031] The strategy π must satisfy the following conditions:

[0032] arg A maxV π (s)=arg A maxEΣ π {r2 n r ss+n+1 |S ss =s}

[0033]

[0034] In some embodiments, obtaining the preferred action value of the sensor to be scheduled based on the first action value and the second action value includes:

[0035] Using a greedy algorithm, based on the values ​​of the first and second actions, the maximum action value output by each first and second agent is determined, thus obtaining the optimal action value for the sensor to be scheduled. The specific formula is as follows:

[0036]

[0037] The optimal action value satisfies the following condition with any first agent and second agent:

[0038]

[0039]

[0040] In the formula, Q tot Represents the value of the preferred action;

[0041] Q 1,i Represents any first intelligent agent;

[0042] Q2 represents the second intelligent agent.

[0043] In some embodiments, obtaining the preferred action value of the sensor to be scheduled includes:

[0044] The first action value and the second action value are input into a neural network with positive weights, and the neural network outputs the optimal action value.

[0045] In some embodiments, the multi-agent decision-making model includes at least two first agents, at least one second agent, and a greedy algorithm model; the method further includes training the multi-agent decision-making model.

[0046] The first and second agents are trained at each step by minimizing the loss function, where the loss function is:

[0047]

[0048] In the formula, The target state value of a multi-agent reinforcement learning model is represented as:

[0049]

[0050] γ tot It is a discount factor, representing the proportion of the value of the subsequent state to the present value of the current state;

[0051] It is the target network in the network training process;

[0052] It is the system state set for the next moment;

[0053] It is the action value function for each agent to select the optimal action in the next state;

[0054]

[0055] By interacting with the environment through the network, experience is gained, and then the network is trained using that experience.

[0056] Secondly, a sensor sampling scheduling device based on reinforcement learning is provided, comprising:

[0057] The construction unit is used to construct a multi-agent decision model based on the classification of the sensors to be scheduled. The sensors to be scheduled include a first sensor and a second sensor. The multi-agent model includes a first agent and a second agent. The first agent corresponds one-to-one with the first sensor, and the second agent corresponds to multiple second sensors.

[0058] The first intelligent agent unit is used to input the state and action of each first sensor into the first intelligent agent corresponding to each first sensor, and the first intelligent agent outputs the first action value corresponding to the action of the first sensor.

[0059] The second intelligent agent unit is used to sequentially input the states and actions of each second sensor into the second intelligent agent, and the second intelligent agent sequentially outputs the second action value corresponding to each second sensor action; wherein, the current state of the second sensor includes the actions of each other second sensor that are input into the second intelligent agent in the order of the current second sensor; the current second type of sensor is the second type of sensor that is currently using the second intelligent agent to calculate the second action value;

[0060] The action value output unit is used to obtain the optimal action value of the sensor to be scheduled based on the first action value and the second action value using a greedy algorithm.

[0061] The result output unit is used to determine the sampling and scheduling method of the sensor to be scheduled based on the value of the preferred action.

[0062] Thirdly, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described reinforcement learning-based sensor sampling scheduling method.

[0063] Fourthly, a computer-readable storage medium is provided, which stores a computer program that, when executed by a processor, implements the steps of the aforementioned reinforcement learning-based sensor sampling scheduling method.

[0064] In the aforementioned sensor sampling and scheduling method, apparatus, computer equipment, and storage medium based on reinforcement learning, a multi-agent decision-making model can be constructed according to the classification of the sensors to be scheduled. The state and actions of each first sensor are input to the first agent corresponding to that first sensor, and the first agent outputs the first action value corresponding to the first sensor's action. The state and actions of each second sensor are sequentially input to the second agent, and the second agent sequentially outputs the second action value corresponding to each second sensor's action. The current state of the second sensor includes the actions of all other second sensors whose input order to the second agent precedes the current second sensor. The current second-class sensor is the second-class sensor whose second action value is currently calculated using the second agent. Based on the first and second action values, the preferred action value of the sensor to be scheduled is obtained. The sampling and scheduling method for the sensor to be scheduled is determined based on the preferred action value. This invention aligns with the development trend of industrial applications, effectively improving the freshness of information used in the Industrial Internet of Things and optimizing the accuracy of industrial applications. Attached Figure Description

[0065] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 This is a flowchart illustrating a sensor sampling scheduling method based on reinforcement learning in one embodiment of the present invention;

[0067] Figure 2 This is a flowchart illustrating a multi-agent decision-making model based on a reinforcement learning-based sensor sampling and scheduling method in one embodiment of the present invention.

[0068] Figure 3 This is a schematic diagram of AoCI and AoI in one embodiment of the present invention;

[0069] Figure 4 This is a flowchart of an information collection scheme in one embodiment of the present invention;

[0070] Figure 5 This is a schematic diagram of a sensor sampling scheduling device based on reinforcement learning in one embodiment of the present invention;

[0071] Figure 6 This is a schematic diagram of the structure of a computer device according to an embodiment of the present invention;

[0072] Figure 7 This is another structural schematic diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0073] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0074] The sensor sampling scheduling method based on reinforcement learning provided in this invention can be applied to application-oriented information acquisition schemes in industrial internet scenarios. For example, consider a factory scenario with a set of sensors (note that, in this invention, sensors refer to internet-connected devices in an industrial IoT system): The sensor connects to the network via a wireless access point (AP), and multiple data-supporting applications exist within the factory. The application set includes: Among them, ap i The required sensor set is as follows ui The corresponding application set is: The relationship between the two can be represented by a matrix:

[0075]

[0076] Sensors upload their sensory information to the wireless access point (AP), maintaining the AP's system information database. The AP replaces outdated information stored at the AP with the most recently received packets. Applications then use this information to make decisions. Due to the real-time nature of industrial processes, assuming the application detects information changes in real time, its effectiveness depends on the timeliness of the information. Because the sensor information associated with the application is interleaved, the application does not require the collected information to be generated simultaneously; instead, it uses the latest information received by the AP.

[0077] In the communication process of sensor access network, assuming the AP has communication resources with bandwidth B, this frequency band is divided into K sub-channels, each with a bandwidth of B / K. Due to the factory's large demand for information, the number of sensors in the system is much greater than K. Therefore, a non-orthogonal multiple NOMA access strategy is considered. Let n represent the set of sensors connected to channel k in time slot t. k (t) represents The number of sensors in the network. Assuming that the transmission power and channel coefficients of the sensors connected to the network remain constant within a time slot, and without loss of generality, we assume... The channel coefficients of the sensor access channel k satisfy the following relationship:

[0078]

[0079] The signal-to-interference-plus-noise ratio (SIR) of the sensor using NOMA access is:

[0080]

[0081] Therefore, the transmission rate that the sensor can achieve in NOMA can be expressed as:

[0082]

[0083] To characterize the sensor's performance in the system, this invention uses... Where ∈ i Sensor types are mainly categorized into advanced sensors with certain computing capabilities, integrating ISP chips or other chips with some computing power (such as surveillance probes and patrol robots), and simple, low-level sensors (such as temperature sensors and humidity sensors). iThis indicates the size of the data generated by the sensor. Since sensors generally focus on data acquisition and do not integrate strong data processing capabilities, it is assumed that the data format is fixed and the size does not change. i (t) represents the sensor u in time slot t. i Whether to transmit data, q i (t) represents u in time slot t. i Selected access channel, p i (t) represents u i Transmission power in time slot t.

[0084] Assuming the time it takes for the sensor to acquire data is negligible, the amount of data the sensor can transmit in one time slot is:

[0085] d u (t)=c u，k (t)×τ

[0086] Where τ is the time of one time slot. The energy consumed in this time slot is:

[0087]

[0088]

[0089] Where E' i For u i The energy consumed in each data acquisition. The energy a sensor can consume within its designed operating cycle is finite:

[0090]

[0091] For sensor u i Assuming the time slot for generating information with index j is... Assume the time slot for the information to be transmitted is The amount of data transmitted during this period satisfies:

[0092]

[0093] In time slot t, the AOI information in the AP is adjusted based on whether it was updated in time slot t. If the update is completed in time slot t, assuming the index is... Sensor u in AP i The AOI value at time t can be expressed as:

[0094]

[0095] For the application in the system, ap i The timeliness of its application is determined by the AOI of all input information, that is, by the set of sensors corresponding to the application. The AOCI representation. To reasonably represent the timeliness of information, the AOCI of the set is represented by the AOI of the information with the worst timeliness in the set, that is:

[0096]

[0097] By scheduling sensor u i Update strategy α i (t)={x i (t), q i (t), p i (t)} updates the information in the database required by the application. i The quality of the scheduling strategy is determined by all parties involved in the IoT system. i The expected long-term average AoCI characterization of the relevant application is denoted as J(α). i ):

[0098]

[0099] For all sensors, their policy set The quality is characterized by the expected long-term average AoCI of all applications in the IoT system, denoted as J(α):

[0100]

[0101] Based on the above analysis, the research objective of this invention is to design an information collection strategy for industrial IoT sensors, considering the multi-source information requirements of industrial applications. To facilitate the representation of energy and other related factors, the problem is modeled as a one-cycle optimization problem. Specifically, the optimization problem can be expressed as:

[0102]

[0103] stp i (t)<p max (1)

[0104]

[0105]

[0106] (1) indicates the communication power constraint of the sensor; the transmission power of each sensor is not allowed to exceed the maximum transmission power. (2) indicates that the capacity of the NOMA channel is limited. In order to control the NOMA decoding time, the number of users that can access each channel is limited. (3) indicates that the energy that the sensor can consume within a certain time range is limited, which is used to constrain the energy used by the sensor in information acquisition.

[0107] To address the information acquisition challenges in the Industrial Internet of Things (IIoT), consider the following issues: 1) If information acquisition is conducted at a fixed frequency, how can energy consumption be maintained while ensuring information timeliness? Excessive acquisition frequency leads to significant energy consumption, while lower acquisition frequencies fail to meet timeliness requirements; 2) Centralized control sampling methods suffer from downlink latency, increasing the delay in information acquisition; 3) Distributed control sampling methods are limited by the computing power of some sensors, hindering the deployment of complex control algorithms, and information communication between multiple sensors is also problematic. To address these issues, this invention employs a combined centralized and distributed approach. Distributed acquisition strategies are deployed on sensors with sufficient computing power, while centralized control is used on sensors with limited computing power, controlling information acquisition through downlink information. Furthermore, to achieve coordination between control strategies, a multi-agent reinforcement learning scheme is introduced.

[0108] Given the critical importance of information freshness for control applications, AoI (Aspect-Oriented Information) is used as an indicator to measure information freshness. AOI represents the time elapsed since the latest message was generated at the destination. Compared to traditional QoS metrics such as delay and throughput, AoI provides a new perspective on the freshness or accuracy of information provided by the system to applications. Due to its characteristic of measuring information freshness, AoI has attracted widespread attention in various network architectures, such as cellular networks, vehicle-to-everything (V2X) networks, and the Internet of Things (IoT). AoI is information-centric, influenced not only by the transmission process but also by other stages of the information lifecycle, such as information collection. The main content of this invention is to optimize network-level AoI in IIoT by scheduling information generation and transmission.

[0109] Current research on AoI primarily focuses on information packets rather than applications utilizing that information. Considering that the purpose of information collection in systems is to control system decisions, research on application-based AoI is necessary and practically significant. Specifically in industrial systems, due to the complexity of industrial production processes, system applications typically make decisions based on multiple pieces of information rather than a single system source. For example, decision-making information in material delivery requires road information collected by the delivery vehicle itself, material storage information, and monitoring information from other road sections to complete route planning services. Generally, when a service depends on an information source, the two can be considered correlated. The service quality of an application depends on all information related to it; therefore, the timeliness of all relevant information will affect the application. Given the prevalence of information correlation in IIoT, studying the impact of correlation on AoI is essential.

[0110] This invention focuses on researching scheduling strategies for relevant information sources to minimize network-level AoI in the IIoT. Specifically, consider an IIoT system with an information cache unit storing time-sensitive information collected from IIoT devices such as cameras and sensors. Some devices with computing capabilities can decide whether to update the collected information based on network conditions, while the remaining devices are updated by a central controller. The central and distributed controllers cooperate to schedule information updates from the information sources within the system and maintain the information in the cache unit. For industrial production activities, some applications run continuously, thus requiring them to access the information in the cache unit in real time. The concept of relevant information age is used to measure the timeliness of application information.

[0111] like Figures 1 to 4 As shown, the present invention will be described in detail below through specific embodiments.

[0112] Please see Figure 1 As shown, Figure 1 A flowchart illustrating a sensor sampling scheduling method based on reinforcement learning provided in an embodiment of the present invention includes the following steps:

[0113] S10. Construct a multi-agent decision model based on the classification of the sensors to be scheduled. The sensors to be scheduled include a first sensor and a second sensor. The multi-agent model includes a first agent and a second agent. The first agent corresponds one-to-one with the first sensor, and the second agent corresponds to multiple second sensors.

[0114] As is understood, this invention deals with a multi-agent collaborative problem, which can be viewed as a cooperative game problem, where multiple agents share the same objective. However, due to the different entities implementing the strategy, the agents need to employ different structures: sensors with computing power and centralized controllers deployed at the AP (Application Processor). For ease of description, the former is referred to as the first agent (source agent), and the latter as the second agent (sink agent). The environment contains multiple source agents and one sink agent.

[0115] In some embodiments, the sensors to be scheduled in S10 above, including the first sensor and the second sensor, may include:

[0116] The sensors to be scheduled include a first sensor that supports agent operations and a second sensor that does not support agent operations.

[0117] In some embodiments, constructing a multi-agent decision-making model based on the classification of the sensors to be scheduled in S10 may include:

[0118] The multi-agent decision-making model is constructed as a Markov game process. The mathematical representation of the multi-agent decision-making model is {N, S, {a}.i} i＝1，2，...N ,Π,Γ,{r i} i＝1，2...N};

[0119] Where N represents the number of agents;

[0120] S represents the set of environment states shared by all agents, S = {S} i} i＝1，2，3，...N In this invention, the shared status information for each sensor includes the AOI value of the sensor information in the AP information pool, and channel occupancy due to incomplete transmission in the previous time slot. In addition to the shared sensor information, each sensor also has private status information including its own battery information and the application relationship corresponding to the information. Therefore:

[0121]

[0122] In the formula, a i This represents the action set of the i-th agent. In this invention, the action set of an agent is represented by the agent's information update strategy:

[0123] a i (t)=α i (t)={x i (t), q i (t), p i (t)}

[0124] Π represents the set of action selection strategies for an agent, Π = {π1, π2, ... π} N}, each π i It represents S i →a i The probability of;

[0125] Γ represents the state transition probability, expressed as S×a1×a2×...×a N →S' represents.

[0126] 'r' represents the immediate reward for multi-agent cooperative behavior; a reasonable immediate reward mechanism is crucial for agent training. The reward is assumed to consist of two parts: one is the absolute value of the application's AoCI (Aspect-Oriented CI, where relevant information refers to the application (in industrial applications), where an application relies on information collected from multiple sensors, and the largest (or average) value among these sensors is the maximum), used to measure the real-time state of the application in the current system; the other is the relative value of the application's AoCI, used to measure the change in the application's AoCI after the execution of the policy in this time slot compared to the previous time slot.

[0127]

[0128]

[0129] The reward is represented by a weighted sum of two parts.

[0130] r=αr0+βr1

[0131] In addition, the agent obtains -r when the sensor runs out of battery power and / or when a preset computation cycle (episode, i.e., a preset running cycle, which may not complete the entire sensor cycle) ends. b The penalty is used as the final reward. Similarly, after the sensor successfully completes the information acquisition task within the cycle, it will receive r based on the peak AoCI during the operation. f (Δ) additional reward.

[0132] S20. Input the state and action of each first sensor to the first intelligent agent corresponding to each first sensor, and the first intelligent agent outputs the first action value corresponding to the action of the first sensor.

[0133] S30. The states and actions of each second sensor are sequentially input into the second intelligent agent, and the second intelligent agent sequentially outputs the second action value corresponding to each second sensor action; wherein, the current state of the second sensor includes the actions of each other second sensor that are input into the second intelligent agent in the order of the current second sensor; the current second type of sensor is the second type of sensor that is currently using the second intelligent agent to calculate the second action value;

[0134] like Figure 2 As shown, the first agent has a relatively simple function, responsible only for the acquisition strategy of one sensor. The first agent can be a DRQN ​​network, which may include an input layer, a fully connected layer, a GRU layer, and an output layer. The first agent's network Φ1, a DRQN ​​network, is used to represent the value of selecting an optional action in a certain state, i.e., Q1(s, a) = Φ1(s, a, θ1). The action is selected based on the action value function corresponding to the action given by the Φ1 network.

[0135] The second agent can be a DRQN ​​network, which may include an input layer, a fully connected layer, a GRU layer, and an output layer. It is responsible for scheduling the acquisition of information from all sensors that lack the corresponding computing power. Therefore, its input information must include the state information of all the above sensors, and it also needs to provide the acquisition strategies for each sensor. The latter, in particular, exponentially increases the dimensionality of the action space, increasing the difficulty of network convergence. The generation of acquisition strategies for different sensors is changed from parallel to serial. The network sequentially inputs the state information of different sensors to obtain the acquisition strategies. Furthermore, actions already performed are input as part of the state to ensure that the network can acquire sufficient information.

[0136] In some embodiments, the above-described S30 may include:

[0137] S301. The second sensor is sequentially numbered, wherein the sequential number is used to indicate the order in which the second sensor inputs to the second intelligent agent;

[0138] S302. The states and actions of each second sensor are sequentially input into the second agent according to their numerical order. For the sink agent, which is responsible for scheduling the acquisition of information from all sensors lacking the corresponding computing power, its input information must include the state information of all the sensors and also provide the acquisition strategies for each sensor. The latter, in particular, exponentially increases the dimensionality of the action space, increasing the difficulty of network convergence. Therefore, the generation of acquisition strategies for different sensors is changed from parallel to serial. The network sequentially inputs the state information of different sensors to obtain the acquisition strategies. Furthermore, actions already performed are input as part of the state to ensure the network can acquire sufficient information. In summary, the sink sensor network Φ2 can be represented as:

[0139] Q2(s,a)=Φ2(s,a_,a,θ2)

[0140] Where a_ represents the set of actions previously taken. Each decision is called a sub-step, and the reward obtained in each sub-step requires a discount factor for updating the Φ2 network, representing the correlation between actions taken before and after. That is:

[0141] Gs s =r ss+1 +γ2r ss+2 +...=∑γ2r i ss+i-1

[0142] Where γ2 is the discount factor, and 0≤γ2≤1, the reward value obtained by using a certain strategy in the ss-th step of the sub-step is R. ss Then the subsequent reward is R ss+1 R ss+2 Value function V π (s) is the expected value of the cumulative reward at state s when using policy π:

[0143] V π (s)=E{G ss |S ss =s}=E π {Σγ2 n r ss+n+1 |S ss =s}

[0144] To achieve optimal strategy and maximize expected cumulative return, strategy π must satisfy the following condition:

[0145] arg A maxV π (s)=arg A maxE π {Σγ2 n r ss+n+1 |S ss =s}

[0146] According to the optimal strategy theorem, the best resource allocation strategy is the strategy corresponding to the maximum value function under the following constraints:

[0147]

[0148] S40. Based on the first action value and the second action value, obtain the preferred action value of the sensor to be scheduled;

[0149] In some embodiments, S40 may include:

[0150] Using a greedy algorithm, based on the values ​​of the first and second actions, the maximum action value output by each first and second agent is determined, thus obtaining the optimal action value for the sensor to be scheduled. The specific formula is as follows:

[0151]

[0152] To ensure the generality of the decomposition, the following conditions must be met between the global value and the value of a single agent:

[0153]

[0154]

[0155] In the formula, Q tot Represents the value of the preferred action;

[0156] Q 1,i Represents any first intelligent agent;

[0157] Q2 represents the second intelligent agent.

[0158] Understandably, after designing heterogeneous agents, it's also necessary to coordinate the relationship between the action value of a single agent and the global action value, enabling each agent to optimize its strategy for the overall global objective. This allows each agent to participate in decentralized execution solely by selecting greedy behaviors related to its Q-value function. Furthermore, if the formula for the optimal action value of the sensors to be scheduled is satisfied, then obtaining the Q-value required for non-policy learning updates becomes straightforward. tot argmax is easy to handle.

[0159] In some embodiments, the multi-agent decision model includes at least two first agents, at least one second agent, and a greedy algorithm model; the method further includes training the multi-agent decision model.

[0160] In global training, at each step, the network Φ1(θ1), Φ1(θ2), Φ... mix (θ mix To minimize the loss function To approximate the true global action value, the loss function is:

[0161]

[0162] in This represents the target state value of the Q-network. It can be represented as:

[0163]

[0164] Where γ tot It is a discount factor, representing the proportion of the value of the subsequent state to the present value of the current state. It is the target network in the network training process. It is the system state set for the next moment. It is the action value function for each agent to select the optimal action in the next time step.

[0165]

[0166] By interacting with the environment through the network, the network gains experience, which is then used to train the network. The trained network continues to interact with the environment in a loop, resulting in the Q-value function. Based on the Q-value function, an information scheduling strategy for IIoT devices is obtained.

[0167] S50. Determine the sampling and scheduling method for the sensors to be scheduled based on the value of the preferred actions.

[0168] As can be seen, in the above scheme, to introduce a multi-agent reinforcement learning algorithm, the original problem is first transformed into a Markov game problem. The mathematical language of a Markov game problem can be expressed as {N, S, {a...} i} i＝1，2，...N ,Π,Γ,{r i} i＝1，2...N}. Where N represents the number of agents. S represents the set of environment states shared by all agents, S = {S i} i＝1，2，3，...NIn this invention, the shared state information for each sensor includes the AOI (Area of ​​Interest) value of the sensor information in the AP information pool (AOI represents the time elapsed since the generation of the latest received message in the AP information pool), and channel occupancy due to incomplete transmission in the previous time slot. In addition to the shared sensor information, each sensor also has private state information, including its own battery information and the application relationship corresponding to the information.

[0169] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0170] In one embodiment, a reinforcement learning-based sensor sampling scheduling device is provided, which corresponds one-to-one with the reinforcement learning-based sensor sampling scheduling method described in the above embodiments. For example... Figure 4 As shown, the reinforcement learning-based sensor sampling and scheduling device includes a construction unit 101, a first agent unit 102, a second agent unit 103, an action value output unit 104, and a result output unit 105. Detailed descriptions of each functional module are as follows:

[0171] The construction unit 101 is used to construct a multi-agent decision model based on the classification of the sensors to be scheduled. The sensors to be scheduled include a first sensor and a second sensor. The multi-agent model includes a first agent and a second agent. The first agent corresponds one-to-one with the first sensor, and the second agent corresponds to multiple second sensors.

[0172] The first intelligent agent unit 102 is used to input the state and action of each first sensor into the first intelligent agent corresponding to each first sensor, and the first intelligent agent outputs the first action value corresponding to the action of the first sensor.

[0173] The second intelligent agent unit 103 is used to sequentially input the states and actions of each second sensor into the second intelligent agent, and the second intelligent agent sequentially outputs the second action value corresponding to each second sensor action; wherein, the current state of the second sensor includes the actions of each other second sensor that are input into the second intelligent agent in the order of the current second sensor; the current second type of sensor is the second type of sensor that is currently using the second intelligent agent to calculate the second action value;

[0174] Action value output unit 104 is used to obtain the preferred action value of the sensor to be scheduled based on the first action value and the second action value using a greedy algorithm;

[0175] The result output unit 105 is used to determine the sampling and scheduling method of the sensor to be scheduled based on the preferred action value.

[0176] Specific limitations regarding the reinforcement learning-based sensor sampling scheduling device can be found in the limitations of the intelligent question-answering method above, and will not be repeated here. Each module in the aforementioned reinforcement learning-based sensor sampling scheduling device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0177] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile and / or volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external clients via a network connection. When the computer program is executed by the processor, it implements the functions or steps of a reinforcement learning-based sensor sampling scheduling method on the server side.

[0178] In one embodiment, a computer device is provided, which may be a client, and its internal structure diagram may be as follows: Figure 7 As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements client-side functions or steps of a reinforcement learning-based sensor sampling scheduling method.

[0179] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0180] A multi-agent decision-making model is constructed based on the classification of sensors to be scheduled. The sensors to be scheduled include a first sensor and a second sensor. The multi-agent model includes a first agent and a second agent. The first agent corresponds one-to-one with the first sensor, and the second agent corresponds to multiple second sensors.

[0181] The state and action of each first sensor are input to the first intelligent agent corresponding to each first sensor, and the first intelligent agent outputs the first action value corresponding to the action of the first sensor.

[0182] The states and actions of each second sensor are sequentially input into the second intelligent agent, and the second intelligent agent sequentially outputs the second action value corresponding to each second sensor action; wherein, the current state of the second sensor includes the actions of each other second sensor that are input into the second intelligent agent in the order of the current second sensor; the current second type of sensor is the second type of sensor that is currently using the second intelligent agent to calculate the second action value;

[0183] Based on the first action value and the second action value, the optimal action value of the sensor to be scheduled is obtained;

[0184] The sampling and scheduling method for the sensors to be scheduled is determined based on the value of the preferred actions.

[0185] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0186] A multi-agent decision-making model is constructed based on the classification of sensors to be scheduled. The sensors to be scheduled include a first sensor and a second sensor. The multi-agent model includes a first agent and a second agent. The first agent corresponds one-to-one with the first sensor, and the second agent corresponds to multiple second sensors.

[0187] The state and action of each first sensor are input to the first intelligent agent corresponding to each first sensor, and the first intelligent agent outputs the first action value corresponding to the action of the first sensor.

[0188] The states and actions of each second sensor are sequentially input into the second intelligent agent, and the second intelligent agent sequentially outputs the second action value corresponding to each second sensor action; wherein, the current state of the second sensor includes the actions of each other second sensor that are input into the second intelligent agent in the order of the current second sensor; the current second type of sensor is the second type of sensor that is currently using the second intelligent agent to calculate the second action value;

[0189] Based on the first action value and the second action value, the optimal action value of the sensor to be scheduled is obtained;

[0190] The sampling and scheduling method for the sensors to be scheduled is determined based on the value of the preferred actions.

[0191] It should be noted that the functions or steps that can be implemented by the computer-readable storage medium or computer device described above can be referred to the relevant descriptions on the server side and client side in the foregoing method embodiments. To avoid repetition, they will not be described one by one here.

[0192] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0193] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0194] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A sensor sampling scheduling method based on reinforcement learning, characterized in that, include: A multi-agent decision-making model is constructed based on the classification of sensors to be scheduled. The sensors to be scheduled include a first sensor and a second sensor. The multi-agent model includes a first agent and a second agent. The first agent corresponds one-to-one with the first sensor, and the second agent corresponds to multiple second sensors. The state and action of each first sensor are input to the first intelligent agent corresponding to each first sensor, and the first intelligent agent outputs the first action value corresponding to the action of the first sensor. The states and actions of each of the second sensors are sequentially input into the second agent, and the second agent sequentially outputs the second action value corresponding to each action of the second sensor; wherein, the current state of the second sensor includes the actions of each of the other second sensors that are input into the second agent in the order preceding the current second sensor; the current second type of sensor is the second type of sensor that is currently using the second agent to calculate the second action value; Based on the first action value and the second action value, the preferred action value of the sensor to be scheduled is obtained; The sampling and scheduling method for the sensor to be scheduled is determined based on the preferred action value; The method is used to minimize the network-level AOI in the system. The AOI is used as an indicator to measure the freshness of information. The AOI represents the time elapsed since the latest message was generated at the destination. The step of sequentially inputting the states and actions of each of the second sensors into the second intelligent agent, and the second intelligent agent sequentially outputting the second action value corresponding to each action of the second sensor, includes: The second sensor is sequentially numbered, wherein the sequential number is used to indicate the order in which the second sensor inputs to the second intelligent agent; The states and actions of each of the second sensors are sequentially input into the second agent according to the stated order numbering. Each decision is a sub-step, and the reward obtained in each sub-step requires a discount factor for updating the second agent, specifically expressed as follows: In the formula, It is a discount factor, and The reward value obtained by the strategy used in the ss-th sub-step is The subsequent reward is , , It is the reward value obtained by using the strategy in the next step after step ss. It is the reward value obtained by using the strategy in the last two steps of step ss. It is the reward value obtained by the strategy adopted in the (i+1)th step after the ss-th step, the value function. Is adopting a strategy The expected value of the cumulative reward at state s is specifically expressed as: in, {} represents the expected value. π {} represents the strategy adopted. Expected value at time, strategy The following conditions must be met: in, This refers to finding, within the set A of all possible actions, the action that makes the value function... The biggest movement, It is a strategy , Is adopting a strategy The expected value at that time.

2. The sensor sampling scheduling method based on reinforcement learning as described in claim 1, characterized in that, The sensors to be scheduled include a first sensor and a second sensor, including: The sensors to be scheduled include a first sensor that supports agent operations and a second sensor that does not support agent operations.

3. The sensor sampling scheduling method based on reinforcement learning as described in claim 1, characterized in that, The construction of a multi-agent decision-making model based on the classification of sensors to be scheduled includes: The multi-agent decision-making model is constructed as a Markov game process, and its mathematical expression is as follows: ; in, Indicates the number of agents; This represents the set of environment states shared by all intelligent agents. The shared status information for each sensor includes the AOI value of the sensor information in the AP information pool and the battery information of each sensor itself. The information corresponds to the application relationship, and the specific expression is as follows: In the formula, Indicates the first A set of actions for an agent, represented by the agent's information update strategy: This represents the set of action selection strategies for an agent. Each It means The probability of; This represents the set of actions of agent i in time slot t; This represents the information update strategy of agent i in time slot t; Indicates the sensor in time slot t Whether to transmit data; This indicates that in time slot t, Selected access channel; express Transmission power in time slot t; Representing the state transition probability, using express; This represents the immediate reward for multi-agent cooperative behavior; the immediate reward consists of the weighted sum of the absolute value of the application's AoCI and the relative value of the application's AoCI. The agent obtains [the necessary information] when the sensor runs out of battery power and / or at the end of a preset computation cycle. The penalty is used as the final reward; after the sensor successfully completes the information acquisition task within the cycle, a reward will be given based on the peak AoCI during the operation. Additional rewards.

4. The sensor sampling scheduling method based on reinforcement learning as described in claim 1, characterized in that, The step of obtaining the preferred action value of the sensor to be scheduled based on the first action value and the second action value includes: Using a greedy algorithm, based on the first action value and the second action value, the maximum action value output by each of the first and second agents is determined, thus obtaining the optimal action value for the sensor to be scheduled. The specific formula is as follows: in, It is the preferred action value of action a in the set A. It is the maximum action value of action a1 in the set A. It is the maximum action value of action a2 in the set A. Action a in set A n The maximum action value, and the optimal action value, satisfy the following conditions with respect to either the first agent or the second agent: In the formula, Represents the value of the preferred action; Represents any first intelligent agent; It represents the second intelligent agent.

5. The sensor sampling scheduling method based on reinforcement learning as described in claim 4, characterized in that, The process of obtaining the preferred action value of the sensor to be scheduled includes: The first action value and the second action value are input into a neural network with positive weights, and the neural network outputs the preferred action value.

6. The sensor sampling scheduling method based on reinforcement learning as described in claim 1, characterized in that, The multi-agent decision-making model includes at least two first agents, at least one second agent, and a greedy algorithm model; the method further includes training the multi-agent decision-making model. The first agent and the second agent are trained at each step by minimizing a loss function, wherein the loss function is: In the formula, The target state value of the multi-agent decision-making model is represented as: It is a discount factor, representing the proportion of the value of the subsequent state to the present value of the current state; It is the target network in the network training process; It is the system state set for the next moment; It is the action value function for each agent to select the optimal action in the next state; By interacting with the environment through the network, experience is gained, and then the network is trained using that experience.

7. A device sampling and scheduling device based on reinforcement learning, characterized in that, include: The construction unit is used to construct a multi-agent decision model based on the classification of the sensors to be scheduled. The sensors to be scheduled include a first sensor and a second sensor. The multi-agent model includes a first agent and a second agent. The first agent corresponds one-to-one with the first sensor, and the second agent corresponds to multiple second sensors. The first intelligent agent unit is used to input the state and action of each first sensor into the first intelligent agent corresponding to each first sensor, and the first intelligent agent outputs the first action value corresponding to the action of the first sensor. The second intelligent agent unit is used to sequentially input the states and actions of each of the second sensors into the second intelligent agent, and the second intelligent agent sequentially outputs the second action value corresponding to each action of the second sensor; wherein, the current state of the second sensor includes the actions of each of the other second sensors that were input into the second intelligent agent in the order preceding the current second sensor; the current second type of sensor is the second type of sensor that is currently using the second intelligent agent to calculate the second action value; An action value output unit is used to obtain the preferred action value of the sensor to be scheduled based on the first action value and the second action value using a greedy algorithm. The result output unit is used to determine the sampling and scheduling method of the sensor to be scheduled based on the preferred action value. The method is used to minimize the network-level AOI in the system. The AOI is used as an indicator to measure the freshness of information. The AOI represents the time elapsed since the latest message was generated at the destination. The step of sequentially inputting the states and actions of each of the second sensors into the second intelligent agent, and the second intelligent agent sequentially outputting the second action value corresponding to each action of the second sensor, includes: The second sensor is sequentially numbered, wherein the sequential number is used to indicate the order in which the second sensor inputs to the second intelligent agent; The states and actions of each of the second sensors are sequentially input into the second agent according to the stated order numbering. Each decision is a sub-step, and the reward obtained in each sub-step requires a discount factor for updating the second agent, specifically expressed as follows: In the formula, It is a discount factor, and The reward value obtained by the strategy used in the ss-th sub-step is The subsequent reward is , , It is the reward value obtained by using the strategy in the next step after step ss. It is the reward value obtained by using the strategy in the last two steps of step ss. It is the reward value obtained by the strategy adopted in the (i+1)th step after the ss-th step, the value function. Is adopting a strategy The expected value of the cumulative reward at state s is specifically expressed as: in, {} represents the expected value. π {} represents the strategy adopted. Expected value at time, strategy The following conditions must be met: in, This refers to finding, within the set A of all possible actions, the action that makes the value function... The biggest movement, It is a strategy , Is adopting a strategy The expected value at that time.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the sensor sampling scheduling method based on reinforcement learning as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the sensor sampling scheduling method based on reinforcement learning as described in any one of claims 1 to 6.

Images