An actor-critic based sensor network data uploading method

Abstract

The application discloses a kind of sensor network data upload method based on Actor-Critic. Age of Information (AoI) and energy consumption are used as measurement indicators. Each sensor node monitors information of the corresponding physical process and transmits it to the central controller. The physical process can be regarded as a series of tasks. Due to limited wireless resources, only one sensor node can transmit after monitoring each task. Therefore, the reinforcement learning Actor-Critic algorithm is used to determine which sensor nodes can transmit. The Actor-Critic algorithm enables the scheduling node to select the node that needs to be transmitted for transmission each time, reduces the computational time complexity, and improves the transmission efficiency of the sensor node.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technology, and uses a reinforcement learning method to solve the research problem of sensor data uploading. Background Art

[0002] The future trend of the Internet of Things (IoT) is to achieve ubiquitous sensing and connectivity systems, enabling data-driven decision-making and automation without human intervention. In such systems, packet latency is not the primary performance metric, but rather the freshness of sensor data received at the destination. Outdated sensor data is of little significance to the destination and may even lead to incorrect decisions. AoI is a metric that measures information freshness, defined as the time since the last update packet was received at the destination. This metric differs significantly from classic metrics such as throughput and latency, and more comprehensively reflects the freshness of information at the destination at any given moment.

[0003] Currently, research on AoI is a hot topic. In existing literature, communication between access points and smart devices is bidirectional. Access points have a constant power supply, while smart devices do not. Access points transmit both information and energy to smart devices over a block-fading channel. Smart devices receive both energy and information from the access point and store the energy in an energy buffer. Different weighting coefficients are assigned to the AoI and transmission rate, and a trade-off is made between these two factors. An efficient algorithm is then used to propose the optimal power allocation ratio and weighting coefficient. However, this literature does not consider the possibility that the AoI may exceed a certain threshold. Other literature proposes the AoI violation probability, defined as the probability that the AoI exceeds a given value. Reducing the violation probability is crucial for ensuring data freshness in IoT systems. The system consists of a sensor node, a monitor, and an M / M / 1 queue that follows a first-come, first-served policy. At every moment, the sensor node transmits a data packet to the monitor. The paper derives the packet generation rate that minimizes the violation probability. Most research focuses on the average AoI violation probability, while some propose optimizing the peak AoI violation probability. This study examines a last-come, first-served preemptive strategy for processing packets based on their last arrival time in a point-to-point communication system. To address potential packet loss, this paper employs an automatic repeat protocol for retransmission within the channel, along with quadrature phase-shift keying modulation and maximum likelihood channel estimation. For a given packet size, certain physical layer parameters, the number of transmit and receive antennas, and the number of pilot symbols, this paper ensures that the system operates below a peak AoI violation probability. However, these studies do not impose a penalty upon exceeding a threshold.

[0004] Because transmission failures may occur during transmission, some literature considers a transmission source measuring a time-varying variable and periodically reporting the measurement information to a monitoring station. Available communication channels are limited, and the transmission source must use one of these channels each time it sends measurement information. The monitoring station does not know the success probabilities of these channels. This article models this as a multi-arm bandit problem. Through simulation, it demonstrates that AoI-Aware measurement is superior to existing AoI strategies. However, the literature does not consider the differences in the probability of transmission failure across different channels.

[0005] Using reinforcement learning to optimize AoI has a very good effect. Some literature considers that the system is a server serving multiple streams, and uses policy gradient and deep Q-learning methods to learn the scheduling strategy of the intelligent agent and compares it with the benchmark method. However, the policy gradient algorithm usually converges to the global optimum rather than the local optimum, and the deep Q-learning algorithm is difficult to use in high-dimensional or continuous action spaces. The Actor-Critic algorithm combines the advantages of the policy gradient algorithm and the deep Q-learning algorithm. Some literature uses the Actor-Critic algorithm. The article studies industrial IoT devices that focus on latency. The emerging network function virtualization allows service providers to use virtual network functions (VNFs). Virtual Network Functions (VNFs) deliver a given network service. However, the appropriate placement and scheduling of VNFs in these scenarios is NP-hard, and finding the global optimal solution using traditional methods is extremely complex. Deep reinforcement learning (RL) offers a promising approach to this problem. This paper first uses a low-complexity single-agent RL actor-critic algorithm to jointly optimize VNF cost and AoI. To address the capacity limitations of single-agent learning, this paper uses a multi-agent RL algorithm where agents can cooperate with each other. The paper demonstrates that the single-agent RL algorithm outperforms the greedy algorithm in optimizing average network cost and AoI. Furthermore, the multi-agent RL algorithm reduces average overhead by separating the tasks of the agents. However, due to the need for cooperation between agents, the multi-agent RL algorithm requires more iterations. Some literature compares RL algorithms with classical algorithms to demonstrate their superiority. Another paper proposes a scenario where multiple sensor nodes communicate with a remote control center. This paper aims to optimize the AoI of all sensor nodes and achieve ultra-reliable low latency communication (URLLC). The paper compares the deep RL algorithm with the two classical algorithms. However, the above literature does not take the energy issue into consideration. Generally speaking, data transmission consumes energy, and for a many-to-one system, the energy consumed by different devices during transmission also varies.

[0006] In summary, the models mentioned in the above literature still have the following problems:

[0007] (1) When optimizing AoI, the situation of exceeding a certain threshold is not considered, or after considering the threshold, the penalty value that should be given after exceeding the threshold is not considered. When the AoI exceeds a certain value, it means that the transmission strategy is very poor and needs to be avoided as much as possible;

[0008] (2) When considering data transmission on different channels, the problem of transmission failure is taken into account, but the differences in the probability of transmission failure between different channels are ignored;

[0009] (3) Only considering the optimization of AoI, the energy consumption of each transmission is not considered. In fact, sensor nodes have a fixed amount of energy, and each transmission consumes a certain amount of energy. After the energy is consumed, the sensor node needs to be replaced, which incurs a certain cost.

[0010] (4) The actual conditions of different sensor nodes are different, and the energy consumed in each transmission and the size of the data packet generated are also different. This situation is not taken into account when designing many models. Summary of the Invention

[0011] Technical Problem: The technical problem to be solved by the present invention is to propose a research method for sensor data uploading based on reinforcement learning. Age of Information (AoI) and energy consumption are used as measurement indicators. Each sensor node monitors the information of the corresponding physical process and transmits it to the central controller. The physical process can be regarded as a series of tasks. Due to limited wireless resources, only one sensor node can transmit after monitoring each task. Therefore, the reinforcement learning Actor-Critic algorithm is used to decide which sensor nodes can transmit. The Actor-Critic algorithm can enable the scheduling node to select the nodes that need to transmit for transmission each time, reducing the complexity of calculation time and improving the efficiency of sensor node transmission.

[0012] Technical solution: The technical solution adopted by the present invention is as follows:

[0013] (1) First, establish the mathematical model corresponding to the wireless transmission problem of sensor nodes.

[0014] (2) Build a corresponding simulation environment based on the mathematical model of sensor node transmission and initialize various parameters used in the environment, including the AoI of sensor node k, the maximum energy that sensor node k can store, the energy consumed by sensor node k transmission, and the data packet size L transmitted by sensor node k. k , the rate λ(i) at which sensor node k transmits data packets.

[0015] (3) On the scheduling node side, it is first converted into a Markov decision process (MDP) according to the characteristics of the problem being studied: (S, A, R), where S represents the state space, A represents the action space, and R represents the reward function.

[0016] (4) Finally, the actor-critic algorithm is used to train the agent to make a decision on the transmission scheme that satisfies all constraints and minimizes the AoI and energy consumption.

[0017] The step (1) is specifically as follows:

[0018] (11) Considering a network composed of K sensor nodes and a scheduling node, the scheduling node selects a sensor node to generate a new data packet and transmit it to the central controller. The monitored physical process can be regarded as a series of tasks, and the sequence of tasks is represented by i∈{1, 2,..., T}. The scheduling strategy is that when monitoring the ith task, the scheduling node selects a sensor node k∈{1, 2,..., K} to transmit a new data packet at a rate of λ(i),

[0019] (12) The AoI is introduced to measure the timeliness of information transmission. In terms of the scheduling node, the AoI is defined as the number of time slots elapsed since the last successful reception of the latest data packet. A maximum value β k can be set to limit the upper limit value of the AoI, and a penalty of δ k is given when the AoI exceeds the maximum value β k .

[0020] (13) It is assumed that the scheduling node side has a fixed energy source, and each sensor node has a battery that can store a maximum of B joules. When the battery is used up, the corresponding sensor node is replaced, and the power of the sensor becomes full, which costs c. The energy consumed by the sensor node k to transmit the ith task is e k (i), and the power of the sensor node k before transmitting the ith task is E k (i).

[0021] (14) In terms of the scheduling node, the AoI of the kth sensor node at the scheduling node side can be represented as:

[0022]

[0023] The power of the selected sensor node k before transmitting the ith task is:

[0024]

[0025] The optimization objective is:

[0026]

[0027] The step (3) is specifically as follows:

[0028] (31) First, the scheduling nodes are regarded as intelligent agents and numbered as {1, 2, ..., K}, where K represents the total number of sensor nodes.

[0029] (32) Setting of state space S: After monitoring the i-th task, its state space S i Defined as: S i =(A k (i),E k (i),λ(i),L k ) K , where A s represents the AoI of sensor node k, E k (i) represents the power consumption of sensor node k before transmitting the i-th task, λ(i) represents the size of the data packet transmitted by sensor node k, L k represents the size of the data packet transmitted on behalf of sensor node k.

[0030] (33) Setting of action space A: After monitoring the i-th task, its action space A i Can be expressed as a row vector form: A i ={0,1} K Where 1 and 0 represent whether the scheduling node selects a sensor node for transmission, and K is the number of sensor nodes. For each task i, only one sensor node can be selected for transmission, so 1 is selected once and 0 is selected K-1 times.

[0031] (34) Setting of reward function R: After monitoring the i-th task, the agent’s reward function R i It is defined as the AoI on the scheduling node side, the penalty value given after exceeding the AoI threshold, the energy consumption of sensor node transmission, and the negative value of the cost of transmitting node replacement.

[0032] The step (4) is specifically as follows:

[0033] (41) First, initialize the Actor network parameters θ of each agent in the Actor-Critic algorithm a And the critic network parameter θ c .

[0034] (42) The agent observes the current state of the environment and moves from the action space A according to the current strategy. i Select an action a.

[0035] (43) The joint action acts on the current state and updates the current state from s to s′, while obtaining the immediate reward R.

[0036] (44) Afterwards, the reward will be given to the agent as feedback to update its policy neural network parameters θa and the value neural network parameters c .

[0037] (45) repeatedly steps (41-44) until the reward R obtained converges, the whole process is shown as Figure 1 .

[0038] Beneficial effects: the application proposes a sensor data uploading research method based on reinforcement learning. Age of Information (AoI) and energy consumption are used as measurement indicators. Each sensor node monitors information of the corresponding physical process and transmits it to the central controller. The physical process can be regarded as a series of tasks. Due to limited wireless resources, only one sensor node can transmit after monitoring each task. Therefore, the reinforcement learning Actor-Critic algorithm is used to decide which sensor node can transmit. The Actor-Critic algorithm can make the scheduling node select the node that needs to be transmitted each time, reduce the time complexity of calculation, and improve the transmission efficiency of the sensor node. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 Schematic diagram for training the reinforcement learning algorithm. DETAILED DESCRIPTION

[0040] The technical solutions of the application will be described in detail below, but the protection scope of the application is not limited to the described embodiments.

[0041] The application proposes a sensor data uploading research method based on reinforcement learning.

[0042] An embodiment is given below:

[0043] (1) First, a mathematical model corresponding to the wireless transmission problem of the sensor node is established.

[0044] Consider a network composed of K sensor nodes and a scheduling node. The scheduling node selects a sensor node to generate a new data packet and transmits it to the central controller. The monitored physical process can be regarded as a series of tasks, represented by i∈{1,2,...,T}. The scheduling strategy is to select a sensor node k∈{1,2,,...,K} to transmit a new data packet at a rate of λ(i) when monitoring the ith task.

[0045] When monitoring the ith task, use I k (i)=1 to represent that the sensor node k is selected to transmit, and I k(i) = 0 means that sensor node k is not selected to transmit. The size of the data packet transmitted by different sensor nodes is different, and L k represents the size of the data packet transmitted by node k, and the probability that the data packet is successfully received at the scheduling node is p k ∈(0,1], use r k (i) represents the number of attempts made by node k to successfully transmit the data packet. AoI is introduced to measure the timeliness of information transmission. The AoI of the kth sensor node on the scheduling node side can be expressed as:

[0046]

[0047] λ t (i) represents the transmission rate at the tth transmission. k When (i) = 1, that is, when the scheduling node selects sensor node k in monitoring task i, the AoI of the sensor node on the scheduling node side is the time it takes to transmit the data packet. k When (i) = 0, that is, when other nodes u≠k are selected to transmit task i, for task i+1, the AoI of sensor node k on the scheduling node side is the AoI of task i plus the time it takes for sensor node u to transmit the data packet.

[0048] Assume that each sensor node has a battery with a battery capacity of B. When the battery runs out, the corresponding sensor node is replaced, and then the battery of the sensor becomes fully charged. The energy consumed by sensor node k to transmit the i-th task is e k (i), the power consumption of sensor node k before transmitting the i-th task is The power consumption of the sensor node k selected for transmission before transmitting the i+1th task is:

[0049]

[0050] The optimization goal is to minimize the AoI on the scheduling node side, the penalty value given for exceeding the AoI threshold, the energy consumption of transmission, and the cost of replacing the sensor node. The optimization objective function can be expressed as:

[0051]

[0052] in:

[0053]

[0054]

[0055] The first item represents AoI, the second item represents the penalty value when exceeding the AoI threshold, β kRepresents the AoI upper limit of sensor node k. When this upper limit is exceeded, δ k The third term represents the energy consumed by transmitting data, the fourth term represents the cost of replacing the sensor node when the energy of the sensor node is exhausted, and c is the cost required for each replacement of the sensor node. After monitoring each task, Equation (4) means that only one sensor node can transmit, and Equation (5) means that the sensor node either chooses to transmit or not.

[0056] (2) The key to solving the above problem is to jointly optimize AoI and energy consumption, while considering factors such as AoI exceeding the threshold and replacement when the sensor node is exhausted. Since the AoI and energy consumption at each moment are only related to the previous moment, the problem can be converted into a discrete-time Markov process and solved using reinforcement learning methods.

[0057] The present invention defines the state space, action space, and reward function. In the framework of reinforcement learning, the agent is a scheduling node that takes an action at every moment and receives a corresponding reward from the environment, thereby moving to the next state.

[0058] 1. State Space

[0059] After monitoring the i-th task, the state space S of the agent i Defined as:

[0060]

[0061] Among them A k (i) represents the AoI of sensor node k after transmitting the i-th task, represents the remaining power of sensor node k before transmitting the i-th task, L k represents the size of the data packet transmitted by sensor node k, λ(i) represents the transmission rate after monitoring the i-th task, and K is the number of sensor nodes.

[0062] 2. Action Space

[0063] After monitoring the i-th task, the action space A of the agent i Defined as:

[0064] A i ={0,1} K (7)

[0065] Where 1 and 0 represent whether the scheduling node selects a sensor node for transmission, and K is the number of sensor nodes. For each task i, only one sensor node can be selected for transmission, so 1 is selected once and 0 is selected K-1 times.

[0066] 3. Reward Function

[0067] After monitoring the i-th task, the agent's reward function R i Defined as:

[0068]

[0069] The first item represents the sum of the AoIs of all sensor nodes after the completion of task i. The second item represents the penalty received when the AoI of a sensor node exceeds a certain threshold. The third item represents the energy consumed by the selected sensor node to transmit data. The fourth item represents the cost of replacing the sensor node when the battery energy is exhausted. c represents the cost of replacing each sensor node. Every action of the intelligent agent needs to take these four factors into consideration and find a balance among them. Therefore, the reward function reflects the performance of each action.

[0070] (3) This paper uses reinforcement learning to jointly optimize AoI and energy consumption. Due to the large state space and action space in the problem, classic reinforcement learning algorithms such as Q-learning can no longer meet the requirements of the problem. Therefore, the Actor-Critic algorithm under the deep reinforcement learning framework is used to solve the problem.

[0071] When using the Actor-Critic algorithm to train the sensor node transmission model, first initialize the actor network parameters θ a and critic network parameters θ c , and then perform each round of training. In each round of training, first reset the gradient dθ a =0,dθ c = 0, and from the state space Select a state s0 to initialize the state of the agent. When monitoring each task, the agent will start from the action space A i ={0,1} K Select action a i , that is, choosing which sensor node to transmit, obtain rewards, move to the next state, and update the parameters of the actor network and critic network.

[0072] The network diagram of the Actor-Critic algorithm based on sensor data upload is as follows Figure 1 As shown. The strategy of the agent is determined by the neural network. The agent observes the state of the environment, including the throughput of the past j tasks, the AoI of all sensors, that is, A k (i) The size of the transmitted data packet L k (i) and the current remaining energy These values ​​are then fed into a neural network, which then outputs an action. This action determines which sensor node the scheduling node will select for transmission in the next task. The reward is then fed back to the agent, which uses this reward information to train and improve its neural network model. Aside from the different output dimensions, the actor and critic networks share the same structure.

[0073] The simulation parameters for training the agent using the reinforcement learning Actor-Critic algorithm are set as follows:

[0074] Consider a system consisting of a scheduler node and ten sensor nodes. There are a total of 100 tasks, the data packet transmission rate of the sensor node is 10 bytes / s, and the data packet size generated by the sensor node is 10kbytes, that is, from 10 bytes to 100 bytes, the first node is 10 bytes, and each subsequent node is 10 bytes more than the previous node. The AoI threshold of each sensor node is different. The AoI threshold of the first node is 20s, and the AoI threshold of each subsequent node is 20s more than the previous one. The penalty value of the node with a low AoI threshold will be greater. In terms of transmission energy consumption, the battery capacity of the sensor node is 30 Joules, but the energy consumed by different sensors to transmit data once is also different. When a sensor node runs out of power, the node needs to be replaced, which costs 20. After the replacement, the battery power returns to the maximum power.

Claims

1. A sensor network data uploading method based on Actor-Critic, wherein the sensor network includes K sensor nodes and a scheduling node, the sensor nodes monitor tasks, and the scheduling node selects one sensor node for transmission each time and transmits the obtained data packet to a central controller, characterized in that: The steps include: Establishing a sensor node transmission mathematical model corresponding to the sensor node wireless transmission problem; the optimization goal of the sensor node transmission mathematical model is to minimize the sum of the following items: the AoI on the scheduling node side, the penalty value imposed after exceeding the AoI threshold, the energy consumption of the sensor node transmission, and the cost of replacing the transmission node; According to the mathematical model of sensor node transmission, the corresponding simulation environment is built, and the relevant parameters in the simulation environment are initialized, including the AoI of sensor node k, the maximum energy that sensor node k can store, and the energy e consumed by sensor node k to transmit the i-th task. k (i) The size of the data packet transmitted by sensor node k is L k , the rate λ(i) at which sensor node k transmits data packets; k=1,…,K; The sensor node scheduling problem of the scheduling node is transformed into a Markov decision process (S, A, R), where S represents the state space, A represents the action space, and R represents the reward function. The reinforcement learning Actor-Critic algorithm is used to train the intelligent agent to decide on the data upload scheme that satisfies various constraints and minimizes the AoI and energy consumption.

2. The Actor-Critic based sensor network data uploading method according to claim 1, characterized in that: On the scheduling node side, the AoI of the kth sensor node on the scheduling node side is expressed as: The power consumption of the sensor node k selected for transmission before transmitting the i+1th task is: The optimization goal is: Among them, r k (i) represents the number of attempts made by node k to successfully transmit the data packet; when monitoring the i-th task, I k (i) = 1 means that sensor node k is selected to transmit, and I k (i) = 0 means that sensor node k is not selected to transmit; L k represents the size of the data packet transmitted by node k; is the power of sensor node k before transmitting the i-th task; B is the battery power of the sensor node; e k (i) is the energy consumed by sensor node k to transmit the i-th task; u≠k; c is the cost required for each replacement of the sensor node; β k is the upper limit of AoI of sensor node k; δ k is the penalty given when the AoI of sensor node k exceeds the upper limit.

3. The Actor-Critic based sensor network data uploading method according to claim 1, characterized in that: Setting of state space S: After the sensor node monitors the i-th task, the state space S of the agent i Defined as: S i =(A k (i),E k (i),λ(i),L k ) K , where A k represents the AoI of sensor node k, E k (i) represents the power consumption of sensor node k before transmitting the i-th task, λ(i) represents the size of the data packet transmitted by sensor node k, L k represents the size of the data packet transmitted on behalf of sensor node k; Setting of action space A: After the sensor node monitors the i-th task, the action space A of the agent i Expressed as a row vector: A i ={0,1} K , where 1 and 0 represent whether the scheduling node selects a sensor node for transmission, and K is the number of sensor nodes; for each task i, only one sensor node can be selected for transmission, so 1 is selected once and 0 is selected K-1 times; Setting of reward function R: After the sensor node monitors the i-th task, the agent's reward function R i It is defined as the AoI on the scheduling node side, the penalty value given after exceeding the AoI threshold, the energy consumption of sensor node transmission, and the negative value of the cost of replacing the transmission node.

4. The Actor-Critic based sensor network data uploading method according to claim 1, characterized in that: Training an agent using the reinforcement learning Actor-Critic algorithm involves the following steps: First, initialize the Actor network parameters θ of each agent in the Actor-Critic algorithm a And the critic network parameter θ c ; The agent observes the current state of the environment and moves from the action space A according to the current strategy. i Select an action a; The joint action acts on the current state and updates the current state from s to s′, while obtaining the immediate reward R; The reward will be given to the agent as feedback to update its policy neural network parameters θ a and the value neural network parameter θ c ; Repeat the above steps until the reward R converges.

Images