A method for state updating based on relevant information age in an energy harvesting Internet of Things

Abstract

This invention discloses a state update method based on relevant information age in an energy harvesting Internet of Things (IoT), comprising the following steps: establishing an evolution model of the network, energy consumption, and relevant information age in the energy harvesting IoT; based on the established IoT model, proposing a sensor state update optimization problem with the objective of minimizing the average relevant information age of the data fusion center; modeling the state update optimization problem as a partially observable Markov decision process with unknown sensor power; combining a flexible action commentator algorithm and a long short-term memory network, and proposing an action decomposition and mapping mechanism, designing a deep reinforcement learning algorithm to solve the established partially observable Markov decision process. The designed algorithm can effectively address the problems caused by unknown environmental dynamics, unobservable sensor power, and a large-scale discrete action space, and achieves the minimization of the average relevant information age of the data fusion center.

Description

Technical Field

[0001] This invention relates to the field of sensor state update technology, and in particular to a state update method based on the age of relevant information in an energy harvesting Internet of Things. Background Technology

[0002] In recent years, the rapid development of wireless communication and artificial intelligence technologies has driven the deployment and application of various Internet of Things (IoT) real-time application systems (such as monitoring systems and status update systems). In IoT systems designed for real-time applications, outdated information can degrade the accuracy and reliability of derived decisions. To further measure the impact of dynamic data updates on information timeliness, i.e., information freshness, the concept of Information Age (AoI) has been proposed: the AoI of a system at any given time is the time elapsed since the latest information at the destination was generated.

[0003] The limited energy of sensors in the Internet of Things (IoT) significantly restricts the improvement of information freshness. Traditional sensors mainly rely on batteries as energy buffers, but the limited battery capacity restricts their lifespan. Furthermore, when sensor nodes are deployed in remote, rugged, and harsh environments (such as battlefields, primeval forests, and volcanoes), the difficulty in replacing batteries hinders the widespread adoption of battery-powered sensors, thus impeding the development of the IoT. In recent years, the development of energy harvesting (EH) technology has enabled energy conservation and emission reduction in traditional networks, providing a solution for extending the lifespan of energy-constrained networks. EH technology is a new power supply method that converts surrounding natural energy sources (such as solar, tidal, and wind power) into usable electrical energy, making node energy replenishment more convenient and environmentally friendly. Utilizing EH technology can effectively extend sensor operating time while saving significant manpower and resources.

[0004] In many current practical IoT applications, the state updates of different sensors are correlated. For such applications, the concept of Related Information Age (AoCI) has been proposed to characterize the freshness of integrated information from multiple sensors. Although the dynamic data update process based on related data AoCI faces more limitations and difficulties compared to optimization based on independent data AoI, the increasing demand for integrated multi-sensor information and its timeliness in practical applications makes the research on AoCI optimization methods of significant practical importance in the development of large-scale real-time IoT. On the other hand, introducing signaling interaction in large-scale applications can cause network congestion and latency, which is detrimental to information age optimization. Furthermore, frequent sensor scheduling consumes a large amount of battery energy, shortening its lifespan and indirectly affecting information age optimization, thus significantly reducing network performance. Additionally, in basic IoT composed of energy-harvesting sensors (referred to as energy-harvesting IoT), the sensor energy harvesting process is usually affected by the environment, exhibiting strong randomness. Moreover, sensors are subject to energy causality constraints, and data fusion nodes may not be able to obtain real-time information on the available power of sensors. This greatly increases the control difficulty of data fusion nodes, posing a significant challenge to optimizing information freshness. For such complex dynamic systems, traditional optimization methods (such as heuristic algorithms, dynamic programming algorithms, etc.) are difficult to implement. Therefore, this invention proposes a state update method based on relevant information age in the Internet of Things for energy harvesting to solve the problems existing in the prior art. Summary of the Invention

[0005] To address the aforementioned problems, the present invention aims to propose a state update method based on the age of relevant information in energy harvesting IoT systems. Specifically, for energy harvesting IoT systems, this invention studies an information freshness optimization method based on data relevance: A system model of the energy harvesting IoT system is modeled, and a sensor state update optimization problem is proposed with the objective of minimizing the average age of relevant information in the data fusion center. This state update optimization problem is modeled as a partially observable Markov decision process where sensor battery level is unknown. A flexible action commentator algorithm and a long short-term memory network are combined, and an action decomposition and mapping mechanism is proposed. A deep reinforcement learning algorithm is designed to solve the established partially observable Markov decision process, overcoming the difficulties of unknown environmental dynamics, unobservable sensor battery level, and a large-scale discrete action space.

[0006] To achieve the objectives of this invention, the present invention is implemented through the following technical solution: a method for updating the state based on relevant information age in an energy harvesting Internet of Things, comprising the following steps:

[0007] Step 1: Based on an Internet of Things (IoT) network consisting of N sensors with energy harvesting devices, K related sensing points, and a data fusion center, establish an energy harvesting IoT model, including a network model, an energy consumption model, and an information related age evolution model.

[0008] Step 2: Based on the IoT model established in Step 1, the state of the corresponding sensors is updated by activating them in order to minimize the age of relevant information at the data fusion center. A sensor state update optimization problem is proposed with the goal of minimizing the average age of relevant information at the data fusion center.

[0009] Step 3: Based on the sensor state update optimization problem proposed in Step 2, the dynamic state update of the sensor with unknown power is modeled as a partially observable Markov decision process, which consists of tuples including state space, observation space, action space and reward function.

[0010] Step 4: Use an LSTM neural network to predict the actual battery level of the sensor, and propose an action decomposition and mapping mechanism to improve the SAC algorithm. Design a deep reinforcement learning algorithm to solve the partially observable Markov decision process established in Step 3, and complete the solution of the sensor state update optimization problem.

[0011] A further improvement is made in the following: In step one, the network model represents the sensor set as N = {1, 2, ..., N}, and the relevant sensing point set as K = {1, 2, ..., K}. Each sensor n ∈ N can only sense and transmit the state information of a specific relevant sensing point k ∈ K. When a sensor is activated to observe the CSP, a state update data packet with a timestamp is generated. The update data packet is transmitted to the data fusion center for further processing. Let N... k Let the set of sensors used to observe CSPk be denoted as the sum of all sensors. Then we have:

[0012]

[0013] Let A(t) = (A1(t), A2(t), ..., A K (t) represents the state update decision of the data fusion center at the beginning of time slot t, where k∈K, and A n (t) = {0, 1}, k∈K, representing the state update decision of sensor n in time slot t, that is, when sensor n is scheduled in time slot t, A n (t) = 1, otherwise it is 0.

[0014] A further improvement is made in the energy consumption model in step one, where a sensor, after being scheduled by the data fusion center, will only be activated to sense the CSP and transmit corresponding data when the power supply is sufficient. n (t) represents the battery energy of sensor n in time slot t, using This indicates whether sensor n is activated in time slot t and has sufficient energy to perform a state update; if so, Otherwise, it is 0. Therefore, the constraint of energy causality is expressed as:

[0015]

[0016] in This indicates that sensor n is performing sensing and transmission actions when no energy arrives. The remaining power after that, therefore and A n The relationship between (t) can be expressed as:

[0017]

[0018] That is, when a sensor is asked to update the state of a CSP, it will only respond to the data fusion center's scheduling if it has sufficient energy.

[0019] A further improvement is made in the following: In step one, the relevant information age evolution model uses the relevant information age to evaluate the freshness of the integrated information at the data fusion center, denoted as Δ(t), where Y... n (t)∈{0,1}, k∈K, when Y n When (t) = 1, it indicates that sensor n successfully transmitted its state update to the data fusion center at time t; otherwise, it does not. This represents the importance threshold for updating CSPk, i.e. in The aggregation of CSPk data packets received at the data fusion center at the end of time t represents the importance of the data packets. Y(t)∈{0,1} indicates whether the required integration information was successfully generated at the data fusion center at the end of time t. When Y(t)=1, it means that the data was successfully generated; otherwise, it was not.

[0020]

[0021] Among them, Y k (t) = 1 indicates that CSPk was successfully sensed and transmitted data to the data fusion center at time t, and vice versa. The evolution process of AoCI can be represented as:

[0022]

[0023] Where Y(t-1) = 1 indicates that the integrated information was successfully generated at the end of the t-1 time period. Without loss of generality, Δ(0) is initialized to 0.

[0024] A further improvement is made in step three, where, in the state space, it is assumed that the state of each sensor n at time t is S. n (t)=(G n (t),X n (t),e n (t)), where:

[0025] G n (t)=t-1-T n (t)

[0026] T n (t)=max{t0|Y n (t0)=1,t0<t-1}

[0027]

[0028] Among them G n (t) represents the time interval from the last successful transmission by sensor n to the start of time t, X n (t) represents the sensor n at time G. n The number of times e is scheduled during period (t). n (t) represents the battery level of the sensor at time t.

[0029] A further improvement lies in the following: In step three, in the observation space, the agent cannot obtain the true state S(t) in time slot t, but instead receives an observation O(t) = (O1(t), O2(t), ..., O N (t)), where This represents the battery state of sensor n as observed by the data fusion center.

[0030] A further improvement is made in step three, where, in each time slot, after executing action A(t) in state S(t), the reward function is defined as follows:

[0031] U(S(t),A(t))=-Δ(t+1)

[0032] That is, the AoCI is negative at the beginning of each time slot, and the utility of an action can only be evaluated after a time slot has ended.

[0033] A further improvement lies in: in step four, the SAN network π in the SAC... φ Its parameter φ can be trained by minimizing the expected KL divergence, as shown in the following equation:

[0034]

[0035] Where α represents the temperature parameter, the reparameterization technique is applied to make the variable A(t) differentiable, and the strategy is reparameterized using a Gaussian distribution, i.e.:

[0036] A(t) = f φ (τ,Z(t))=μ φ (Z(t))+σ φ (Z(t))τ

[0037] Where the condition variable μ φ and σ φ Let τ be the output of the SAN network, and τ be a noise variable that follows a standard Gaussian distribution with an expected value of 0 and a standard deviation of 1. Therefore:

[0038]

[0039] A further improvement lies in the following: In step four, the action decomposition and mapping mechanism specifically involves first reconstructing an action space consisting of K subspaces. Right now Each subspace The expression is as follows:

[0040]

[0041] in Represents the zero vector, the original action space yes A subset of the set, proposing a mapping function M: It deterministically assigns each original action Mapped to The effective action A in the example is shown below:

[0042]

[0043] By incorporating the ADM mechanism into the SAC framework, a vector with K components, i.e., the original action, is generated using SAN. Each component belongs to a subspace. primitive action After passing through the mapping function M, the effective action A can be obtained.

[0044] The beneficial effects of this invention are as follows: This invention establishes an evolutionary model of network, energy consumption, and related information age in an energy harvesting Internet of Things (IoT); based on the established IoT model, it proposes a sensor state update optimization problem with the objective of minimizing the average related information age of the data fusion center; it models the state update optimization problem as a partially observable Markov decision process where sensor power is unknown; combining a flexible action commentator algorithm and a long short-term memory network, and proposing an action decomposition and mapping mechanism, it designs a deep reinforcement learning algorithm to solve the established partially observable Markov decision process. The designed algorithm can effectively address the problems caused by unknown environmental dynamics, unobservable sensor power, and a large-scale discrete action space, and achieves the minimization of the average related information age of the data fusion center. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0047] Figure 2 This is a schematic diagram of an Internet of Things network with an EH sensor in an embodiment of the present invention;

[0048] Figure 3 This is a schematic diagram of the RSS algorithm in an embodiment of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] See Figure 1 , Figure 2 , Figure 3 This embodiment provides a method for updating the state of an energy harvesting Internet of Things (IoT) based on relevant information and age, including the following steps:

[0051] Step 1: Based on an Internet of Things (IoT) network consisting of N sensors with energy harvesting devices, K related sensing points, and a data fusion center (DFC), establish an energy harvesting IoT model including a network model, an energy consumption model, and an related information age (AoCI) evolution model.

[0052] In the network model, the sensor set is represented as N = {1,2,...,N}, and the related sensing point set is represented as K = {1,2,...,K}. Each sensor n ∈ N can only sense and transmit the state information of a specific related sensing point k ∈ K. When a sensor is activated to observe a related sensing point (CSP), a state update data packet with a timestamp is generated. The update data packet is sent to the DFC for further processing. This model is used in applications such as real-time security monitoring or environmental monitoring systems, where different CSPs are associated with different areas. After obtaining the state of all CSPs with the same timestamp, the DFC is responsible for generating valuable integrated information.

[0053] Let N k Let the set of sensors used to observe CSPk be denoted as the sum of all sensors. Then we have:

[0054]

[0055] Let A(t) = (A1(t), A2(t), ..., A K (t) represents the state update decision of DFC at the beginning of time slot t, where k∈K, and A n (t) = {0, 1}, k∈K, representing the state update decision of sensor n in time slot t, that is, when sensor n is scheduled in time slot t, A n (t) = 1, otherwise it is 0;

[0056] Suppose that the transmission failure probability of a sensor n is independent and identically distributed across different time slots, denoted as p. n The specific details regarding DFC and sensors are unknown. Furthermore, due to limitations in transmission resources, this embodiment assumes that for each group of sensors N... k , In the middle, it is allowed to not exceed M (M≤|N) k Each sensor simultaneously sends its updated data packets, thus avoiding collisions on orthogonal channels. Therefore:

[0057]

[0058] In step one, within the energy consumption model, once a sensor is scheduled by DFC, it will only be activated to sense the CSP and transmit corresponding data when the battery is sufficiently charged. n(t) represents the battery energy of sensor n in time slot t, using This indicates whether sensor n is activated in time slot t and has sufficient energy to perform a state update; if so, Otherwise, it is 0. Therefore, the constraint of energy causality is expressed as:

[0059]

[0060] in This indicates that sensor n is performing sensing and transmission actions when no energy arrives. The remaining power after that, therefore and A n The relationship between (t) can be expressed as:

[0061]

[0062] That is, when a sensor is asked to update the state of a CSP, it will only respond to the DFC's scheduling if it has sufficient energy.

[0063] e n (t) represents the remaining battery charge of sensor n at the start of time slot t, and its constraint can be expressed as: Where E n Indicates the battery capacity of sensor n, o n (t-1)∈{0,1} represents the energy collected by sensor n at time t-1. The energy collection process is a random arrival process, when o n When (t-1) = 1, it means that sensor n has collected one unit of energy at time t-1; conversely, when o n When (t-1) = 0, it means that no energy arrives;

[0064] In the relevant information age evolution model, at the end of time slot t, only when the quality of the transmitted update packets is good enough for all CSPs, i.e., the importance threshold is met, can the required integrated information be generated at DFC. The freshness of the integrated information at DFC is evaluated by the relevant information age, denoted as Δ(t). Let Y n (t)∈{0,1}, k∈K, when Y n When (t) = 1, it indicates that sensor n successfully transmitted information to DFC at time t; otherwise, it does not. This represents the importance threshold for updating CSPk, i.e. in The aggregation of CSPk data packets received at the DFC at the end of time t represents the importance of the data packets. Y(t)∈{0,1} indicates whether the required integration information was successfully generated at the DFC at the end of time t. When Y(t)=1, it means that the data was successfully generated; otherwise, it was not.

[0065]

[0066] Among them, Y k (t) = 1 indicates that CSPk was successfully sensed and transmitted data to DFC at time t, and vice versa. The evolution of AoCI can be represented as:

[0067]

[0068] Where Y(t-1) = 1 indicates that the integrated information was successfully generated at the end of the t-1 time period. Without loss of generality, Δ(0) is initialized to 0.

[0069] As can be seen from the above, only by requiring a suitable sensor to observe all CSPs simultaneously within any given time period can the AoCI at DFC be reduced. Therefore, a feasible update decision, A(t), should further satisfy the following constraints:

[0070]

[0071] Where 1(·) is an indicator function (its value is 1 if the condition in parentheses is true, and 0 otherwise), and It is the aggregation function of importance, corresponding to the update decision A(t) made by DFC;

[0072] Step 2: Based on the IoT model established in Step 1, the state is updated by activating the corresponding sensors to minimize the relevant information age at the DFC, and a sensor state update optimization problem with the objective of minimizing the average relevant information age at the DFC is proposed.

[0073] Step 3: Based on the sensor state update optimization problem proposed in Step 2, the dynamic state update of the sensor with unknown power is modeled as a partially observable Markov decision process (POMDP), which consists of tuples including state space, observation space, action space and reward function.

[0074] In the state space, assume that the state of each sensor n at time t is S. n (t)=(G n (t),X n (t),e n (t)), where:

[0075] G n (t)=t-1-T n (t) (7)

[0076] T n (t)=max{t0|Y n(t0)=1,t0<t-1} (8)

[0077]

[0078] Among them G n (t) represents the time interval from the last successful transmission by sensor n to the start of time t, X n (t) represents the sensor n at time G. n The number of times e is scheduled during period (t). n (t) represents the battery level of the sensor at time t.

[0079] The state of the POMDP model is defined as a combination of all sensor states and AoCI, i.e., S(t) = (S1(t), S2(t), ..., S... N (t), Δ(t)), and in addition, to ensure the finiteness of the state space, let G n The maximum value of (t) is G. max X n The maximum value of (t) is X. max e n The maximum value of (t) is E n The maximum value of Δ(t) is Δ max These values ​​are all finite, but can be arbitrarily large;

[0080] In step three, within the observation space, at time slot t, because the actual battery level of the sensor is unknown, the agent cannot obtain the true state S(t). Correspondingly, it will receive observations O(t) = (O1(t), O2(t), ..., O...). N (t)), where This represents the battery state of sensor n as observed by DFC. Note that sensor n only successfully transmitted a state update, i.e., Y, in the previous time slot t-1. n (t-1) = 1, then, Only then will it be updated to Otherwise, DFC cannot obtain the sensor's current energy state, in which case, Set to a constant value E O To distinguish it from other valid battery status information, therefore, there is It should be noted that in each time slot t, the observation O(t) is determined by the tuple (S(t-1), A(t-1), S(t)) and the unknown environmental dynamics. This embodiment uses... To represent all possible observation spaces;

[0081] At the beginning of each time slot, the DFC scheduling part of the sensors senses and transmits state updates; therefore, the action space can be represented as follows:

[0082]

[0083] In step three, within the reward function, in each time slot, after executing action A(t) in state S(t), the reward function is defined as follows:

[0084] U(S(t),A(t))=-Δ(t+1)

[0085] That is, the AoCI is negative at the beginning of each time slot, and the utility of an action can only be evaluated after a time slot has ended;

[0086] Step 4: Use an LSTM neural network to predict the actual battery level of the sensor, and propose an action decomposition and mapping mechanism to improve the Flexible Action Critic (SAC) algorithm. Design a deep reinforcement learning algorithm to solve the partially observable Markov decision process established in Step 3, and complete the solution of the sensor state update optimization problem.

[0087] The architecture of the RSS algorithm proposed in this embodiment is as follows: Figure 3 As shown, based on the standard SAC algorithm, an artificial neural network (ANN) with parameters φ is introduced to approximate the learning of policy π, called a soft actor network (SAN). φ (Z(t)) represents that, simultaneously, two independent artificial neural networks are used to approximate the soft-value function to mitigate the positive bias in the policy improvement process. These are parameterized by parameters θ1 and θ2, respectively. and This indicates that the two ANN networks are named Soft Critics Network (SCN)1 and 2, respectively, and they are trained by minimizing the soft Bellman residual, as shown in the following formula:

[0088]

[0089] in, To represent the target SCN (TSCN), this embodiment uses two independent TSCNs (referred to as TSCN1 and TSCN2) to mitigate the positive bias, which are respectively represented by... and Parameterization, Decomposition and This means that the parameters of the two SCNs (θ1 and θ2) can be updated using stochastic gradient descent (SGD), and their gradients can be calculated using backpropagation. Meanwhile, the parameters of the TSCNs are updated using exponential averaging.

[0090] For SAN networks in SAC π φ Its parameter φ can be trained by minimizing the expected KL divergence, as shown in the following equation:

[0091]

[0092] Where α represents the temperature parameter. However, π φ Training cannot be performed via backpropagation because Q... θ The variable A(t) in (Z(t),A(t)) is the policy network π. φ The sampled gradient cannot be backpropagated in the normal manner. To address this issue, a reparameterization technique can be applied to make the variable A(t) differentiable. For simplicity, this strategy is reparameterized using a Gaussian distribution, i.e.:

[0093] A(t) = f φ (τ,Z(t))=μ φ (Z(t))+σ φ (Z(t))τ

[0094] Where the condition variable μ φ and σ φ Let τ be the output of the SAN network, and τ be a noise variable that follows a standard Gaussian distribution with an expected value of 0 and a standard deviation of 1. Therefore:

[0095]

[0096] The Action Decomposition and Mapping (ADM) mechanism works as follows: First, an action space consisting of K subspaces is reconstructed. Right now Each subspace The expression is as follows:

[0097]

[0098] in Represents the zero vector, the original action space yes A subset of the set, proposing a mapping function M: It deterministically assigns each original action Mapped to The effective action A in the example is shown below:

[0099]

[0100] By incorporating the ADM mechanism into the SAC framework, a vector with K components, i.e., the original action, is generated using SAN. Each component belongs to a subspace. primitive action After passing through the mapping function M, the effective action A can be obtained;

[0101] In addition to the ADM mechanism, this embodiment further modifies the standard SAC network structure, such as... Figure 3 As shown, replacing the second fully connected layer (FCL) with an LSTM layer allows the neural network to effectively learn and extract key information from history, which is crucial for solving the POMDP problem. In this embodiment, the SCN, TSCN, and SAN that combine LSTM are referred to as Recurrent SCN (RSCN), Recurrent TSCN (RTSCN), and Recurrent SAN (RSAN), respectively. They all consist of one LSTM layer and two FCL layers.

[0102] At the start of the RSS algorithm, the experience pool is first cleared. and temporary experience pool The parameters φ, θ1, and θ2 of RSAN and RSCN are randomly initialized, and the parameters of RTSCN are set to... and The learning process consists of W rounds, where in each round the initial observations are initialized to O(1), where e n (t)=E n And all other elements are set to 0. Additionally, a temporary experience pool... This is used to store data for an entire round and is initialized to empty. Then, in time slot t, the original actions are first generated according to the reparameter formula. Secondly, through discretization and mapping relationships... Given a valid action A(t), after executing A(t), the environment will enter the next state S(t+1), the agent's observations are O(t+1), and the reward is U(t). This is the current temporary experience pool. Will be updated to: exist In this embodiment, four data points related to the t-th (t≥1) iteration are stored, referred to as an empirical tuple. This indicates that at the end of each round, a temporary experience pool... All experience tuples were transferred to the experience pool. It is prepared for use during training.

[0103] In W s After each round, training begins, starting with the experience pool. Randomly select W b Then, from each selected round, L consecutive experience tuples are randomly drawn. For a given round w, if the initial experience tuple is consistent with the t-th round... w Since each iteration is relevant, the set of empirical tuples sampled in round w is denoted as: in,

[0104]

[0105] For a sampled empirical tuple This embodiment uses two RTSCNs networks to calculate the corresponding target soft action value. As shown in the following formula:

[0106]

[0107] in, This indicates that in the current training, in RSAN, i.e., π φ Input observations in the network The original action is then output. Then, the sampled empirical tuples and target action values ​​are used. Calculate the Bellman residuals of the two RSCN networks:

[0108]

[0109] Accordingly, the parameters of the two RSCNs, i.e. It can be updated using stochastic gradient descent (SGD), as shown below:

[0110]

[0111] Where, η j This represents the learning rate.

[0112] In addition, SGD is used to update the RSAN network parameters π. φ As shown below:

[0113]

[0114] Where η0 represents the update parameter π φ The learning rate The expectation of the KL divergence is expressed as follows:

[0115]

[0116] in, Represents the network RSANπ φ In the input The original action obtained later, and the last term on the right side of the formula. It can be represented as:

[0117]

[0118] When the parameters of two RSCN networks are updated, the parameters of their corresponding target networks are updated as follows:

[0119]

[0120] Here, 0 < τ ≤ 1 is a constant that ensures the target network parameters track the RSCN while maintaining a smooth change.

[0121] This embodiment addresses the situation where the DFC (Data Fusion Center) cannot obtain the real-time energy of the sensors. It considers an IoT system consisting of multiple Related Target Sensing Points (CSPs), multiple Energy Harvesting (EH) sensors, and a Data Fusion Center (DFC). Using the relevant information age (AoCI) as the optimization metric, the dynamic data update problem of the sensors in the IoT system is modeled as a sensor state update optimization problem with the objective of minimizing the average relevant information age of the data fusion center. The state update optimization problem is modeled as a partially observable Markov decision process (POMDP) ​​where the sensor power is unknown. Combining the flexible action commentator algorithm and long short-term memory network, and proposing an action decomposition and mapping mechanism, a deep reinforcement learning algorithm is designed to solve the established partially observable Markov decision process.

[0122] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for status update based on relevant information age in energy harvesting IoT, characterized in that, The method comprises the following steps: Step one: based on an Internet of Things network composed of N sensors with energy collection devices, K related sensing points and a data fusion center, an energy collection Internet of Things model is established, including network model, energy consumption model and related information age evolution model; Step two: based on the Internet of Things model established in step one, the state of the corresponding sensor is updated to minimize the related information age at the data fusion center, and a sensor state update optimization problem is proposed to minimize the average related information age at the data fusion center; Step three: based on the sensor state update optimization problem proposed in step two, the dynamic state update of the sensor power-agnostic is modeled as a partially observable Markov decision process, which is composed of a tuple including state space, observation space, action space and reward function; Step four: an LSTM neural network is used to predict the real power of the sensor, and an action decomposition and mapping mechanism is proposed to improve the SAC algorithm, and a deep reinforcement learning algorithm is designed to solve the partially observable Markov decision process established in step three, and the sensor state update optimization problem is solved. SAN network in SAC , parameters It can be trained by minimizing the expected KL divergence as follows: ； where denotes the temperature parameter, reparameterization techniques are applied to make the variable differentiable, the variable is reparameterized with a Gaussian distribution, i.e.: ； where the conditional variables and are outputs of the SAN network, is a noise variable that follows a standard Gaussian distribution with mean 0 and standard deviation 1, so that ； The action decomposition and mapping mechanism is as follows: first, reconstruct an action space composed of K subspaces , i.e. , wherein each subspace is expressed as follows: ； wherein represents a zero vector, the original action space is a subset of the original action space, a mapping function is proposed which deterministically maps each original action to a valid action A in as follows: ； By incorporating the ADM mechanism into the SAC framework, a vector with K components, i.e., the original action , is generated by the SAN , , the original action is mapped through a mapping function M to obtain the effective action A.

2. The method of claim 1, wherein: In step one, in the network model, the sensor set and is represented as , the relevant set of perception points and is represented as , and each sensor can only perceive and transmit the state information of a certain relevant perception point , and when the sensor is activated to observe the CSP, a state update data packet with a timestamp is generated, and the update data packet is transmitted to the data fusion center for further processing, where represents the sensor set used to observe the CSP k , then ； set up This indicates that the data fusion center is in the time slot. The initial state update decision, where ,and , indicating in time slot sensor The state update decision, i.e., when the sensor In the time slot When scheduled Conversely, it is 0.

3. The method of claim 1, wherein: In step one, within the energy consumption model, once a sensor is scheduled by the data fusion center, it will only be activated to sense the CSP and transmit corresponding data when it has sufficient power. Indicates sensor In the time slot Battery energy, using To represent the sensor Is it in a time slot? If it is activated and has sufficient energy to perform a state update, then... Otherwise, it is 0. Therefore, the constraint of energy causality is expressed as: ； wherein indicates that when no energy arrives, the sensor In performing the sensing transmission action The relationship between the remaining power after the sensing transmission action and may be expressed as: ； That is, when a sensor is required to update the state of a CSP, it will only respond to the scheduling of the data fusion center when it has enough energy.

4. The method of claim 1, wherein: In step one, the relevant information age evolution model uses the relevant information age to assess the freshness of the integrated information at the data fusion center, denoted as... ,set up ,when When, it indicates the sensor exist Successful transmission of status updates to the data fusion center at all times, and vice versa. CSP k The importance threshold for updates, i.e. ,in represent At the end of the time, the data fusion center received information about CSP. k Data packet importance aggregation, using To indicate in At the end of the time step, has the required integration information been successfully generated at the data fusion center? A '0' indicates a successful generation, otherwise it indicates no generation. ； wherein, CSP is represented k In The evolution of AoCI can be represented as follows: ； wherein indicates that at t - the integrated information has been successfully generated at the end of the time period, without loss of generality, is initialized to 0.

5. The method of claim 1, wherein: In step three, in the state space, assume that each sensor At the time instant is where: ； ； ； in Indicates sensor Since the last successful transmission The time interval before the start of the moment, Indicates sensor In time The number of times it was scheduled during the period, For the sensor in Battery level at the start of the moment.

6. The method of claim 1, wherein: In step three, in the observation space, the agent observes at time slot The true state cannot be obtained Instead, an observation is received where , represents the battery state of the sensor observed by the data fusion center.

7. The method of claim 1, wherein: In step three, the reward function is defined as: where the action is performed after the state That is, the negative value of AoCI at the beginning of each time slot, and only after the end of a time slot, the utility of an action can be evaluated.

Images