Intelligent Internet of Things data transmission method based on deep reinforcement learning

Abstract

The present invention relates to the technical field of data transmission, and discloses a method for data transmission of an intelligent Internet of Things based on deep reinforcement learning, including: S1, obtaining a sensor network, and dividing the working time of the entire sensor network into an initial stage and a plurality of continuous superframe periods; S2, finding the optimal K value based on the intra-cluster sum of squares and the inter-cluster sum of squares in the initial stage; S3, establishing a cluster head selection model based on deep reinforcement learning; S4, finding the optimal cluster head nodes corresponding to K clusters in each superframe period through the cluster head selection model, and inputting the reward information obtained after transmitting the data in each superframe period and the corresponding experience information obtained when selecting the optimal cluster head node into the cluster head selection model for training. The present invention adopts an improved K-means clustering algorithm, which effectively solves the limitations of traditional K-means in processing discrete data by considering the characteristics of discrete data and the similarity of samples within the cluster.

Description

Technical Field

[0001] The present invention relates to the technical field of data transmission, and in particular to an intelligent Internet of Things data transmission method based on deep reinforcement learning. Background Art

[0002] With the continuous development of Internet of Things (IoT) and Wireless Sensor Network (WSN), sensor networks are widely used in various fields, from environmental monitoring to industrial automation. In wireless sensor networks, node clustering and cluster head selection are key issues. Traditional clustering methods such as K-means algorithm have some limitations in efficiency and performance, and cluster head selection methods also need better algorithms that adapt to dynamic network environments.

[0003] Existing technical solutions have some limitations in clustering and cluster head selection in wireless sensor networks. The traditional K-means algorithm may not perform well for non-convex clusters and is difficult to adapt to various complex network topologies. In addition, the existing cluster head selection method may lack comprehensive consideration of real-time and network life optimization when considering dynamic network conditions. Therefore, a new clustering method and cluster head selection method are needed to improve the efficiency, performance and adaptability of the network. Summary of the invention

[0004] The present invention provides an intelligent Internet of Things data transmission method based on deep reinforcement learning, and a clustering method based on an improved K-means to cope with the shortcomings of traditional methods in processing non-convex clusters. At the same time, the present invention also proposes an intelligent cluster head selection method based on Soft Actor-Critic for Discrete Action Space (SACD) to optimize cluster head allocation in a dynamic network environment. The introduction of SACD makes cluster head selection more flexible and real-time, and effectively balances the relationship between network life and data transmission performance. Through the improvement of K-means and the application of SACD, the technical solution of the present invention effectively optimizes cluster head selection while improving clustering accuracy, so that the wireless sensor network in a dynamic network environment can better adapt to changing conditions, and improves the reliability and practicality of the entire network.

[0005] The present invention is achieved through the following technical solutions:

[0006] An intelligent Internet of Things data transmission method based on deep reinforcement learning, comprising:

[0007] S1, obtaining a sensor network, and dividing the working time of the entire sensor network into an initial stage and a plurality of continuous superframe periods;

[0008] S2, in the initial stage, based on the intra-cluster sum of squares and the inter-cluster sum of squares, find the optimal K value, and re-divide the entire sensor network into K clusters according to the optimal K value and the K-means algorithm;

[0009] S3. Establish a cluster head selection model based on deep reinforcement learning;

[0010] S4, finding the optimal cluster head nodes corresponding to the K clusters in each superframe period respectively through the cluster head selection model, and performing data transmission based on the optimal cluster head nodes respectively, and inputting the reward information obtained after the data is transmitted in each superframe period and the corresponding state information, action information and next state information obtained when the optimal cluster head node is selected into the cluster head selection model for training until the cluster head selection model converges;

[0011] S5. In each superframe period of the subsequent data transmission process, the trained cluster head selection model is used to select the optimal cluster head nodes of the K clusters, and the K clusters perform data transmission through the optimal cluster head nodes;

[0012] S6. When there is a change in the nodes in the sensor network, repeat S1-S5 until the energy of the sensor network is exhausted.

[0013] As an optimization, the superframe period includes a control period and a data transmission period.

[0014] As an optimization, the specific steps of S2 are:

[0015] S2.1, predefine the value range of K, K∈[1,N], N is a positive integer not equal to 1;

[0016] S2.2, for each K value within the preset range, the K-Means algorithm is used to divide the entire sensor network into K clusters;

[0017] S2.3. Calculate the intra-cluster sum of squares WSS for each of the K clusters. k , total sum of squares TSS and between-cluster sum of squares BSS K ;

[0018] S2.4. Find the K value that maximizes Ratio as the optimal K value, where: BSS k =TSS-WSS k ;

[0019] S2.5. Based on the obtained optimal K value, the entire sensor network is re-divided into K clusters based on the K-means algorithm.

[0020] As an optimization, the intra-cluster sum of squares WSSk The specific expression is:

[0021]

[0022] Where K is the number of clusters, n i is the number of nodes in the ith cluster, x i,j is the location information of the jth node in the ith cluster, c i is the center position information of the i-th cluster;

[0023] The specific expression of the total square sum TSS is:

[0024]

[0025] Where K is the total number of clusters, It is the central location information of the entire sensor network.

[0026] As an optimization, the cluster head selection model adopts the SACD model, which includes an actor network Actor and a critic network Critic.

[0027] As an optimization, in S4, the optimal cluster head nodes corresponding to the K clusters are found in each superframe period through the cluster head selection model, and data transmission is performed based on the optimal cluster head nodes respectively, and the reward information obtained after the data is transmitted in each superframe period and the corresponding state information, action information and next state information obtained when the optimal cluster head node is selected are input into the cluster head selection model for training. The specific steps are:

[0028] S4.1. In the control cycle, for the sensor network, the global agent selects m nodes with the largest residual energy in the K clusters as candidate cluster head nodes, and abstracts the measurement information of the candidate cluster head nodes into a two-dimensional image quantity as a state set and inputs it into the actor network Actor of the SACD model. The state set is two-dimensional state information.

[0029] S4.2, the actor network Actor obtains the optimal action information according to the state set, and the node corresponding to the optimal action information is the optimal cluster head node in the cluster;

[0030] S4.3, during the data transmission period, the nodes in the sensor network perform channel access and data transmission based on a specific channel access protocol according to the assigned optimal cluster head node;

[0031] S4.4. After each data transmission cycle, the agent calculates the average energy consumption and data transmission reliability of the nodes in the cluster according to the data transmission results in each cluster as the reward information of the SACD model:

[0032]

[0033] In the formula, s i Represents the status information of the i-th cluster, a i Indicates that the status information s i The action information generated by the Actor network of the SACD model is input, PDR i represents the average data packet delivery rate of all nodes in the ith cluster in the last data transmission cycle, Ec i represents the average energy consumption of all nodes in the i-th cluster in the last data transmission cycle;

[0034] S4.5. The experience information obtained after receiving each superframe period is stored in the experience pool D, and mini-batch data is extracted from the experience pool D each time to train the SACD model until the SACD model converges.

[0035] As an optimization, the empirical information is expressed as e=(s i ,a i ,r i ,s i _), where s i _ indicates that the i-th cluster is in state s i Execute action a i The latest status information of the post-environment, r i =R(s i ,a i ).

[0036] As an optimization, the measurement information includes the average distance between all nodes in the same cluster except the candidate cluster head node and the candidate cluster head node, the average RSSI value between all nodes in the same cluster except the candidate cluster head node and the candidate cluster head node, and the residual energy information of the candidate cluster head node. The expression of the two-dimensional image quantity is:

[0037]

[0038] s represents the two-dimensional image quantity, that is, the state set of candidate cluster head nodes, RSSI 1 The RSSI information of the candidate cluster head node with the largest residual energy is represented by the RSSI information of all nodes in the cluster except the candidate cluster head node and the RSSI of the candidate cluster head node; D 1 Represents the distance information of the candidate cluster head node with the largest residual energy. The distance information here refers to the average distance from all nodes in the cluster except the candidate cluster head node to the candidate cluster head node; RE 1Indicates the residual energy information of the candidate cluster head node with the largest residual energy. In the same cluster, the RSSI information, average distance, and residual energy information with subscripts 1 to m decrease in sequence.

[0039] As an optimization, the specific steps of training the SACD model are:

[0040] A1. Initialize the parameters of the actor network Actor and the critic network CRITIC, and set the experience replay buffer pool D;

[0041] A2. Use the current actor network Actor to interact with the environment, collect some experience information, and store the obtained experience information in the experience playback pool D. For state s i , the corresponding action a i It can be calculated by the following formula:

[0042]

[0043] In the formula, Represents the actor network Actor, Parameters of the actor network Actor;

[0044] s i _~p(s i _|s i ,a i );

[0045] In the formula, p(s i _|s i ,a i ) indicates that in the status information s i Next, perform action a i After that, the environment state transfers to s i The probability density function of _;

[0046] D←D∪{(s i ,a i ,r i ,s i _)};

[0047] In the formula, (s i ,a i ,r i ,s i _) Experience information generated by the interaction between the agent and the environment;

[0048] A3, randomly sample the experience information of mini-batch from the experience replay pool D, calculate the target Q value, and use this experience information to train the Q-network and Value network in the critic network Critic, let θ1 and θ 2 Represent the network parameters of Q-network and Value network respectively, then the network parameter update formula of Q-network and Value network can be expressed as:

[0049]

[0050]

[0051] In the formula, J(θ 1 ) and J(θ 2 ) are the loss functions of Q-network and Value network respectively; and are the policy gradients of Q-network and Value network, respectively, Q is the learning rate of Q-network and Value network;

[0052] A4. Use the negative gradient of the critic network Critic to update the actor network Actor parameters to improve the actor network strategy;

[0053] A5. During each update, the update of the target critic network Critic is smoothed by moving the parameters of the target critic network Critic toward the parameters of the current network by a certain proportion, thereby adopting a soft update strategy to update the parameters of the target critic network Critic. The target critic network Critic includes the Target Q-Network and the Target Value Network, specifically:

[0054] make and Represent the parameters of Target Q-Network and Target Value Network respectively, then the parameter update formula of TargetQ-Network and Target Value Network can be expressed as:

[0055]

[0056]

[0057] In the formula, τ is the weight factor, which is used to control the update speed;

[0058] A6. Adjust the temperature parameter α in the SACD model:

[0059] α=max(α min,α-decay·step);

[0060] In the formula, α min is the minimum value of the temperature parameter, which is used to limit the lower limit of the temperature parameter. decay is a decay coefficient, which indicates the decay of each step of the temperature parameter. step indicates the number of training rounds.

[0061] A7. Repeat A2 to A6 until the set number of training rounds is reached or the stopping condition is met.

[0062] As an optimization, the update formula of the actor network Actor using the negative gradient of the critic network Critic in A4 can be expressed as:

[0063]

[0064] In the formula, λ π is the learning rate of the Actor network. is the objective function of the Actor network, is the policy gradient of the Actor network.

[0065] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0066] Efficient clustering method: The present invention is an intelligent Internet of Things data transmission method based on deep reinforcement learning, and proposes a clustering method based on improved K-means. The improved K-means clustering algorithm is adopted. By considering the characteristics of discrete data and the similarity of samples within the cluster, the limitations of traditional K-means in processing discrete data are effectively solved. The clustering method can not only adapt to different types of data, but also improve the accuracy of clustering while maintaining high efficiency, and can effectively assign similar data points to the same cluster. Such a clustering method is efficient and helps to improve the efficiency of data processing and analysis.

[0067] Intelligent cluster head selection strategy: The present invention introduces a cluster head selection method based on SACD, adopts the Soft Actor-Critic for Discrete Action Space (SACD) algorithm, and makes the allocation of cluster heads in real-time networks more intelligent by dynamically adjusting the selection of cluster heads. This reinforcement learning-based method can optimize the selection of cluster heads according to the dynamic changes and real-time needs of the network, and adaptively and dynamically select cluster heads through DRL. It has strong adaptability, improves network life, and enhances the performance of the entire network. It effectively combines the characteristics of deep reinforcement learning and the Internet of Things, and provides an innovative solution for the adaptability and intelligence of the network.

[0068] Comprehensive consideration of clustering and cluster head selection: The present invention proposes a new Internet of Things management method by organically combining K-means and SACD. This method not only considers discrete data in the clustering process, but also intelligently selects cluster heads through reinforcement learning, so that the entire network can achieve efficient and intelligent data transmission and management in different environments. This comprehensive consideration not only makes full use of the advantages of traditional clustering methods, but also flexibly selects cluster heads through reinforcement learning, making the entire system more intelligent and adaptable.

[0069] Improve network performance Through efficient clustering and optimized cluster head selection, the present invention is expected to significantly improve the performance of the Internet of Things. The clustering method reduces the complexity of data processing, while the SACD cluster head selection method optimizes the energy utilization efficiency of the network and comprehensively improves the performance indicators of the network, such as network life and data transmission efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] The drawings described herein are used to provide a further understanding of the embodiments of the present invention, constitute a part of this application, and do not constitute a limitation of the embodiments of the present invention. In the drawings:

[0071] Figure 1 A schematic diagram showing the structure of a superframe period in an embodiment of the present invention is shown;

[0072] Figure 2 A flow chart of a K-means algorithm in an embodiment of the present invention is shown;

[0073] Figure 3 The diagram shows the architecture of the SACD model in the embodiment of the present invention. DETAILED DESCRIPTION

[0074] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with embodiments and drawings. The exemplary embodiments of the present invention and their description are only used to explain the present invention and are not intended to limit the present invention.

[0075] This embodiment 1 provides an intelligent Internet of Things data transmission method based on deep reinforcement learning.

[0076] The entire time is divided into an initial phase and multiple consecutive superframe periods, such as Figure 1 In the initial stage, a range of K values ​​is predefined, and the K-Means algorithm is executed for each K value to cluster the nodes in the network (the flowchart of the K-Means algorithm is shown in Figure 2As shown); the optimal K value is calculated based on the clustering results, and then the final clustering is performed based on the optimal K value; each subsequent superframe period includes a control and a data transmission period. In the control period, the agent selects the optimal node as the cluster head node based on the state of each cluster and the Actor network; in the data transmission period, the node sends its data to the cluster head node in the cluster; the agent counts the average energy consumption and packet delivery rate of all nodes in each cluster in the superframe period and generates a corresponding reward value, and stores the experience information generated by the interaction with the environment in the experience pool D. During the SACD model training process, the agent only obtains experience data from the experience pool D to train the SACD model (the SACD model architecture diagram is shown in Figure 3 The specific implementation process is as follows:

[0077] S1, obtain the sensor network, and divide the working time of the entire sensor network into an initial stage and multiple continuous superframe periods; Figure 1 As shown, the superframe period includes a control period and a data transmission period.

[0078] S2, in the initial stage, based on the intra-cluster sum of squares and the inter-cluster sum of squares, find the optimal K value, and re-divide the entire sensor network into K clusters according to the optimal K value and the K-means algorithm;

[0079] According to the scale of the sensor network, a value range of K is predefined, that is, K∈[1,N]. The larger the scale of the sensor network, the larger the value of K, and therefore the larger the value range of K. According to the value range of K and the location characteristics of the nodes in the network, the entire network is divided into K clusters according to the K-means algorithm, and for each K∈[1,N], the corresponding within-cluster sum of squares (WSS), total sum of squares (TSS) and between-cluster sum of squares (BSS) are calculated to obtain an optimal K value;

[0080] The specific steps are:

[0081] S2.1, predefine the value range of K, K∈[1,N], N is a positive integer not equal to 1;

[0082] S2.2, for each K value within the preset range, the K-Means algorithm is used to divide the entire sensor network into K clusters;

[0083] S2.3. Calculate the intra-cluster sum of squares WSS for each of the K clusters. k , total sum of squares TSS and between-cluster sum of squares BSS K ;

[0084] S2.4. Find the K value that maximizes Ratio as the optimal K value, where: BSS k =TSS-WSS k ;

[0085] S2.5. Based on the obtained optimal K value, the entire sensor network is re-divided into K clusters based on the K-means algorithm.

[0086] For example, in the initial stage, we first predefine a range of K values, such as K∈[1,10]. Based on the value of K and the location characteristics of the nodes in the network, we use the K-means algorithm to divide the entire network into K clusters. Specifically, for each K∈[1,10], WSS K The calculation formula is as follows:

[0087]

[0088] Where K is the number of clusters, n i is the number of nodes in the ith cluster, x i,j is the location information of the jth node in the ith cluster, c i is the center position information of the i-th cluster, and the position information is the coordinate information.

[0089] Next, calculate the total sum of squares (TSS), which represents the sum of the square distances between all samples and the overall center point of the data set (global mean). TSS reflects the dispersion of the entire data set. The larger the TSS, the greater the difference between the data points. Specifically, the calculation formula of TSS is:

[0090]

[0091] Where K is the total number of clusters, It is the central location information of the entire sensor network.

[0092] Based on the values ​​of WSS and TSS, the Between-Cluster Sum of Squares (BSS) can be calculated. BSS represents the weighted sum of the square distances between the center of each cluster and the center of the entire data set. BSS measures the separation between clusters. The larger the BSS, the more significant the differences between clusters and the less similar the samples between clusters. The calculation formula of BSS is:

[0093] BSS k =TSS-WSS k ;

[0094] Where n iis the number of nodes in the ith cluster.

[0095] Finally, calculate the ratio of BSS to TSS:

[0096]

[0097] The optimal K value can be obtained by the following formula:

[0098] K = argmax K∈[1,N] Ratio K .

[0099] Next, the entire network is re-divided into K clusters using the calculated optimal K value and K-means algorithm. The workflow diagram of the K-means algorithm is as follows: Figure 2 As shown, this workflow is prior art and will not be described in detail here.

[0100] S3, establish a cluster head selection model based on deep reinforcement learning (DRL);

[0101] In this embodiment, the cluster head selection model adopts the SACD model, and the SACD model includes an actor network Actor and a critic network Critic.

[0102] S4, finding the optimal cluster head nodes corresponding to the K clusters in each superframe period respectively through the cluster head selection model, and performing data transmission based on the optimal cluster head nodes respectively, and inputting the reward information obtained after the data is transmitted in each superframe period and the corresponding state information, action information and next state information obtained when the optimal cluster head node is selected into the cluster head selection model for training until the cluster head selection model converges;

[0103] The specific steps of finding the optimal cluster head nodes corresponding to the K clusters in each superframe period through the cluster head selection model, and performing data transmission based on the optimal cluster head nodes, and inputting the reward information obtained after the data is transmitted in each superframe period and the corresponding state information, action information and next state information obtained when the optimal cluster head node is selected into the cluster head selection model for training are as follows:

[0104] S4.1. In the control cycle, for the sensor network, the global agent selects m nodes with the largest residual energy in the K clusters as candidate cluster head nodes, and abstracts the measurement information of the candidate cluster head nodes into a two-dimensional image quantity as a state set and inputs it into the actor network Actor of the SACD model. The state set is two-dimensional state information.

[0105] The measurement information includes the average distance between all nodes in the same cluster except the candidate cluster head node and the candidate cluster head node, the average RSSI value between all nodes in the same cluster except the candidate cluster head node and the candidate cluster head node, and the residual energy information of the candidate cluster head node. The expression of the two-dimensional image quantity is:

[0106]

[0107] s represents the two-dimensional image quantity, that is, the state set of candidate cluster head nodes, RSSI 1 The RSSI information of the candidate cluster head node with the largest residual energy is represented by the RSSI information of all nodes in the cluster except the candidate cluster head node and the RSSI of the candidate cluster head node; D 1 Represents the distance information of the candidate cluster head node with the largest residual energy. The distance information here refers to the average distance from all nodes in the cluster except the candidate cluster head node to the candidate cluster head node; RE 1 Indicates the residual energy information of the candidate cluster head node with the largest residual energy. In the same cluster, the RSSI information, average distance, and residual energy information with subscripts 1 to m decrease in sequence.

[0108] For example, in the control cycle, the agent selects m = 6 nodes with the largest residual energy in each cluster as candidate cluster head nodes, and abstracts the measurement information of these candidate nodes into a two-dimensional image quantity, which can be expressed as:

[0109]

[0110] S4.2, the actor network Actor obtains the optimal action information according to the state set, and the node corresponding to the optimal action information is the optimal cluster head node in the cluster;

[0111] The Actor network generates a probability distribution of actions based on the current state, and the action with the largest probability value is considered the optimal action. Specifically, the output of the Actor network is the probability distribution of m actions. The action with the largest probability value is the optimal action.

[0112] S4.3, in the data transmission period, the nodes in the sensor network perform channel access and data transmission based on a specific channel access protocol according to the assigned optimal cluster head node, and the cluster head node forwards the data to the base station;

[0113] Then, the agent inputs the state information s of each cluster into Figure 3In the Actor network of the SACD model shown in FIG, the optimal action a is obtained, where a∈[1,m]. Specifically, if a=1, it means that the candidate cluster head node corresponding to the first row in the state s will serve as the cluster head node of the cluster and forward data for all nodes in the cluster.

[0114] At the end of the data transmission cycle, the global agent counts the global network performance of the nodes in each cluster (i.e., the average node energy consumption and data transmission reliability), and maps these performance indicators to the reward value r for the agent to perform the action in this state, specifically:

[0115] S4.4. After each data transmission cycle, the agent calculates the average energy consumption and data transmission reliability of the nodes in the cluster according to the data transmission results in each cluster as the reward information of the SACD model:

[0116]

[0117] In the formula, s i Represents the status information of the i-th cluster, a i Indicates that the status information s i The action information generated by the Actor network of the SACD model is input, PDR i represents the average data packet delivery rate of all nodes in the ith cluster in the last data transmission cycle, Ec i represents the average energy consumption of all nodes in the i-th cluster in the last data transmission cycle; in addition, for a given state-action pair, that is, i ,a i >, reward r of environmental feedback i =R(s i ,a i );

[0118] S4.5. The experience information obtained after receiving each superframe period is stored in the experience pool D, and a mini-batch of data is extracted from the experience pool D each time to train the SACD model until the SACD model converges. The experience information is expressed as e=(s i ,a i ,r i ,s i _), where s i _ indicates that the i-th cluster is in state s i Execute action a i The latest state information of the environment, that is, the experience information collected by the agent in each cluster in this cycle, namely (s, a, r, s′), is stored in the experience pool D.

[0119] ​The agent stores the experience information generated by each cluster's interaction with the environment in the previous superframe period into the experience pool. Specifically, for the i-th cluster, its experience information can be expressed as e = (s i ,a i ,r i ,s i _), where s i _ indicates that the i-th cluster is in state s i Execute action a i After that, the latest status information of the environment.

[0120] In each model training cycle, a mini-batch of data is extracted from the experience pool D to train the SACD model. When the model converges, the model training task is paused.

[0121] The specific steps of training the SACD model are:

[0122] A1. Initialize the parameters of the actor network Actor (Actor Network) and the critic network CRITIC (CriticNetwork) (including θ 1 ,θ 2 , and ), and set up the experience replay buffer pool D;

[0123] A2. Use the current actor network Actor (strategy network) to interact with the environment, collect some experience information, including state, action, reward, and next state information, and store the obtained experience information in the experience playback pool D. For state s i , the corresponding action a i It can be calculated by the following formula:

[0124]

[0125] In the formula, Represents the actor network Actor, Parameters of the actor network Actor;

[0126] s i _~p(s i _|s i ,a i );

[0127] In the formula, p(s i _|s i ,a i ) indicates that in the status information s i Next, perform action a i After that, the environment state transfers to si The probability density function of _;

[0128] D←D∪{(s i ,a i ,r i ,s i _)};

[0129] In the formula, (s i ,a i ,r i ,s i _) Experience information generated by the interaction between the agent and the environment;

[0130] A3. Randomly sample the experience information of mini-batch from the experience replay pool D, calculate the target Q value (here Q(s,a) is the output value of the Critic network, input the state s into the critic network, the critic outputs the probability distribution of m actions, and the ath output is the value of Q(s,a)d, where a belongs to [1,m]), and use this experience information to train the Q-network and Value network in the critic network Critic; the calculation formula of the target Q value is:

[0131]

[0132] Where r is the reward value of the environment feedback when the agent performs action a in state s. γ∈(0,1] is the discount factor. s' is the new state of the environment when the agent performs action a in state s. The action a' here is based on the parameter The Critic network is selected, that is Specifically, θ 1 and θ 2 They are the parameters of Q-Network and ValueNetwork in the Critic network respectively.

[0133] The network parameter update formula of Q-network and Value network can be expressed as:

[0134]

[0135]

[0136] In the formula, J(θ 1 ) and J(θ 2 ) are the loss functions of Q-network and Value network respectively; and are the policy gradients of Q-network and Value network, respectively, Q is the learning rate of Q-network and Value network;

[0137] A4. Using the critic network Critic (i.e. ) updates the actor network Actor parameters to improve the actor network strategy, specifically:

[0138] In the formula, λ π is the learning rate of the Actor network. is the objective function of the Actor network, is the policy gradient of the Actor network;

[0139] The objective function of the Actor network is:

[0140]

[0141] Where α is the temperature coefficient, represents the strategy function, i.e., the Actor network; Q θ (s,a) is the state-action value function of the state-action pair (s,a), represents the state-action value function of the state-action pair (s, a) calculated by the Q-Network in the Critic network; Represents the state-action value function of the state-action pair (s, a) calculated by the Value Network in the Critic network;

[0142] A5. When performing each update, the update of the target actor network Actor and the target critic network Critic is smoothed by moving the parameters of the target critic network Critic toward the parameters of the current network in a certain proportion, so as to adopt a soft update strategy to update the parameters of the target actor network Actor and the target critic network Critic. In other words, the target actor network and the target critic network are updated by adopting a soft update strategy, that is, when performing each update, the update of the target network is smoothed by moving the parameters of the target network toward the parameters of the current network in a certain proportion. The target critic network Critic includes a Target Q-Network and a Target Value Network, specifically:

[0143] make and Represent the parameters of Target Q-Network and Target Value Network respectively, then the parameter update formula of TargetQ-Network and Target Value Network can be expressed as:

[0144]

[0145]

[0146] In the formula, τ is a weight factor used to control the update speed. τ can be set to 0.001 and can be adjusted dynamically according to the environment. and The initial values ​​of 1 and θ 2 same;

[0147] A6. Adjust the temperature parameter α in the SACD model:

[0148] α=max(α min ,α-decay·step);

[0149] In the formula, α min is the minimum value of the temperature parameter, which is used to limit the lower limit of the temperature parameter. decay is a decay coefficient, which indicates the decay of each step of the temperature parameter. step indicates the number of training rounds.

[0150] Specifically, the goal of the SACD model is to maximize the expected cumulative reward, which includes the reward from the environment and the reward from the policy entropy. Represents the uncertainty of the policy, i.e. the randomness of the generated actions. By adjusting the temperature coefficient, the impact of the entropy reward can be controlled to encourage more exploration.

[0151] Specifically, the objective function of the Actor network is:

[0152]

[0153] Where α is the temperature coefficient, represents the strategy function, i.e., the Actor network; Q θ (s,a) is the state-action value function of the state-action pair (s,a). represents the state-action value function of the state-action pair (s, a) calculated by the Q-Network in the Critic network; Represents the state-action value function of the state-action pair (s, a) calculated by the Value Network in the Critic network.

[0154] The temperature parameter is used to adjust the exploration and exploitation trade-off of the policy. In the SACD model, the temperature parameter is usually expressed as α, which is used to adjust the entropy of the policy. The higher the entropy of the policy, the more random the policy's choice of actions, which increases exploration. By adjusting the temperature parameter, you can balance exploration and exploitation and affect the certainty of the policy.

[0155] A7. Repeat A2 to A6 until the set number of training rounds is reached or the stopping condition is met (model convergence).

[0156] It is worth mentioning that A2 here includes the process in which the intelligent agent selects a cluster head node for each cluster in each control cycle, and the node performs corresponding data transmission and performance statistics according to the assigned cluster head node in the data transmission cycle, that is, the process of interacting with the environment.

[0157] S5. In each superframe period of the subsequent data transmission process, the trained cluster head selection model is used to select the optimal cluster head nodes of the K clusters, and the K clusters perform data transmission through the optimal cluster head nodes;

[0158] S6. When there is a change in the nodes in the sensor network, repeat S1-S5 until the energy of the sensor network is exhausted.

[0159] When there is no change in the nodes, the K value will not change, but the cluster head selection model will continue to train until the model converges.

[0160] When the cluster head selection model is being trained, if the node changes, it starts again from S1.

[0161] It should be noted that the advantages of the present invention are not only reflected in sensor networks, but can also be extended to various application scenarios that require data clustering and intelligent cluster head selection, including environmental monitoring, industrial automation and other fields, and has broad application prospects.

[0162] The specific implementation methods described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for intelligent Internet of Things data transmission based on deep reinforcement learning, characterized in that: include: S1, obtaining a sensor network, and dividing the working time of the entire sensor network into an initial stage and a plurality of continuous superframe periods; S2, in the initial stage, based on the intra-cluster sum of squares and the inter-cluster sum of squares, find the optimal K value, and re-divide the entire sensor network into K clusters according to the optimal K value and the K-means algorithm; The specific steps of S2 are: S2.1, predefine the value range of K, K∈[1,N], N is a positive integer not equal to 1; S2.2, for each K value within the preset range, the K-Means algorithm is used to divide the entire sensor network into K clusters; S2.

3. Calculate the intra-cluster sum of squares WSS for each of the K clusters. k , total sum of squares TSS and between-cluster sum of squares BSS K ; S2.

4. Find the K value that maximizes Ratio as the optimal K value, where: S2.5, based on the obtained optimal K value, the entire sensor network is re-divided into K clusters based on the K-means algorithm; S3. Establish a cluster head selection model based on deep reinforcement learning; S4, finding the optimal cluster head nodes corresponding to the K clusters in each superframe period respectively through the cluster head selection model, and performing data transmission based on the optimal cluster head nodes respectively, and inputting the reward information obtained after the data is transmitted in each superframe period and the corresponding state information, action information and next state information obtained when the optimal cluster head node is selected into the cluster head selection model for training until the cluster head selection model converges; S5. In each superframe period of the subsequent data transmission process, the trained cluster head selection model is used to select the optimal cluster head nodes of the K clusters, and the K clusters perform data transmission through the optimal cluster head nodes; S6. When there is a change in the nodes in the sensor network, repeat S1-S5 until the energy of the sensor network is exhausted.

2. According to claim 1, a method for intelligent Internet of Things data transmission based on deep reinforcement learning is characterized in that: The superframe period includes a control period and a data transmission period.

3. According to claim 1, a method for intelligent Internet of Things data transmission based on deep reinforcement learning is characterized in that: The within-cluster sum of squares WSS k The specific expression is: Where K is the number of clusters, n i is the number of nodes in the ith cluster, x i,j is the location information of the jth node in the ith cluster, c i is the center position information of the i-th cluster; The specific expression of the total square sum TSS is: Where K is the total number of clusters, It is the central location information of the entire sensor network.

4. According to claim 3, a method for transmitting data in an intelligent Internet of Things based on deep reinforcement learning is characterized in that: The cluster head selection model adopts the SACD model, and the SACD model includes an actor network Actor and a critic network CRITIC.

5. According to claim 4, a method for intelligent Internet of Things data transmission based on deep reinforcement learning is characterized in that: In S4, the optimal cluster head nodes corresponding to the K clusters are found in each superframe period by the cluster head selection model, and data is transmitted based on the optimal cluster head nodes respectively, and the reward information obtained after the data is transmitted in each superframe period and the corresponding state information, action information and next state information obtained when the optimal cluster head node is selected are input into the cluster head selection model for training. The specific steps are: S4.

1. In the control cycle, for the sensor network, the global agent selects m nodes with the largest residual energy in the K clusters as candidate cluster head nodes, and abstracts the measurement information of the candidate cluster head nodes into a two-dimensional image quantity as a state set and inputs it into the actor network Actor of the SACD model. The state set is two-dimensional state information. S4.2, the actor network Actor obtains the optimal action information according to the state set, and the node corresponding to the optimal action information is the optimal cluster head node in the cluster; S4.3, during the data transmission period, the nodes in the sensor network perform channel access and data transmission based on a specific channel access protocol according to the assigned optimal cluster head node; S4.

4. After each data transmission cycle, the agent calculates the average energy consumption and data transmission reliability of the nodes in the cluster according to the data transmission results in each cluster as the reward information of the SACD model: In the formula, s i Represents the status information of the i-th cluster, a i Indicates that the status information s i The action information generated by the Actor network of the SACD model is input, PDR i represents the average data packet delivery rate of all nodes in the ith cluster in the last data transmission cycle, Ec i represents the average energy consumption of all nodes in the i-th cluster in the last data transmission cycle; S4.

5. The experience information obtained after receiving each superframe period is stored in the experience pool D, and mini-batch data is extracted from the experience pool D each time to train the SACD model until the SACD model converges.

6. According to claim 5, a method for intelligent Internet of Things data transmission based on deep reinforcement learning is characterized in that: The empirical information is expressed as e=(s i ,a i ,r i ,s i _), where s i _ indicates that the i-th cluster is in state s i Execute action a i The latest status information of the post-environment, r i =R(s i ,a i ).

7. The method for transmitting data of intelligent Internet of Things based on deep reinforcement learning according to claim 6 is characterized in that: The measurement information includes the average distance between all nodes in the same cluster except the candidate cluster head node and the candidate cluster head node, the average RSSI value between all nodes in the same cluster except the candidate cluster head node and the candidate cluster head node, and the residual energy information of the candidate cluster head node. The expression of the two-dimensional image quantity is: s represents the two-dimensional image quantity, that is, the state set of candidate cluster head nodes, RSSI1 represents the RSSI information of the candidate cluster head node with the largest residual energy, and the RSSI information here represents the average RSSI between all nodes in the cluster except the candidate cluster head node and the candidate cluster head node; D1 represents the distance information of the candidate cluster head node with the largest residual energy, and the distance information here refers to the average distance from all nodes in the cluster except the candidate cluster head node to the candidate cluster head node; RE1 represents the residual energy information of the candidate cluster head node with the largest residual energy. In the same cluster, the RSSI information, average distance, and residual energy information with subscripts 1 to m decrease in sequence.

8. The method for transmitting data of intelligent Internet of Things based on deep reinforcement learning according to claim 7, characterized in that: The specific steps of training the SACD model are: A1. Initialize the parameters of the actor network Actor and the critic network Critic, and set the experience playback buffer pool D; A2. Use the current actor network Actor to interact with the environment, collect some experience information, and store the obtained experience information in the experience playback pool D. For state s i , the corresponding action a i It can be calculated by the following formula: In the formula, Represents the actor network Actor, Parameters of the actor network Actor; s i _~p(s i _|s i ,a i ); In the formula, p(s i _|s i ,a i ) indicates that in the status information s i Next, perform action a i After that, the environment state transfers to s i The probability density function of _; D←D∪{(s i ,a i ,r i ,s i _)}; In the formula, (s i ,a i ,r i ,s i _) Experience information generated by the interaction between the agent and the environment; A3. Randomly sample the experience information of mini-batch from the experience replay pool D, calculate the target Q value, and use the experience information to train the Q-network and Value network in the critic network Critic. Let θ1 and θ2 represent the network parameters of Q-network and Value network respectively. The network parameter update formula of Q-network and Value network can be expressed as: Where J(θ1) and J(θ2) are the loss functions of Q-network and Value network respectively; and are the policy gradients of Q-network and Value network, respectively, Q is the learning rate of Q-network and Value network; A4. Use the negative gradient of the critic network Critic to update the parameters of the actor network Actor to improve the strategy of the actor network; specifically, the objective function of the Actor network is: Where α is the temperature coefficient, represents the strategy function, i.e., the Actor network; Q θ (s,a) is the state-action value function of the state-action pair (s,a), represents the state-action value function of the state-action pair (s, a) calculated by the Q-Network in the Critic network; Represents the state-action value function of the state-action pair (s, a) calculated by the Value Network in the Critic network; A5. During each update, the update of the target critic network Critic is smoothed by moving the parameters of the target critic network Critic toward the parameters of the current network by a certain proportion, thereby adopting a soft update strategy to update the parameters of the target critic network Critic. The target critic network Critic includes the Target Q-Network and the Target Value Network, specifically: make and Represent the parameters of Target Q-Network and Target Value Network respectively, then the parameter update formula of Target Q-Network and Target Value Network can be expressed as: In the formula, τ is the weight factor, which is used to control the update speed; A6. Adjust the temperature parameter α in the SACD model: α=max(α min ,α-decay·step); In the formula, α min is the minimum value of the temperature parameter, which is used to limit the lower limit of the temperature parameter. decay is a decay coefficient, which indicates the decay of each step of the temperature parameter. step indicates the number of training rounds. A7. Repeat A2 to A6 until the set number of training rounds is reached or the stopping condition is met.

9. The method for transmitting data of intelligent Internet of Things based on deep reinforcement learning according to claim 1, characterized in that: The update formula of the actor network Actor in A4 using the negative gradient of the critic network Critic can be expressed as: In the formula, λ π is the learning rate of the Actor network, is the objective function of the Actor network, is the policy gradient of the Actor network.

Images