Internet of Things cooperative routing method based on multi-agent deep reinforcement learning

Abstract

The invention provides an Internet of Things cooperative routing method based on multi-agent deep reinforcement learning. The method comprises the following steps: constructing a wireless sensor network model; constructing an energy consumption model; establishing an optimization problem of network routing according to the constructed wireless sensor network model and the energy consumption model; modeling the optimization problem into a multi-agent reinforcement learning problem, and selecting an optimal routing path according to the multi-agent reinforcement learning problem; wherein in the multi-agent reinforcement learning problem, each sensor node is regarded as an independent agent, and each agent learns a strategy for optimizing the overall network performance according to a local state and a reward mechanism. According to the invention, the energy utilization efficiency of the routing protocol is improved, routing with balanced energy is realized, and the network life cycle is effectively prolonged.

Description

Technical Field

[0001] The present application relates to the field of Internet of Things communication technology, and in particular to an Internet of Things collaborative routing selection method based on multi-agent deep reinforcement learning. Background Art

[0002] In recent years, the Internet of Things (IoT) has developed rapidly in many application fields. IoT interconnects a large number of devices, sensors and systems to realize data collection, exchange and intelligent management. As an important part of IoT, wireless sensor networks (WSNs) are responsible for sensing and collecting data in the physical environment, providing real-time environmental information support for IoT, and then realizing intelligent and automated decision-making. They are widely used in industries, traffic management, smart cities, agricultural irrigation and other fields. WSNs are usually composed of a large number of low-power, widely distributed sensor nodes that can monitor environmental parameters and collect data, and transmit data to gateway nodes or edge devices through multi-hop communication. However, the resource limitations of wireless sensor nodes in terms of bandwidth, communication distance, energy and processing power have brought many challenges to the application of WSNs. Among them, limited energy supply is one of the main problems faced by WSNs. Sensor nodes rely on batteries for power supply, and these batteries are usually not rechargeable, and their limited service life directly affects the working cycle of sensor nodes. Therefore, energy efficiency is very important for WSNs.

[0003] In order to solve the problem of limited energy, a reinforcement learning based routing protocol (RLBR) is proposed in the prior art. RLBR designs four reward functions, combining path life estimation, node energy loss and weighted distance to the sink to make path selection more in line with the long-term energy requirements of the network. The EER-RL protocol is also proposed, which uses the Q learning algorithm in reinforcement learning to optimize the energy consumption in the IoT network and extend the network life by dynamically adjusting the routing strategy. EER-RL applies the Q value update mechanism in the three stages of network initialization, cluster formation and data transmission, so that the device selects the optimal route according to the remaining energy and the number of hops. Experimental results show that EER-RL is superior to traditional protocols in energy efficiency and network life. In addition, an energy-efficient cluster routing protocol EEM-CRP based on the dragonfly algorithm is proposed in the prior art to optimize energy consumption and extend network life in WSN. EEM-CRP includes two stages: optimal cluster head selection (OCHS) and optimal route selection (ORS). OCHS selects the optimal cluster head based on the density, residual energy and distance of nodes, while ORS uses random walk and Levy distribution to select the best route to cover a larger area and reduce energy consumption and network overhead. A novel distributed two-hop cluster routing protocol (D2CRP) is proposed in the prior art to improve the energy efficiency of wireless sensor networks (WSNs). D2CRP achieves energy balance by comprehensively considering the transmission distance and residual energy in cluster head election. Each cluster head uses a one-hop neighbor node for data relay, which reduces the transmission distance within the cluster. In addition, the chain routing method between cluster heads optimizes inter-cluster communication, gradually transmits data to the base station (BS), and effectively reduces the total energy consumption.

[0004] Although existing WSN routing optimization methods have achieved certain results in improving energy efficiency and network life, they also have shortcomings. Traditional routing protocols are insufficient in terms of energy utilization efficiency and often fail to fully consider the remaining energy of nodes, causing some nodes to quickly run out of energy. Solutions based on heuristic algorithms can improve energy utilization, but it is difficult to guarantee the optimality of the solution in large-scale networks and lack real-time performance. In addition, existing methods still have room for improvement in terms of the balance of energy consumption and the flexibility of routing strategies, making it difficult to effectively delay the early failure of nodes, thereby limiting the overall life cycle of the network.

[0005] Therefore, the technical problem that urgently needs to be solved is: how to improve the energy utilization efficiency of routing protocols, achieve energy-balanced routing, and effectively extend the network life cycle. Summary of the invention

[0006] The purpose of this application is to provide an IoT collaborative routing selection method based on multi-agent deep reinforcement learning, which improves the energy utilization efficiency of the routing protocol and realizes energy-balanced routing, effectively extending the network life cycle.

[0007] To achieve the above-mentioned purpose, the present application provides an Internet of Things collaborative routing selection method based on multi-agent deep reinforcement learning, the method comprising: constructing a wireless sensor network model; constructing an energy consumption model; establishing a network routing optimization problem based on the constructed wireless sensor network model and energy consumption model; modeling the optimization problem as a multi-agent reinforcement learning problem, and selecting the optimal routing path based on the multi-agent reinforcement learning problem; wherein each sensor node in the multi-agent reinforcement learning problem is regarded as an independent agent, and each agent learns a strategy to optimize the overall network performance based on the local state and reward mechanism.

[0008] As described above, the IoT collaborative routing selection method based on multi-agent deep reinforcement learning, wherein building a wireless sensor network model includes:

[0009] Construct a sink node and multiple sensor nodes in the wireless sensor network model;

[0010] A bidirectional transmission link is established for data transmission between two sensor nodes or between a sensor node and a sink node.

[0011] In the IoT collaborative routing selection method based on multi-agent deep reinforcement learning as described above, building an energy consumption model includes:

[0012] Based on the first-order wireless point model, a calculation formula for the energy consumed by a node to transmit K bits of data is constructed;

[0013] When the transmission distance between nodes exceeds the distance threshold, the energy consumed by the node to transmit K bits of data is calculated using multipath propagation according to the constructed calculation formula for the energy consumed by the node to transmit K bits of data;

[0014] The formula for calculating the energy consumed by transmitting K bits of data using multipath propagation is:

[0015] E TX (K,d)=KE elec +K∈ mp d 4 ;

[0016] Among them, E elec It is the energy consumed by the transmitting circuit to send one bit of data; mp is the energy coefficient of the multipath attenuation model; d is the transmission distance between nodes.

[0017] In the IoT collaborative routing selection method based on multi-agent deep reinforcement learning as described above, the energy consumed by a node receiving K bits of data is calculated as follows:

[0018] E RX (k) = kE elec ;

[0019] The energy consumed by a node to process K bits of data is calculated as:

[0020] E proc (k) = kE proc ;

[0021] Among them, E proc is the energy consumed to process one bit of data; k is the amount of bit data processed by the node.

[0022] As described above, the IoT collaborative routing selection method based on multi-agent deep reinforcement learning, wherein, according to the constructed wireless sensor network model and energy consumption model, the optimization problem of establishing network routing includes:

[0023] Pre-set constraints for wireless sensor network models;

[0024] Based on the constraints of the wireless sensor network model and the objectives of network routing, the optimization problem of network routing is established.

[0025] As described above, the IoT collaborative routing selection method based on multi-agent deep reinforcement learning, in which the network routing optimization problem is expressed as:

[0026] Y=min(∑ i,j E i,j ·x i,j +Var E );

[0027] Where Y represents the optimization problem of network routing; min() represents the minimum value; E i,j represents the energy consumption of data transmission between node i and node j; x i,j Represents the data transmission link between node i and node j in this round of routing; Var E Represents the variance of energy consumption.

[0028] As described above, the IoT collaborative routing selection method based on multi-agent deep reinforcement learning, wherein the optimization problem satisfies the following conditions:

[0029]

[0030] Among them, x i,j is the data transmission link between node i and node i in this round of routing, xi,j =1 means node i transmits a data packet to node j, x i,j =0 means that there is no data packet transmission between node i and node j. In one round of routing, node i only transmits data packets to one node; E i is the energy of node i; Var E is the variance of energy consumption; E th represents the energy threshold, d(i, Sink) represents the distance from node i to the SINK node. If node i transmits a data packet to node j, then node i selects a next-hop node j that is closer to the sink node to avoid forming a loop; d(j, Sink) represents the distance from node j to the SINK node.

[0031] In the IoT collaborative routing selection method based on multi-agent deep reinforcement learning as described above, the optimization problem is modeled as a multi-agent reinforcement learning problem, and the optimal routing path is selected according to the multi-agent reinforcement learning problem, including:

[0032] Model the optimization problem as a multi-agent reinforcement learning problem;

[0033] Based on the multi-agent reinforcement learning problem, the strategy is improved through iterative interaction and training learning to select the optimal routing path.

[0034] In the IoT collaborative routing selection method based on multi-agent deep reinforcement learning as described above, modeling the optimization problem as a multi-agent reinforcement learning problem includes:

[0035] Each sensor node is considered as an independent intelligent agent;

[0036] When the agent v i When a data packet needs to be transmitted, it will be transmitted according to the current state s i In the action space A i Select action a i , transmit the data packet to the next hop node;

[0037] Agent v i Select the forward action a from the available actions i As the next hop, in forwarding action a i Rewards received after

[0038] All agents calculate their rewards based on the same global reward function;

[0039] Among them, the calculation formula of the global reward function is:

[0040] R=∑ i r i -β·Var E ;

[0041] Among them, R represents the reward obtained by the agent; r i Represents the agent v i The reward function; β is the weight coefficient of energy balance; Var E represents the variance of the residual energy.

[0042] The IoT collaborative routing selection method based on multi-agent deep reinforcement learning as described above, wherein each agent has its own actor network, target actor network, critic network, and target critic network;

[0043] In the iterative interaction phase, each agent’s actor network calculates the probability of selecting each neighbor node based on local observations;

[0044] During the training and learning phase, batches of experience are drawn from the experience replay pool and the critic network is updated using the minimized loss function.

[0045] The beneficial effects achieved by this application are as follows:

[0046] (1) This application comprehensively considers the node energy consumption, node data transmission direction and the reward function of the remaining energy, realizes low-energy consumption and energy-balanced routing, avoids loop paths, and selects routing paths with low energy consumption and more balanced, thereby effectively extending the network life cycle and improving energy efficiency.

[0047] (2) This application effectively handles the problems of incomplete information and local observation between nodes, allowing each node to make decisions based on limited local information, thereby more realistically simulating the actual network environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For those skilled in the art, other drawings can also be obtained based on these drawings.

[0049] Figure 1 This is a flowchart of an Internet of Things collaborative routing selection method based on multi-agent deep reinforcement learning according to an embodiment of the present application.

[0050] Figure 2 A schematic diagram of a wireless sensor network model according to an embodiment of the present application.

[0051] Figure 3 A schematic diagram of a framework of an Internet of Things collaborative routing selection method based on multi-agent deep reinforcement learning according to an embodiment of the present application.

[0052] Figure 4This is a curve chart showing how the reward value changes with the number of training rounds during the routing selection method training of an embodiment of the present application.

[0053] Figure 5 This is a curve chart showing the change of energy consumption value with the number of training rounds during the training of the routing selection method according to an embodiment of the present application.

[0054] Figure 6 This is a curve chart showing how the arrival rate changes with the number of training rounds during the training process of the routing selection method according to an embodiment of the present application.

[0055] Figure 7 This is a curve diagram of the variation of energy consumption variance with the number of training rounds during the training process of the routing selection method in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The following is a clear and complete description of the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0057] like Figure 1 As shown, the present application provides an IoT collaborative routing selection method based on multi-agent deep reinforcement learning, the method comprising:

[0058] Step S1, constructing a wireless sensor network model.

[0059] Step S1 includes:

[0060] Step S110: construct a sink node and multiple sensor nodes in the wireless sensor network model.

[0061] like Figure 2 As shown, a WSN scenario including a sink node (SINK node) and multiple sensor nodes is constructed in the wireless sensor network model.

[0062] Among them, sensor nodes continuously collect data packets from the physical environment and transmit them to SINK nodes. SINK nodes are responsible for integrating, processing and uploading these data to the server. Due to limited power and processing capabilities, nodes can only transmit data packets within the transmission range R.

[0063] As a specific embodiment of the present invention, the wireless sensor network model is represented as an undirected graph

[0064] in, represents the wireless sensor network model; Represents a set of nodes, including a SINK node and a set of sensor nodes {n1, n2, ..., n N}. Represents node n i and node n j The bidirectional transmission link between them, the nodes of the bidirectional transmission link can transmit data packets to each other. In each round of transmission, all sensor nodes transmit data packets to the SINK node through a single hop or multi-hop method.

[0065] Step S120: establishing a bidirectional transmission link for data transmission between two sensor nodes or between a sensor node and a sink node.

[0066] As a specific embodiment of the present invention, the sensor node is usually composed of four main modules: a data processing module (including a microcontroller unit MCU), a communication module (integrated radio transceiver), a sensor / actuator module (including sensors and actuators), and a power module (including batteries). Sensor nodes need to consume energy to complete tasks such as perception, data processing and data transmission. Data processing, communication and sensing modules are usually the main sources of energy consumption. Therefore, reducing the energy consumption of these modules is a key goal to extend the service life of sensor nodes. Transmission energy consumption refers to the energy consumed by the node during data transmission. The transmission energy consumption of the present invention refers to the energy consumed by the node during data transmission. The factors affecting the energy consumption include: the size of the data packet and the transmission distance; the reception energy consumption of the present invention refers to the energy consumption of the node when receiving data, which is proportional to the size of the received data packet, but has nothing to do with the transmission distance. Processing energy consumption involves the energy consumption of the node during data processing, including the energy required for operations such as data compression, encryption and decryption.

[0067] Step S2, constructing an energy consumption model.

[0068] Step S2 includes:

[0069] Step S210, based on the first-order wireless point model, construct a calculation formula for the energy consumed by a node to transmit K bits of data.

[0070] The calculation formula for the energy consumed by a node to transmit K bits of data is:

[0071] E TX (K,d)=KE elec +K∈ fs d 2 ;

[0072] Among them, E TX (K,d) represents the energy consumed by a node to transmit K bits of data; ∈ fs is the energy coefficient of the free space model, E elecis the energy consumed by the transmitting circuit to send one bit of data, d is the transmission distance between nodes; K represents the amount of bit data transmitted by the node.

[0073] The calculation formula of the transmission distance between node i and node j is:

[0074]

[0075] Among them, d I and d J are the horizontal and vertical distances between node i and node j, respectively.

[0076] Step S220, when the transmission distance between nodes exceeds a distance threshold, the energy consumed by the node to transmit K bits of data is calculated using multipath propagation according to the constructed calculation formula for the energy consumed by the node to transmit K bits of data.

[0077] Specifically, when d exceeds the distance threshold d0, the formula for calculating the energy consumed by the energy-consuming node to transmit K bits of data using multipath propagation is as follows:

[0078] E TX (K,d)=KE elec +K∈ mp d 4 ;

[0079] Among them, E elec It is the energy consumed by the transmitting circuit to send one bit of data; mp is the energy coefficient of the multipath attenuation model; d is the transmission distance between nodes.

[0080] The calculation formula for the energy consumed by a node receiving K bits of data is:

[0081] E RX (k) = kE elec ;

[0082] The energy consumed by a node to process K bits of data is calculated as:

[0083] E proc (k) = kE proc ;

[0084] Among them, E proc is the energy consumed to process one bit of data; k is the amount of bit data processed by the node.

[0085] Step S3, establishing the optimization problem of network routing based on the constructed wireless sensor network model and energy consumption model.

[0086] Step S3 includes:

[0087] Step S310: pre-set restriction conditions of the wireless sensor network model.

[0088] Specifically, the constraints of the wireless sensor network model include:

[0089] ⑴Nodes (including sensor nodes and sink nodes) are randomly deployed in the WSN (Wireless Sensor Network) scenario, and their locations will not change after deployment. Sensor nodes are powered by batteries, while sink nodes have unlimited energy.

[0090] ⑵Each sensor node can only transmit data with its neighboring nodes within its communication range.

[0091] ⑶ As long as the routing is set correctly, all data can be transmitted to the sink node through the sensor node.

[0092] (4) When the energy of the sensor node is less than the energy threshold, the sensor node no longer transmits data packets.

[0093] Among them, the sensor node sends data to the sink node through a point-to-point link, forming a multi-hop routing mechanism. In WSN distributed routing, each node independently determines the next hop node in the path to the sink node, thereby realizing hop-by-hop forwarding of data packets. For example, node i generates a data packet and selects a neighbor node j as the next hop node in the neighbor node set through routing strategy π.

[0094] Step S320, based on the constraints of the wireless sensor network model and the goal of network routing, establish the optimization problem of network routing.

[0095] It should be explained that the goal of the network routing of the present invention is to minimize energy consumption to improve energy efficiency, while balancing the energy consumption of nodes to avoid premature death of nodes.

[0096] Among them, the optimization problem of network routing is expressed as:

[0097] Y=min(∑ i,j E i,j ·x i,j +Var E );

[0098] Where Y represents the optimization problem of network routing; min() represents the minimum value; E i,j represents the energy consumption of data transmission between node i and node j. i,j Represents the data transmission link between node i and node j in this round of routing; Var E Representing the variance of energy consumption, it is possible to balance the energy consumption of each node while considering the energy consumption.

[0099] The optimization problem satisfies the following conditions:

[0100]

[0101] Among them, x i,j is the data transmission link between node i and node i in this round of routing, x i,j =1 means node i transmits a data packet to node j, x i,j = 0 means that there is no data packet transmission between node i and node j. In one round of routing, node i only transmits a data packet to one node. i is the energy of node i. Var E is the variance of energy consumption, which can balance the energy consumption of each node while considering the energy consumption. th Represents the energy threshold. If the node energy E i Less than E th The node is considered dead. d(i,Sink) represents the distance from node i to the SINK node. If node i transmits a data packet to node j, node i selects a next-hop node j that is closer to the sink node to avoid forming a loop. d(j,Sink) represents the distance from node j to the SINK node. Sink represents the SINK node. Node i and node j represent sensor nodes.

[0102] Step S4, modeling the optimization problem as a multi-agent reinforcement learning problem, and selecting the optimal routing path based on the multi-agent reinforcement learning problem. In the multi-agent reinforcement learning problem, each sensor node is regarded as an independent agent, and each agent learns a strategy to optimize the overall network performance based on the local state and reward mechanism. The strategy refers to the selected optimal routing path.

[0103] Step S4 includes:

[0104] Step S410, modeling the optimization problem as a multi-agent reinforcement learning problem.

[0105] As a specific embodiment of the present invention, modeling the optimization problem as a multi-agent reinforcement learning problem includes:

[0106] Step S411, each sensor node is regarded as an independent intelligent agent, and based on the local observation information, the selection of the optimal routing path is transformed into the following decision-making process:

[0107] Decision making process using tuples<N,S,A,O,R,P> express.

[0108] Where N is the number of agents, S is the environment state space, and A = A1×…×A N is the joint action space of all agents, O = O1 × ... × O Nis the observation space of all agents, and R represents the reward obtained by the agent; is the reward obtained by each agent, and P(s′|s,a) is the probability that the agent takes action a from state s to state s′.

[0109] It is understandable that the present application models the routing problem as a partially observable Markov decision process, enabling each node to make autonomous decisions based on local observations. The present application effectively handles the problems of incomplete information between nodes and local observations, allowing each node to make decisions based on limited local information, thereby more realistically simulating the actual network environment.

[0110] This application designs a reward function by considering node energy consumption, transmission direction and remaining energy, looking for paths with lower energy consumption and more balanced energy consumption between nodes, effectively avoiding loop routing and extending the network life cycle.

[0111] Among them, the intelligent agent v i The observation space is defined as O i :{D,U i ,W i}. Where D is the destination of the data packet, U i is the local information of the agent, including node energy and location, W i It is the information shared by the agent’s neighbor nodes.

[0112] It needs to be explained that due to limited power and processing capabilities, nodes can only transmit data within a limited range.

[0113] Among them, the intelligent agent v i The action space is defined as A i :N i Where N i For node n i The set of neighbor nodes.

[0114] Step S412, when agent v i When a data packet needs to be transmitted, it will be transmitted according to the current state s i In the action space A i Select action a i , transmit the data packet to the next hop node.

[0115] It needs to be explained that each node acts as an intelligent agent and performs packet forwarding operations based on available routes and the state of the network.

[0116] Step S413, agent v i Select the forward action a from the available actions i As the next hop, in forwarding action a i Rewards received afterwards.

[0117] Among them, the overall goal of sensor nodes in WSN is to consume the minimum energy and transmit data packets to SINK nodes while considering balanced energy consumption.

[0118] Among them, the intelligent agent v i The reward function is defined as:

[0119]

[0120] Among them, r i Represents the agent v i The reward function; α is the weight, if the node can transmit forward, then the agent v is given i A positive reward, otherwise give the agent v i A negative reward. E i,j Represents the data transmission energy consumption between node i and node j. d(i,Sink) represents the distance from node i to SINK node; d(j,Sink) represents the distance from node j to SINK node.

[0121] The above method enables the nodes to move data packets toward the SINK node to avoid forming a loop path. The actions of all nodes constitute a round of WSN data packet transmission path, and the sum of all actions consumed is the total energy consumption of a round. In WSN, all agents should work together to minimize energy consumption.

[0122] In step S414, all agents calculate their rewards based on the same global reward function.

[0123] Among them, the calculation formula of the global reward function is:

[0124] R=∑ i r i -β·Var E ;

[0125] Among them, R represents the reward obtained by the agent; r i Represents the agent v i The reward function; β is the weight coefficient of energy balance. Var E Represents the variance of the residual energy, Var E The larger the value, the higher the imbalance of energy consumption among nodes.

[0126] It needs to be explained that Var E The larger the value, the more likely it is that some nodes may consume too much energy due to their frequent participation in data reception and forwarding, while other nodes consume less energy, resulting in uneven energy distribution.

[0127] It can be understood that the present application comprehensively considers the node energy consumption, node data transmission direction and the reward function of the remaining energy, and realizes low-energy consumption and energy-balanced routing, thereby effectively extending the network life cycle.

[0128] Step S420, based on the multi-agent reinforcement learning problem, improve the strategy through iterative interaction and training learning to select the optimal routing path.

[0129] Among them, the decision-making process of selecting the optimal routing path conversion follows the centralized training decentralized execution mechanism.

[0130] Each agent has its own actor network μ θ , target actor network μ θ′ , Critic Network Q φ , target critic network Q φ′ .

[0131] In the iterative interaction phase, the actor network of each agent calculates the probability of selecting each neighbor node based on local observations. Each agent selects its own action through a soft maximization strategy. The soft maximization strategy converts the Q value of the action into a probability distribution, so that each action has a chance to be selected, and actions with high Q values ​​are more likely to be selected.

[0132] Among them, the selection probability of each action of the agent is:

[0133]

[0134] Among them, P(a i ) represents the agent action a i The selection probability of ; τ is the temperature parameter, which is used to control the smoothness of the action probability distribution. Q(s,a i ) indicates action a i Q value; Q(s,a j ) indicates action a j The Q value of . n represents the number of neighbor nodes of the current node; e is the base of the natural logarithm.

[0135] It needs to be explained that the soft maximization strategy can flexibly control the balance between exploration and utilization, so that the agent maintains exploration in the early stage of training and gradually transitions to the utilization stage in the later stage of training, thereby improving the learning effect of the overall strategy.

[0136] like Figure 3 As shown in the figure, after each agent selects an action, it obtains a global reward and the next state. The tuple (s, o, a, r, s′) is stored as an experience in the experience replay pool.

[0137] In the training and learning phase, batches of experience are extracted from the experience replay pool and the critic network is updated using the minimized loss function. Specifically, the formula for minimizing the loss function is as follows:

[0138]

[0139] in, represents the minimization loss function; s represents the state value; The target value estimated by the target critic network. Represents the critic network state s i Take action a i action value.

[0140] in, s ′i Indicates the next state of the network; a ′i Represents the target actor network according to the next state s′ i Generated action; Q φ′ (s′ i ,a′ i ) represents the target critic network according to the next state s ′i and the next action a′ i The calculated Q value; γ represents the discount factor; r i is the agent v i Instant rewards.

[0141] It needs to be explained that Q φ′ (s′ i ,a′ i ) is the direct output of the target critic network, It is the final target value calculated by combining the immediate reward, discount factor and target critic network output Q value, which is used to train the current critic network.

[0142] Train the agent's actor network using a gradient descent policy:

[0143]

[0144] in, Represents the objective function Gradient of the policy network parameter φ; μ θ represents the actor network. i Indicates the next state; a i Indicates the next action. represents logμ θ (s i ) with respect to the gradient of the policy network parameter φ; Q φ (s i ,a i ) indicates state si Take action a i action value.

[0145] Soft update the weights of the target actor network:

[0146] θ′=τθ+(1-τ)θ′;

[0147] Where θ′ represents the weight of the target actor network. τ is the temperature parameter. θ represents the weight of the actor network.

[0148] Soft update the weights of the target critic network:

[0149] φ′=τφ+(1-τ)φ′.

[0150] Where φ′ represents the weight of the target critic network. φ represents the weight of the critic network.

[0151] As a specific embodiment of the present invention, the routing algorithm of the present invention is shown in Table 1.

[0152] Table 1, routing algorithm:

[0153]

[0154]

[0155] As a specific embodiment of the present invention, the routing selection method of the present invention is simulated and analyzed. The specific simulation and analysis method is as follows:

[0156] The routing selection method of the present invention is compared and analyzed with the existing EER-RL algorithm, EEM-CRP algorithm and D2CRP algorithm in terms of energy consumption, throughput, number of remaining nodes, remaining node energy and energy efficiency. The EER-RL algorithm is an energy-saving routing protocol based on reinforcement learning. The EEM-CRP algorithm is a cognitive radio power allocation algorithm based on energy efficiency and load balancing. The D2CRP algorithm is a distributed dynamic cognitive radio power allocation algorithm.

[0157] The comparison and analysis includes the following steps:

[0158] Step T1, simulation setting.

[0159] As a specific embodiment of the present invention, the simulation setting includes, in a 200*200 WSN scenario, the SINK node is located in the center, and 50 nodes with an initial energy of 1J are evenly distributed in the scenario. In each round of WSN routing, the sensor node collects 4096 bits of data from the environment and transmits it to the SINK node. When the node energy is less than the energy threshold E thWhen =0.1mJ, the sensor node no longer participates in WSN routing.

[0160] The proposed routing method of the present invention is implemented using the Pytorch framework (an open source deep learning framework for machine learning and deep learning). The actor and critic networks of each agent have 2 hidden layers, and the number of hidden units is set to 64 and 128 respectively. The activation function is ReLU, the optimizer uses Adam, and the learning rates of the actor network and the critic network are set to 0.0003 and 0.0004 respectively. The playback buffer size is set to 10000.

[0161] Step T2, performance analysis.

[0162] like Figure 4 As shown, the reward value in the training of the routing selection method of the present invention changes with the number of training rounds. In the early stage of training, the reward is unstable and the reward value fluctuates greatly because the algorithm tries different strategies in the exploration stage. As the number of training rounds increases, the reward value gradually increases and stabilizes after about 800 rounds. After the early exploration stage, the routing selection method of the present invention gradually found a relatively stable strategy that can obtain a higher reward value in each round of training. In the later stage of training, the reward value is basically stable, the algorithm is close to the convergence state, and the adaptability of the model to the environment and the stability of the strategy are improved. The convergence trend shows that the routing selection method of the present invention can effectively learn and maintain a stable high-reward strategy in this environment, achieving the expected training goal.

[0163] like Figure 5 As shown in the figure, the energy consumption value in the training of the routing selection method of the present invention changes with the number of training rounds. In the early stage of training, the energy consumption is high and fluctuates significantly, and the model has not yet found the path with the best energy efficiency. As the training progresses, the energy consumption gradually decreases and stabilizes after about 800 rounds, and finally maintains at about 0.02J. After completing the exploration phase, the routing selection method of the present invention successfully found a low-energy path, thereby significantly reducing the overall energy consumption.

[0164] like Figure 6 As shown in the figure, the arrival rate of the routing selection method of the present invention changes with the number of training rounds during the training process. The arrival rate is defined as the proportion of forward transmission nodes to the total number of nodes. In the early stage of training, the arrival rate fluctuates greatly, ranging from 0.7 to 0.9, and the network has not yet found the optimal transmission path. As the training progresses, the arrival rate gradually increases and stabilizes at around 0.95 after about 250 rounds, reaching 1 at about 800 rounds. This shows that the routing selection method of the present invention successfully optimizes the transmission path, can fully forward transmit, and avoid loop routing.

[0165] like Figure 7As shown in the figure, the energy consumption variance of the routing selection method of the present invention changes with the number of training rounds during the training process. In the early stage of training, the energy consumption variance is large and fluctuates significantly, and the energy consumption between nodes is uneven. As the training progresses, the variance gradually decreases and stabilizes after about 750 rounds, and finally stabilizes at 1×10 -9 The routing selection method of the present invention effectively optimizes the path selection during the training process, makes the energy consumption of each node more balanced, and reduces the difference in energy consumption in the network.

[0166] The difference in energy efficiency between different algorithms in terms of average energy consumption in each round of simulation. The routing strategy of EER-RL tends to select relay nodes with shorter path distances, so in the early simulation stage, the energy consumption of these nodes increases rapidly, resulting in uneven network load. The energy consumption curve of the D2CRP algorithm also has high fluctuations, especially in the middle and late stages. This is mainly because the two-hop transmission mechanism is adopted in intra-cluster and inter-cluster routing. Some cluster head nodes are prone to premature energy exhaustion under high load, thereby affecting the energy balance of the overall network. In contrast, the energy consumption curve of the EEM-CRP algorithm is relatively stable. By optimizing cluster head selection and routing through the dragonfly algorithm, EEM-CRP reduces energy consumption to a certain extent, but is still slightly inferior to the routing selection method of the present invention in terms of global optimality. Compared with other algorithms, the energy consumption curve of the routing selection method of the present invention is stable and always remains at a low level. The routing selection method of the present invention can dynamically adjust the transmission path of the node, so that each node selects the optimal path according to its own residual energy and the state of the adjacent nodes, avoiding frequent use of the same node as a relay, thereby achieving balanced optimization of global energy consumption. In addition, the routing selection method of the present invention also ensures the effectiveness and energy efficiency of data transmission through a loop-free topology structure, further balancing the energy consumption distribution between nodes, thereby extending the life cycle of the network. In summary, the routing selection method of the present invention has significant advantages in energy efficiency, while other algorithms have certain shortcomings in terms of the stability and persistence of energy consumption.

[0167] In terms of the residual energy of different algorithms in each round of simulation. The residual energy of the EER-RL algorithm decreases the fastest, indicating that the routing strategy consumes a large amount of energy of specific nodes in the early stage, causing high-frequency relay nodes to run out of energy prematurely, which significantly shortens the network life cycle. The D2CRP algorithm selects cluster heads and two-hop cluster structures for routing, but the cluster head nodes are heavily loaded, resulting in accelerated energy consumption, and the residual energy decreases rapidly in the middle and late stages. In contrast, the energy consumption of the EEM-CRP algorithm is relatively balanced, and the cluster head selection optimization based on the dragonfly algorithm slows down the energy consumption rate to a certain extent, but is slightly worse than the routing selection method of the present invention in terms of global optimality. The residual energy curve of the routing selection method of the present invention is the most gentle, showing a stable downward trend, which is due to the routing strategy of multi-agent collaboration, which makes the energy consumption between nodes evenly distributed, avoids the premature exhaustion of energy by a single node due to frequent relaying, and thus significantly extends the life cycle of the network. In summary, the routing selection method of the present invention shows significant advantages in terms of residual energy, and effectively improves the energy utilization and survival time of the network.

[0168] In terms of the number of surviving nodes in each round of simulation of different algorithms. The node failure rate of the EER-RL algorithm is the fastest, and the number of nodes decreases rapidly in the early stage of the simulation, indicating that the routing strategy fails to evenly distribute energy consumption, causing some nodes to run out of energy prematurely and the network life cycle is short. The D2CRP algorithm performs routing through a two-hop cluster structure, which delays the failure of nodes to a certain extent, but due to the large load of the cluster head node, the remaining nodes gradually run out of energy, resulting in a rapid decrease in the number of surviving nodes in the middle and late stages. The EEM-CRP algorithm is relatively stable in terms of node survival, and the optimization of the dragonfly algorithm in cluster head selection and path planning balances the energy consumption of some nodes and extends the life cycle of the nodes. The routing selection method of the present invention shows significant advantages in the number of surviving nodes, and the node failure rate is the slowest. Its multi-agent collaboration strategy can adjust the routing selection according to the remaining energy of the node, so that the energy consumption is evenly distributed, effectively avoiding the premature failure of a single node, and ultimately significantly extending the life cycle of the entire network. In summary, the routing selection method of the present invention has significant advantages in node survival rate.

[0169] In terms of the data packet transmission volume of different algorithms in each round of simulation. The data packet transmission volume of the EER-RL algorithm grows relatively slowly, which is mainly due to the early node failure in the network life cycle, resulting in a reduction in effective transmission paths, thereby limiting the total data packet transmission volume. The data packet transmission volume of the D2CRP algorithm is relatively high, and its two-hop cluster structure improves the data transmission efficiency to a certain extent, but in the later stage of the simulation, due to the exhaustion of energy of the cluster head node, the transmission performance gradually decreases. The EEM-CRP algorithm shows a relatively stable data packet transmission growth, and its optimized cluster head selection and path planning enable the transmission efficiency to be maintained. Although it slows down slightly in the later stage, the overall data transmission performance is better than that of EER-RL and D2CRP. The routing selection method of the present invention has the highest data packet transmission volume and maintains a stable growth trend throughout the simulation process, which benefits from the advantages of the multi-agent collaboration strategy. By dynamically adjusting the routing selection, the routing selection method of the present invention can balance the energy consumption of each node, prolong the survival time of the node, and thus ensure that more data packets can be continuously transmitted. In summary, the routing selection method of the present invention has obvious advantages in data transmission capacity and network sustainability.

[0170] In terms of the energy efficiency of different algorithms in each round of simulation. The energy efficiency curves of the EER-RL and D2CRP algorithms remain basically horizontal, showing the lowest and slightly higher energy utilization efficiency respectively, and both have limited effects in terms of energy efficiency. EEM-CRP has a higher energy efficiency. By optimizing cluster head selection and path planning, the data transmission volume per unit energy is significantly improved. The curve shows an increasing trend in the early and middle stages, but the growth slows down in the later stages. This is because some nodes are exhausted and cannot transmit data packets, resulting in a decrease in efficiency. The routing selection method of the present invention has the best energy efficiency performance, and the curve shows a steady upward trend. The multi-agent collaboration strategy effectively balances the energy consumption of each node and achieves the highest data transmission volume and energy utilization efficiency. In summary, the present invention has shown significant advantages in energy efficiency.

[0171] In summary, the present invention can handle the problem of incomplete information between nodes by modeling the routing problem as a decision-making process, so that each node can make autonomous decisions based on local observation information. At the same time, by designing a reward function that comprehensively considers node energy consumption, transmission direction and residual energy, the nodes are guided to select paths with lower energy consumption and balance during the routing selection process, avoiding loop paths, and optimizing network energy efficiency and load balancing. The simulation results show that compared with the EER-RL algorithm, the EEM-CRP algorithm and the D2CRP algorithm, the routing selection method of the present invention has improved performance in terms of network life cycle, energy consumption optimization and energy balance.

[0172] The present application also provides a computer storage medium, wherein the computer storage medium stores computer instructions, and when the computer instructions are called, they are used to execute the address mapping method of the large-capacity solid-state hard disk. The computer storage medium contains one or more program instructions, and the one or more program instructions are used by the processor to execute a collaborative routing selection method for the Internet of Things based on multi-agent deep reinforcement learning.

[0173] The embodiments disclosed in the present invention provide a computer-readable storage medium, in which computer program instructions are stored. When the computer program instructions are executed on a computer, the computer executes the above-mentioned Internet of Things collaborative routing selection method based on multi-agent deep reinforcement learning.

[0174] An embodiment of the present invention provides a processor for processing the above-mentioned Internet of Things collaborative routing selection method based on multi-agent deep reinforcement learning.

[0175] In the embodiment of the present invention, the processor may be an integrated circuit chip having the ability to process signals. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0176] The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The processor reads the information in the storage medium and completes the steps of the above method in combination with its hardware.

[0177] The storage medium may be a memory, which may be, for example, a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memory.

[0178] The beneficial effects achieved by this application are as follows:

[0179] (1) This application comprehensively considers the node energy consumption, node data transmission direction and the reward function of the remaining energy, realizes low-energy consumption and energy-balanced routing, avoids loop paths, and selects routing paths with low energy consumption and more balanced, thereby effectively extending the network life cycle and improving energy efficiency.

[0180] (2) This application effectively handles the problems of incomplete information and local observation between nodes, allowing each node to make decisions based on limited local information, thereby more realistically simulating the actual network environment.

[0181] In the description of this application, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, "plurality" means two or more, unless otherwise clearly and specifically defined.

[0182] In the description of the present application, the word "for example" is used to mean "used as an example, illustration or explanation". Any embodiment described as "for example" in the present application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is given to enable any technician in the field to implement and use the present invention. In the following description, details are listed for the purpose of explanation. It should be understood that a person of ordinary skill in the art can recognize that the present invention can be implemented without using these specific details. In other examples, well-known structures and processes will not be elaborated in detail to avoid obscuring the description of the present invention with unnecessary details. Therefore, the present invention is not intended to be limited to the embodiments shown, but is consistent with the widest scope consistent with the principles and features disclosed in the present application.

[0183] The above description is only an embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of the claims of the present invention.

Claims

1. A collaborative routing selection method for the Internet of Things based on multi-agent deep reinforcement learning, characterized in that: The method includes: Construct wireless sensor network model; Build energy consumption model; Based on the constructed wireless sensor network model and energy consumption model, the optimization problem of network routing is established; Model the optimization problem as a multi-agent reinforcement learning problem, and select the optimal routing path based on the multi-agent reinforcement learning problem; In the multi-agent reinforcement learning problem, each sensor node is regarded as an independent agent, and each agent learns the strategy to optimize the overall network performance based on the local state and reward mechanism.

2. The IoT collaborative routing selection method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: Building a wireless sensor network model includes: Construct a sink node and multiple sensor nodes in the wireless sensor network model; A bidirectional transmission link is established for data transmission between two sensor nodes or between a sensor node and a sink node.

3. The IoT collaborative routing selection method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: Building an energy consumption model includes: Based on the first-order wireless point model, a calculation formula for the energy consumed by a node to transmit K bits of data is constructed; When the transmission distance between nodes exceeds the distance threshold, the energy consumed by the node to transmit K bits of data is calculated using multipath propagation according to the constructed calculation formula for the energy consumed by the node to transmit K bits of data; The formula for calculating the energy consumed by transmitting K bits of data using multipath propagation is: Yes TX (K,d)=KE elec +K∈ mp d 4 ; Among them, E elec It is the energy consumed by the transmitting circuit to send one bit of data; mp is the energy coefficient of the multipath attenuation model; d is the transmission distance between nodes.

4. The IoT collaborative routing selection method based on multi-agent deep reinforcement learning according to claim 3 is characterized in that: The calculation formula for the energy consumed by a node receiving K bits of data is: Yes RX (k)=kE elec ; The energy consumed by a node to process K bits of data is calculated as: Yes proc (k)=kE proc ; Among them, E proc The energy consumed to process one bit of data; k represents the amount of bit data processed by the node.

5. The IoT collaborative routing selection method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: Based on the constructed wireless sensor network model and energy consumption model, the optimization problems of establishing network routing include: Pre-set constraints for wireless sensor network models; Based on the constraints of the wireless sensor network model and the objectives of network routing, the optimization problem of network routing is established.

6. The IoT collaborative routing selection method based on multi-agent deep reinforcement learning according to claim 5 is characterized in that: The network routing optimization problem is expressed as: Y=min(∑ i,j E i,j ·x i,j +Was E ); Where Y represents the optimization problem of network routing; min() represents the minimum value; E i,j represents the energy consumption of data transmission between node i and node j; x i,j Represents the data transmission link between node i and node j in this round of routing; Var E Represents the variance of energy consumption.

7. The IoT collaborative routing selection method based on multi-agent deep reinforcement learning according to claim 6 is characterized in that: The optimization problem satisfies the following conditions: Among them, x i,j is the data transmission link between node i and node j in this round of routing, x i,j =1 means node i transmits a data packet to node j, x i,j =0 means that there is no data packet transmission between node i and node j. In one round of routing, node i only transmits data packets to one node; E i is the energy of node i; Var E is the variance of energy consumption; E th represents the energy threshold, d(i, Sink) represents the distance from node i to the SINK node. If node i transmits a data packet to node j, then node i selects a next-hop node j that is closer to the sink node to avoid forming a loop; d(j, Sink) represents the distance from node j to the SINK node.

8. The IoT collaborative routing selection method based on multi-agent deep reinforcement learning according to claim 1 is characterized in that: The optimization problem is modeled as a multi-agent reinforcement learning problem. The optimal routing path is selected based on the multi-agent reinforcement learning problem, including: Model the optimization problem as a multi-agent reinforcement learning problem; Based on the multi-agent reinforcement learning problem, the strategy is improved through iterative interaction and training learning to select the optimal routing path.

9. The IoT collaborative routing selection method based on multi-agent deep reinforcement learning according to claim 8 is characterized in that: Modeling the optimization problem as a multi-agent reinforcement learning problem involves: Each sensor node is considered as an independent intelligent agent; When the agent v i When a data packet needs to be transmitted, it will be transmitted according to the current state s i In the action space A i Select action a i , transmit the data packet to the next hop node; Agent v i Select the forward action a from the available actions i As the next hop, in forwarding action a i Rewards received after All agents calculate their rewards based on the same global reward function; Among them, the calculation formula of the global reward function is: R=∑ i r i -β·There is E ; Among them, R represents the reward obtained by the agent; r i Represents the agent v i The reward function; β is the weight coefficient of energy balance; Var E represents the variance of the residual energy.

10. The IoT collaborative routing selection method based on multi-agent deep reinforcement learning according to claim 8, characterized in that: Each agent has its own actor network, target actor network, critic network, and target critic network; In the iterative interaction phase, each agent’s actor network calculates the probability of selecting each neighbor node based on local observations; During the training and learning phase, batches of experience are drawn from the experience replay pool and the critic network is updated using the minimized loss function.

Images