Method for resource allocation in mobile wireless powered internet of things based on deep reinforcement learning

Abstract

The application discloses a kind of mobile wireless energy supply Internet of Things resource allocation methods based on deep reinforcement learning, in the control phase of time slot, after receiving the state information containing current residual energy and data packet quantity sent by all Internet of Things nodes in receiving range, resource allocation information is obtained based on deep reinforcement learning, and the resource allocation information is broadcasted to all Internet of Things nodes in receiving range by mixed access point through control channel.In the data transmission phase of time slot, after receiving broadcast data from mixed access point, whether to send data is selected according to the channel obtained by itself.In the energy capture phase of time slot, Internet of Things nodes carry out energy capture.Mixed access point trains deep reinforcement learning through the state information of Internet of Things nodes obtained in current time slot, resource allocation result and throughput.The application increases the utilization rate of network resources, achieves the purpose of energy saving and improving the total throughput of wireless energy supply Internet of Things.

Description

A Resource Allocation Method for Mobile Wireless Power Supply IoT Based on Deep Reinforcement Learning Technical Field

[0001] This application belongs to the field of wireless transmission technology, and in particular relates to a resource allocation method for mobile wireless power supply Internet of Things based on deep reinforcement learning, which is a resource allocation method between nodes in wireless power supply Internet of Things. Background Technology

[0002] Currently, with the increasing number of IoT devices, network resource scarcity has become a key issue in mobile wireless-powered IoT. To alleviate this scarcity, deep reinforcement learning is widely used as a method to rationally allocate network resources and improve network throughput. In mobile wireless-powered IoT, IoT nodes first capture energy and store it in batteries. When an IoT node has communication needs and sufficient energy, it uses the captured energy for data transmission. In this IoT system, IoT nodes move in each time slot, and their movement follows specific mathematical rules. Since rationally allocating network resources can extend the working time of IoT nodes and improve network throughput, it is a crucial issue to consider. Furthermore, the difficulty of rationally allocating network resources increases with the length of time slots. For a long-term resource allocation problem, the resource allocation in each time slot is related because resources need to be allocated in every time slot, and the time slots are continuous. The discrete number of time slots makes this resource allocation problem unsolvable in polynomial time, thus making it an NP-hard problem.

[0003] To address the long-standing resource allocation problem in mobile wireless power IoT, IoT nodes employ energy-saving resource allocation methods for data communication. This allows IoT nodes to make rational use of network resources under limited conditions, thereby improving the overall network throughput, which is highly significant. Summary of the Invention

[0004] The purpose of this application is to provide a resource allocation method for mobile wireless power supply IoT based on deep reinforcement learning, so that IoT nodes can make reasonable use of network resources to improve the total throughput of mobile power supply IoT.

[0005] The technical solution adopted in this invention is: a resource allocation method for mobile wireless power supply IoT based on deep reinforcement learning, wherein the mobile wireless power supply IoT includes IoT nodes and hybrid access points, each time slot includes a control phase and a transmission phase, wherein the transmission phase includes a data transmission phase and an energy capture phase, and the resource allocation method for mobile wireless power supply IoT based on deep reinforcement learning includes:

[0006] During the control phase, after receiving the status information of all IoT nodes within the receiving range, the hybrid access point obtains resource allocation information based on the trained deep reinforcement learning network, and sends the resource allocation information to all IoT nodes within the receiving range through the control channel.

[0007] During the data transmission phase, after receiving the resource allocation information, the IoT node performs data parsing to obtain data transmission time, energy capture time, transmission power and channel acquisition status related to itself, and checks its own channel acquisition status.

[0008] If an IoT node determines that it has not obtained a data transmission channel, then the IoT node uses the data transmission time and transmission power received from the hybrid access point to transmit data;

[0009] If an IoT node finds that it has not obtained a data transmission channel, then the IoT node sets the data transmission time to zero and directly enters the energy capture phase;

[0010] During the energy capture phase, IoT nodes that have obtained a data transmission channel will perform energy capture based on the energy capture time after the data transmission phase ends; IoT nodes that have not obtained a data transmission channel will directly perform energy capture until the energy capture phase ends.

[0011] Furthermore, after receiving the status information of all IoT nodes within its reception range, the hybrid access point obtains resource allocation information based on deep reinforcement learning, including:

[0012] The hybrid access point sends wake-up data to all IoT nodes within its receiving range via a control channel. When an IoT node receives the wake-up data, it sends its status information to the hybrid access point via the control channel.

[0013] After receiving the status information of all IoT nodes within its reception range, the hybrid access point obtains the channel gain status of all IoT nodes within its reception range through channel estimation. It then merges the status information of the IoT nodes with the corresponding channel gain status and inputs the merged status into a trained deep reinforcement learning network to obtain the resource allocation of the IoT nodes. This includes the current data transmission time allocation and the current remaining energy usage allocation of all IoT nodes within its reception range. The current remaining energy usage allocation represents the ratio of the energy consumed by the IoT node in transmitting data to the current remaining energy of the IoT node.

[0014] The hybrid access point obtains the data transmission time of the IoT nodes within its receiving range by multiplying the current data transmission time ratio of the IoT nodes within its receiving range by the transmission phase time.

[0015] The difference between the total time of the transmission phase and the data transmission time is the energy capture time;

[0016] The hybrid access point obtains the transmission power of the IoT nodes within its receiving range by multiplying the percentage of the current remaining energy used by the current remaining energy and then dividing by the data transmission time.

[0017] The hybrid access point calculates the achievable throughput using Shannon's formula by taking the transmission power and data transmission time of IoT nodes within the receiving range. At the same time, it obtains the required throughput by dividing the number of data packets of IoT nodes within the receiving range by the data transmission time. The smaller of the two throughputs is taken as the actual achievable throughput of IoT nodes within the receiving range.

[0018] The actual throughput of IoT nodes within the receiving range is sorted from largest to smallest by the hybrid access point. The top K IoT nodes are allocated data transmission channels, while the remaining IoT nodes are not allocated data transmission channels.

[0019] The data transmission time, energy capture time, transmission power, and channel acquisition status are defined as resource allocation information.

[0020] Furthermore, the mobile wireless power supply IoT resource allocation method based on deep reinforcement learning further includes: at the end of each time slot, the IoT node moves according to a preset rule and generates data packets according to a Poisson process; when the IoT node moves, the channel gain state changes accordingly.

[0021] Furthermore, the coverage area of ​​the wireless power supply IoT is divided into M ring-shaped areas, and the IoT nodes move according to a preset pattern, including:

[0022] IoT nodes move according to formulas (1)-(2).

[0023]

[0024]

[0025] in, This represents the outer radius of the m-th ring. This represents the distribution density of IoT nodes. Let represent the outer radius of the (m-1)th ring, and ∈ denote the degree of non-uniformity. Represents the probability density function. express The area integral on the (m-1)th ring, express The area integral on the m-th ring, Represents the probability function and O m Let m represent a circle. Moving towards the center indicates that the IoT node moves inward toward the center of the circle, while moving outward indicates that the IoT node moves outward in the opposite direction of the center of the circle. Clockwise indicates that the IoT node moves in a clockwise direction perpendicular to the radius of the circle, and counterclockwise indicates that the IoT node moves in a counterclockwise direction perpendicular to the radius of the circle.

[0026] Furthermore, the mobile wireless power supply IoT resource allocation method based on deep reinforcement learning also includes training the deep reinforcement learning network.

[0027] The training of the deep reinforcement learning network includes:

[0028] Step 1: Initialize the deep reinforcement learning network parameters, including the policy network parameters θ a Value network parameters θ c Target policy network parameters θ' a and target value network parameters θ' a Initialize the total number of training iterations T; initialize the distribution of IoT nodes, the energy of IoT nodes, and the number of data packets, and set the number of iterations t = 1;

[0029] Step 2: The hybrid access point acquires the status information of the IoT nodes and the corresponding channel gain status;

[0030] Step 3: The hybrid access point merges the IoT node status information and the corresponding channel gain status into state s(t);

[0031] Step 4: The hybrid access point inputs the state s(t) into the policy network to obtain the action a(t), which includes the proportion of data transmission time of all IoT nodes within the receiving range and the proportion of current remaining energy usage. The proportion of current remaining energy usage represents the ratio of the energy consumed by the IoT node in transmitting data to the remaining energy of the IoT node.

[0032] Step 5: The hybrid access point remaps action a(t) to obtain resource allocation information for all IoT nodes within the receiving range, and then sends it to all IoT nodes within the receiving range;

[0033] Step 6: After all IoT nodes within the receiving range receive the resource allocation information, they perform data parsing to obtain data transmission time, energy capture time, transmission power, and channel acquisition status related to themselves. Then, they use this resource allocation information to transmit data with the hybrid access point.

[0034] Step 7: The hybrid access point calculates the total actual throughput r(t) of all IoT nodes;

[0035] Step 8: The hybrid access point stores the experience E(t-1) = (s(t-1), a(t-1), r(t-1), s(t)) of time slot t-1 into the experience replay pool and randomly selects x experiences, where s(t-1) represents the state of time slot t-1, a(t-1) represents the action of time slot t-1, r(t-1) represents the total throughput of time slot t-1, and s(t) represents the state of time slot t;

[0036] Step 9: Update the policy network parameters and value network parameters using gradient descent based on the selected x experiences according to formulas (3)-(6).

[0037]

[0038]

[0039]

[0040]

[0041] Among them, L c (θ c (t)) and L a (θ a (t) represents the loss function of the value function and the policy function, respectively. i (s i (t),a i (t); θ c (t) represents the action value function for time slot t. L represents s (θ c (t) for θ c Differentiate (t), This indicates taking the derivative with respect to a(t). This indicates that a(t) with respect to θ a (t) Take the derivative, where γ represents the discount factor and β... c and β a These are the learning rates for the value network and the policy network, respectively, where i represents the i-th experience and s represents the learning rate. i (t) represents the state of the i-th experience, a i (t) represents the action of the i-th experience, r i Let Q represent the reward for the i-th experience. i (s i (t+1),a i (t+1); θ c (t) represents the policy value function for time slot t+1, θc $(t)$ represents the value network parameter of time slot $t$, $\theta$ c $(t + 1)$ represents the value network parameter of time slot $t + 1$, $\theta$ a $(t)$ represents the policy network parameter of time slot $t$, $\theta$ a $(t + 1)$ represents the policy network parameter of time slot $t + 1$;

[0042] Step 10: Update the parameters of the target policy network and the target value network according to formulas (7)-(8),

[0043] $\theta'$ a $(t + 1)=\tau$ a $\theta$ a $(t)+(1 - \tau$ a )$\theta'$ a $(t)$ (7)

[0044] $\theta'$ c $(t + 1)=\tau$ c $\theta$ c (t)+(1 - \tau c )$\theta'$ c (t) (8)

[0045] where $\tau$ a and $\tau$ c represent the soft update factor;

[0046] Step 11: If $t < T$, then $t=t + 1$, and jump to Step 2;

[0047] Step 12: If $t\geq T$, the training ends.

[0048] Compared with the prior art, the significant advantages of the present invention are as follows: The IoT node sends the state information of its current remaining energy and the number of data packets to the hybrid access point, and the hybrid access point generates appropriate actions according to the deep reinforcement learning algorithm to rationally utilize the limited network resources. Moreover, the hybrid access point can improve the network performance based on the deep reinforcement learning algorithm through training, thereby increasing the total network throughput. Brief Description of the Drawings

[0049] FIG. 1 is a schematic structural diagram of the mobile wireless power supply IoT of the present application;

[0050] FIG. 2 is a schematic diagram of the time slot structure of the present application;

[0051] FIG. 3 is a flowchart of the data transmission method for one time slot of the mobile wireless power supply IoT of the present application;

[0052] FIG. 4 is a schematic diagram of the movement mode of the mobile wireless power supply IoT of the present application. Detailed Embodiment

[0053] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0054] The following system environment is established: As shown in Figure 1, a mobile wireless power supply IoT system is considered, consisting of a static U-antenna hybrid access point and N mobile single-antenna IoT nodes, where the IoT nodes are unevenly distributed within the coverage area of ​​the hybrid access point. As shown in Figure 2, the entire time slot is divided into a control phase and a transmission phase, where the transmission phase includes a data transmission phase and an energy harvesting phase. At the beginning of each time slot, the hybrid access point allocates network resources to the IoT nodes, including data transmission time, energy harvesting time, transmission power, and channel acquisition status. The IoT nodes move at the end of each time slot, resulting in a change of location. This application can rationally allocate network resources to all IoT nodes within the reception range of the hybrid access point, thereby achieving higher throughput. This application uses a deep reinforcement learning algorithm to optimize the resource allocation method of the IoT nodes, improving the overall network throughput while ensuring the quality of service requirements of the IoT nodes.

[0055] As shown in Figure 3, the specific steps are as follows:

[0056] A resource allocation method for mobile wireless power supply IoT based on deep reinforcement learning, wherein the mobile wireless power supply IoT includes IoT nodes and hybrid access points, and the resource allocation method for mobile wireless power supply IoT based on deep reinforcement learning includes:

[0057] The time is divided into N time slots, each of which is further divided into a control phase and a transmission phase. The transmission phase is further divided into a data transmission phase and an energy capture phase. The mobile wireless power supply IoT system has one control channel, one power supply channel, and K data transmission channels, and the network coverage area is divided into M ring-shaped regions.

[0058] During the time slot control phase:

[0059] The hybrid access point broadcasts wake-up data to all IoT nodes within its receiving range via the control channel. When an IoT node receives this wake-up data, it sends its status information, including its current remaining energy and the number of data packets, to the hybrid access point via the control channel using time-division multiplexing.

[0060] After receiving the status information of all IoT nodes within its reception range, the hybrid access point obtains the channel gain status of all IoT nodes within its reception range through channel estimation, merges the status information of the IoT nodes with the corresponding channel gain status, and sets the merged status as the input parameter of the deep reinforcement learning algorithm.

[0061] The hybrid access point inputs the merged state into the policy network to obtain the resource allocation of IoT nodes, including the data transmission time allocation of all IoT nodes within the receiving range and the current remaining energy usage allocation. The current remaining energy usage allocation represents the ratio of the energy consumed by the IoT node in transmitting data to the remaining energy of the IoT node.

[0062] The hybrid access point obtains the data transmission time of the IoT nodes within its reception range by multiplying the current data transmission time percentage of the IoT nodes within its reception range by the transmission phase time.

[0063] The difference between the total transmission time and the data transmission time is the energy capture time;

[0064] The hybrid access point obtains the transmission power of the IoT nodes within its reception range by multiplying the percentage of their current remaining energy usage by the current remaining energy and then dividing by the data transmission time.

[0065] The hybrid access point calculates the achievable throughput using Shannon's formula by taking the transmission power and data transmission time of IoT nodes within its receiving range. Simultaneously, it obtains the required throughput by dividing the number of data packets from IoT nodes within its receiving range by the data transmission time. The smaller of these two throughputs is taken as the actual achievable throughput of IoT nodes within its receiving range.

[0066] The hybrid access point sorts the actual throughput of IoT nodes within its receiving range from largest to smallest, and then allocates data transmission channels to the K IoT nodes with the largest actual throughput, while the remaining IoT nodes are not allocated data transmission channels.

[0067] Data transmission time, energy capture time, transmission power, and channel acquisition status are defined as resource allocation information.

[0068] The hybrid access point broadcasts resource allocation information to all IoT nodes within its receiving range via a control channel.

[0069] During the data transmission phase of a time slot:

[0070] After receiving broadcast data from the hybrid access point, including data transmission time, energy capture time, transmission power, and channel acquisition status, the IoT node parses the data to obtain its own data transmission time, energy capture time, transmission power, and channel acquisition status, and then checks its own channel acquisition status.

[0071] If an IoT node determines that it has not obtained a data transmission channel, then the IoT node uses the data transmission time and transmission power received from the hybrid access point to transmit data.

[0072] If an IoT node finds that it has not obtained a data transmission channel, then the IoT node sets the data transmission time to zero and sets the energy capture time to the entire transmission phase.

[0073] During the energy capture phase of the time slot:

[0074] IoT nodes that have obtained a data transmission channel will perform energy capture based on the energy capture time after the data transmission phase ends. IoT nodes that have not obtained a data transmission channel will not transmit data and will perform energy capture until the energy capture phase ends.

[0075] At the end of each time slot, the IoT node moves according to formulas (1)-(2) and generates data packets according to the Poisson process.

[0076]

[0077]

[0078] in, This represents the outer radius of the m-th ring. This represents the distribution density of IoT nodes. Let represent the outer radius of the (m-1)th ring, and ∈ denote the degree of non-uniformity. Represents the probability density function. express The area integral on the (m-1)th ring, express The area integral on the m-th ring, Represents the probability function and O m Let m represent a circle. "Towards the center" means the IoT node moves inwards towards the center of the circle; "outwards" means the IoT node moves outwards in the opposite direction of the center; "clockwise" means the IoT node moves in a clockwise direction perpendicular to the circle's radius; and "counterclockwise" means the IoT node moves in a counterclockwise direction perpendicular to the circle's radius.

[0079] As shown in Figure 4, the area covered by the hybrid access point is divided into M rings. IoT nodes move in four directions within one of these rings: towards the center, outwards, counter-clockwise, and clockwise. The movement of the IoT nodes is determined based on their distribution. The data generation process of each IoT node follows a Poisson distribution.

[0080] The hybrid access point trains deep reinforcement learning using the status information, resource allocation results, and throughput of all IoT nodes within its reception range obtained in the current time slot. The training process includes the following steps:

[0081] Step 1: Initialize the deep reinforcement learning network parameters, including the policy network parameters θ a Value network parameters θ c Target policy network parameters θ' a and target value network parameters θ' a Initialize the total number of training iterations T. Initialize the distribution of IoT nodes, the energy of IoT nodes, the number of data packets, and the number of iterations t = 1.

[0082] Step 2: The hybrid access point obtains the current remaining energy, data packet count, and corresponding channel gain status of the IoT node.

[0083] Step 3: The hybrid access point merges the IoT node status information and the corresponding channel status into a state s(t).

[0084] Step 4: The hybrid access point inputs the state s(t) into the policy network to obtain the action a(t), which includes the proportion of data transmission time of all IoT nodes within the receiving range and the proportion of current remaining energy usage. The proportion of current remaining energy usage represents the ratio of the energy consumed by the IoT node in transmitting data to the remaining energy of the IoT node.

[0085] Step 5: The hybrid access point remaps action a(t) to obtain the data transmission time, energy capture time, transmission power and channel acquisition status of all IoT nodes within the receiving range, and then broadcasts it to all IoT nodes within the receiving range.

[0086] Step 6: After all IoT nodes within the receiving range receive the data, they parse the data to obtain the data transmission time, energy capture time, transmission power, and channel acquisition status related to themselves. Then, they use these resource allocation methods to transmit data with the hybrid access point.

[0087] Step 7: The hybrid access point calculates the sum of the actual throughput r(t) of all IoT nodes.

[0088] Step 8: The hybrid access point stores the experience E(t-1) = (s(t-1), a(t-1), r(t-1), s(t)) of time slot t-1 into the experience replay pool and randomly selects x experiences, where s(t-1) represents the state of time slot t-1, a(t-1) represents the action of time slot t-1, r(t-1) represents the total throughput of time slot t-1, and s(t) represents the state of time slot t.

[0089] Step 9: Update the policy network parameters and value network parameters using gradient descent based on the selected x experiences according to formulas (3)-(6).

[0090]

[0091]

[0092]

[0093]

[0094] In the formula, L c (θ c (t)) and L a (θ a (t) represents the loss function of the value function and the policy function, respectively. i (s i (t),a i (t); θ c (t) represents the action value function for time slot t. L represents s (θ c (t) for θ c Differentiate (t), This indicates taking the derivative with respect to a(t). This indicates that a(t) with respect to θ a (t) Take the derivative, where γ represents the discount factor and β... c and β a These are the learning rates for the value network and the policy network, respectively, where i represents the i-th experience and s represents the learning rate. i (t) represents the state of the i-th experience, a i (t) represents the action of the i-th experience, r i Let Q represent the reward for the i-th experience. i (s i (t+1),a i (t+1); θ c (t) represents the action value function for time slot t+1, θ c (t) represents the value network parameters for time slot t, θ c (t+1) represents the value network parameters for time slot t+1, θ a (t) represents the policy network parameters for time slot t, θ a (t+1) represents the policy network parameters for time slot t+1.

[0095] Step 10: Update the parameters of the target policy network and the target value network according to formulas (7)-(8).

[0096] θ' a (t+1)=τ a θa (t)+(1 - τ a )θ' a (t) (7)

[0097] 0' c (t + 1) = τ c θ c (t)+(1 - τ c )θ' c (t) (8)

[0098] In the formula, τ a and τ c represent soft update factors.

[0099] Step 11: If t < T, then t = t + 1, and jump to Step 2.

[0100] Step 12: If t ≥ T, the training ends.

[0101] The above embodiments only represent one or several implementation manners of the present application. The description is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the appended claims.

Claims

1. A resource allocation method for mobile wireless power supply IoT based on deep reinforcement learning, characterized in that, The mobile wireless power supply IoT includes IoT nodes and hybrid access points. Each time slot includes a control phase and a transmission phase, wherein the transmission phase includes a data transmission phase and an energy capture phase. The mobile wireless power supply IoT resource allocation method based on deep reinforcement learning includes: In the control phase, after receiving the status information of all IoT nodes within its receiving range, the hybrid access point obtains resource allocation information based on a trained deep reinforcement learning network, and sends the resource allocation information to all IoT nodes within its receiving range through a control channel; In the data transmission phase, after receiving the resource allocation information, the IoT nodes perform data parsing to obtain their own data transmission time and energy capture data. The system records the time, transmission power, and channel acquisition status, and checks its own channel acquisition status. If an IoT node checks its own channel acquisition status and finds that it has obtained a data transmission channel, then the IoT node uses the data transmission time and transmission power received from the hybrid access point to transmit data. If an IoT node checks its own channel acquisition status and finds that it has not obtained a data transmission channel, then the IoT node sets the data transmission time to zero and sets the energy capture time to the entire transmission phase. During the energy capture phase, IoT nodes that have obtained a data transmission channel perform energy capture based on the energy capture time after the data transmission phase ends, while IoT nodes that have not obtained a data transmission channel directly perform energy capture until the energy capture phase ends.

2. The method for resource allocation in mobile wireless power IoT based on deep reinforcement learning according to claim 1, characterized in that, After receiving the status information of all IoT nodes within its reception range, the hybrid access point obtains resource allocation information based on a trained deep reinforcement learning network. This includes: the hybrid access point sending wake-up data to all IoT nodes within its reception range via a control channel; when an IoT node receives the wake-up data, it sends its status information to the hybrid access point via the control channel; after receiving the status information of all IoT nodes within its reception range, the hybrid access point obtains the channel gain status of all IoT nodes within its reception range through channel estimation, merges the status information of the IoT nodes with the corresponding channel gain status, and inputs the merged status into the trained deep reinforcement learning network to obtain the resource allocation of the IoT nodes, including the current data transmission time allocation and the current remaining energy usage allocation of all IoT nodes within its reception range, where the current remaining energy usage allocation represents the current data transmission time of the IoT node. The energy capture time is calculated as the ratio of consumed energy to the current remaining energy of the IoT nodes. The hybrid access point calculates the data transmission time of IoT nodes within its receiving range by combining the percentage of data transmission time of IoT nodes within its receiving range with the transmission phase time. The difference between the total transmission phase time and the data transmission time is the energy capture time. The hybrid access point calculates the transmission power of IoT nodes within its receiving range by combining the percentage of current remaining energy used, the current remaining energy, and the data transmission time. The hybrid access point calculates the achievable throughput by combining the transmission power and data transmission time of IoT nodes within its receiving range, and simultaneously calculates the required throughput by combining the number of data packets and the data transmission time of IoT nodes within its receiving range. The smaller of these two throughputs is taken as the actual achievable throughput of IoT nodes within its receiving range. The hybrid access point sorts the actual throughput of IoT nodes within its receiving range from largest to smallest and selects the highest throughput. One IoT node is assigned a data transmission channel, while the remaining IoT nodes are not assigned a data transmission channel; the data transmission time, energy capture time, transmission power, and channel acquisition status are defined as resource allocation information.

3. The method for resource allocation in mobile wireless power IoT based on deep reinforcement learning according to claim 1, characterized in that, The method for allocating mobile wireless power IoT resources based on deep reinforcement learning further includes: at the end of each time slot, the IoT node moves according to a preset rule and generates data packets according to a Poisson process.

4. The method for resource allocation in mobile wireless power IoT based on deep reinforcement learning according to claim 3, characterized in that, The coverage area of ​​the wireless power supply Internet of Things is divided into A ring-shaped area, wherein the IoT node moves according to a preset rule, including: the IoT node moves according to formulas (1)-(2), in, Indicates the first The outer radius of each ring, This represents the distribution density of IoT nodes. Indicates the first The outer radius of each ring, Indicates the degree of unevenness. Represents the probability density function. express In the Area fraction on each ring express In the Area fraction on each ring Representing probability functions and Represents a circular ring "Towards the center" means the IoT node moves inwards towards the center of the circle; "outwards" means the IoT node moves outwards in the opposite direction of the center of the circle; "clockwise" means the IoT node moves in a clockwise direction perpendicular to the radius of the circle; and "counterclockwise" means the IoT node moves in a counterclockwise direction perpendicular to the radius of the circle.

5. The method for resource allocation in mobile wireless power IoT based on deep reinforcement learning according to claim 1, characterized in that, The aforementioned resource allocation method for mobile wireless power supply IoT based on deep reinforcement learning further includes training the deep reinforcement learning network; training the deep reinforcement learning network includes: Step 1: Initializing the deep reinforcement learning network parameters, including policy network parameters. Value network parameters Target policy network parameters and target value network parameters Initialize the total number of training iterations Initialize the distribution of IoT nodes, the energy of IoT nodes, and the number of data packets; set the number of iterations. Step 2: The hybrid access point acquires the status information and corresponding channel gain status of the IoT node; Step 3: The hybrid access point merges the IoT node status information and corresponding channel gain status into a single status bar. Step 4: The hybrid access point sets the status. The input is fed into the policy network to obtain the action. This includes the percentage of data transmission time and the percentage of remaining energy usage for all IoT nodes within the receiving range, where the percentage of remaining energy usage represents the ratio of energy consumed by IoT nodes for data transmission to the remaining energy of the IoT nodes; Step 5: The hybrid access point performs the action. Step 6: After receiving the resource allocation information, all IoT nodes within the receiving range are remapped to obtain resource allocation information and then sent to all IoT nodes within the receiving range; Step 7: After receiving the resource allocation information, all IoT nodes within the receiving range perform data parsing to obtain their own data transmission time, energy capture time, transmission power, and channel acquisition status, and then use the resource allocation information to transmit data with the hybrid access point; Step 8: The hybrid access point calculates the total actual throughput of all IoT nodes. Step 8: The hybrid access point allocates time slots experience Stored in the experience replay pool and randomly selected One experience, among which, Indicates time slot status, Indicates time slot Actions Indicates time slot The total throughput and Indicates time slot The state; Step 9: Select the... Based on experience, the policy network parameters and value network parameters are updated using gradient descent according to formulas (3)-(6). in, and Let these represent the loss functions of the value function and the policy function, respectively. Indicates time slot Action value function, express right Differentiate, express right Differentiate, express right Differentiate, Indicates the discount factor. and These are the learning rates of the value network and the policy network, respectively. Indicates the first This experience Indicates the first A state of experience, Indicates the first This is an action based on experience. Indicates the first A reward of experience points. Indicates time slot Action value function, Indicates time slot Value network parameters, Indicates time slot Value network parameters, Indicates time slot The strategy network parameters, Indicates time slot Step 10: Update the parameters of the target policy network and the target value network according to formulas (7)-(8). in, and Indicates the soft update factor; Step 11: If ,but And jump to step 2; Step 12: If Training is over.

Images