Method for improving reliability of Internet of Things sensing communication network in power transmission channel scene

Abstract

The invention discloses a method for improving the reliability of an Internet of Things sensing communication network in a power transmission channel scene, and belongs to the field of intelligent power grid monitoring. The method specifically comprises the following steps: firstly, building a satellite edge computing network in a power transmission channel scene; in a time slot t, the user u enters a satellite edge computing network and uploads a task, and the task is unloaded to a low-orbit satellite h associated with the user u; when the task needs to be unloaded from the satellite h to the satellite h '; if yes, unloading delay # imgabs0 # is calculated, otherwise, transmission delay # imgabs1 # and distance propagation delay # imgabs2 # are calculated respectively, end-to-end delay # imgabs3 # of a user u in a time slot t is calculated in combination with the three delays, so that the total QoE value of the user u is calculated, and the total QoE value of all users in all time slots is maximized under the energy constraint to serve as a target function; and finally, solving the objective function based on a deep reinforcement learning algorithm of convolution-near-end strategy optimization to obtain a cooperative calculation strategy pi theta between satellites. The overall reliability and the intelligent level of the power grid are improved.

Description

Technical Field

[0001] The present invention belongs to the field of smart grid monitoring, and specifically is a method for improving the reliability of an IoT-aware communication network in a power transmission channel scenario. Background Art

[0002] In the "Shagohuang" area, transmission lines face severe challenges from extreme environments; the strong winds, strong pollution, soil erosion, and wind-blown sand that are unique to the area pose a major threat to the safe operation of transmission lines. These environmental factors not only increase the difficulty and risk of manual inspections, but also make it difficult for traditional monitoring methods to meet the requirements of real-time and accuracy. For example, the deposition of dirt on insulators may lead to discharge accidents, the tilting deformation of towers may cause structural failure, and the ice dancing of conductors and damage to hardware may cause line interruption or failure[1][2].

[0003] In addition, how to design a reliable self-powered IoT sensing communication network to meet the large-scale deployment needs of new energy transmission channels in the Xinjiang Uygur Autonomous Region is also an urgent problem to be solved [3].

[0004] Through the evaluation of existing technologies, including the application of satellite services in transmission line monitoring, and how to use satellite communications to improve the monitoring coverage and communication reliability of transmission lines in remote areas [4], a theoretical and practical basis is provided for the design and implementation of a self-powered IoT sensing system that can adapt to extreme environments.

[0005] There are many studies on task scheduling strategies in satellite edge computing scenarios. For example, reference [5] proposed a decentralized method for online edge IoT task scheduling and resource allocation problems, which can significantly reduce offloading response time and improve resource utilization. Reference [6] proposed an innovative resource allocation mechanism in the field of software-defined satellite-ground networks. This mechanism uses deep Q learning to optimize the joint allocation of network, cache, and computing resources.

[0006] The above studies highlight the challenges faced by STNs when integrating satellite networks and terrestrial networks, and dynamically manage and coordinate these resources through a software-defined STN framework. However, the deep Q-learning algorithm used in the above studies has certain limitations. When the state space becomes large and continuous, it will encounter difficulties and it is difficult to meet the requirements we want to achieve.

[0007] References are as follows:

[0008] [1] Ding Xizhen. On the key points of safety work in power transmission line construction under different climatic conditions [J]. Science and Technology Entrepreneur, 2013, (24): 19.

[0009] [2] Liu Yu, Lu Jiazheng, Zhang Hongxian, et al. Research on ice-covered power transmission line dancing prediction technology in Hunan Province [J]. Hunan Electric Power, 2015, 35(03): 23-26.

[0010] [3] Wu Feilong. Application of satellite communication in large-scale UAV power transmission line inspection and control[J]. China Electric Power, 2017, 50(08): 135-140.

[0011] [4]Brandherm F,Gedeon J,Abboud O,et al.BigMEC: Scalable ServiceMigration for Mobile Edge Computing[C] / / 2022IEEE / ACM 7th Symposium on EdgeComputing(SEC).IEEE,2022:136-148.

[0012] [5]Xing N, Wu P, Jin S, et al.Task classification unloading algorithm for mobile edge computing in smart grid[C] / / 2021IEEE 5th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC). IEEE, 2021:1636-1640.

[0013] [6]Qiu C, Yao H, Yu FR, et al. Deep Q-learning aided networking, caching, and computing resources allocation in software-defined satellite-terrestrial networks [J]. IEEE Transactions on Vehicular Technology, 2019, 68(6): 5871-5883. Summary of the invention

[0014] The present invention proposes a method for improving the reliability of the Internet of Things perception communication network in the transmission channel scenario to adapt to the special needs of the new energy transmission channel scenario in the Xinjiang Uygur Autonomous Region. Through an innovative self-power supply mechanism, satellites are used as an auxiliary to the ground network to transmit data, thereby improving the reliability of the Internet of Things communication network. In addition, the present invention can also timely warn of potential structural risks, thereby effectively ensuring the safe operation of the new energy transmission channel and improving the overall reliability and intelligence level of the power grid.

[0015] The method for improving the reliability of the IoT sensing communication network in the power transmission channel scenario has the following specific steps:

[0016] Step 1: Build a satellite edge computing network in the power transmission channel scenario;

[0017] The network includes H satellites, each of which is surrounded by several edge servers, each of which is surrounded by a base station and a resource scheduler;

[0018] Each satellite has different computing resources, including CPU and memory. When computing tasks are offloaded to the satellite, they will be allocated according to the computing resources available to the satellite.

[0019] Step 2: At time slot t, user u enters the satellite edge computing network and uploads the task, which is then offloaded to the low-orbit satellite h associated with user u via a wireless link.

[0020] Step 3: Determine whether to offload the task of user u from satellite h to satellite h'; if so, calculate the offloading delay of the task from satellite h and satellite h' Otherwise, go to step 4;

[0021]

[0022] Decision variables Indicates whether to offload the task of user u from satellite h to satellite h'; if Indicates that satellite h will offload the task of user u to satellite h', indicates that the task of user u is calculated by satellite h, that is, the satellite covering user u; H is the set of all satellites; U t is the set of all users; T is the set of all time slots.

[0023] is the offloading ratio of user u’s tasks from satellite h to satellite h’; is the distance from user u to satellite h'; v c is the speed at which light travels in a vacuum; The computational delay calculated by satellite h for user u's task is:

[0024]

[0025] is the available computing resource allocated to users by satellite h at time slot t, and represents the number of CPU cycles that the edge server can provide per second. It obeys Gaussian distribution, denoted as where μ and σ 2 are the mean and variance respectively. The number of CPU cycles per second required for a given task.

[0026] Step 4: Calculate the transmission delay of user u to satellite h for offloading tasks and distance propagation delay

[0027] Transmission delay The calculation formula is:

[0028]

[0029] is the task size of user u; is the propagation rate from user u to the satellite, and its calculation formula is:

[0030]

[0031] M is the number of orthogonal subcarriers that the Ka band is divided into; B is the bandwidth of the subcarrier; σ 2 is the Gaussian noise power; G I and G R are the gains of the transmitting antenna and the receiving antenna respectively; is the transmission power of user u at time slot t; The channel gain between user u and satellite h at time slot t;

[0032] Distance Propagation Delay The calculation formula is:

[0033]

[0034] is the distance from user u to satellite h;

[0035] Step 5: Calculate the end-to-end delay of user u in time slot t by combining the calculated delay, transmission delay and distance propagation delay.

[0036]

[0037] Step 6: Determine end-to-end delay With threshold N uThe relationship between , and thus calculate the total QoE value of user u:

[0038]

[0039] gather is the set of unloading decisions between satellites, is the satellite unloading ratio set,

[0040] Step 7: Maximize the total QoE value of all users in all time slots under energy constraints as the objective function:

[0041]

[0042] γ t-1 is the discount factor;

[0043] Step 8: Solve the objective function based on the deep reinforcement learning algorithm optimized by convolution-proximal strategy to obtain the collaborative computing strategy π between satellites θ , thereby improving the reliability of the communication network.

[0044] The specific steps are:

[0045] Step 801: Initialize the structure and learning rate of the neural network, set the number and size of convolution kernels, set the convolution stride; and initialize the policy network parameter θ and value network parameter ω, constellation information and the position of each user.

[0046] Step 802: Execute the current strategy in the environment to obtain interaction data (a t ,s t ,r t ,s t+1 ), including state a t , actions t , reward r t and possible next states s t+1 ;

[0047] Step 803: Input the state matrix into the neural network. The convolution layer extracts features from the state matrix and flattens it before inputting it into the fully connected layers of the policy network and the value network. The value network gives an evaluation function.

[0048] The height and width of the state matrix are determined by the height and width of the constellation, and the number of channels is determined by the dimension of the state space, that is, the distance between the user and the satellite, and the available computing resources of the satellite.

[0049] The value network gives the evaluation function: The expression of L(ω) is as follows:

[0050]

[0051] where r t is the reward given by the environment at time slot t, γ is the discount factor, V ω (s t ) is the state value function given by the value network;

[0052] Step 804: Using the evaluation function of the value network, use the generalized advantage estimation (GAE) to estimate the advantage function:

[0053]

[0054] Among them, ψ is a parameter used to adjust the bias-variance trade-off, δ t As the TD-error identification parameter of time slot t, its calculation formula is:

[0055] δ t =r t +γV ω (s t+1 )-V ω (s t )

[0056] Step 805: Calculate the target of the policy network using the advantage function, as shown below:

[0057] max E t [min(r t (θ)A t ,clip(r t (θ),1-∈,1+∈)A t )]

[0058] is the importance sampling ratio, which is used to measure the relative performance of the new strategy and the old strategy. The clip function converts r t The value of (θ) is restricted to the range [1-∈, 1+∈], where ∈ is a hyperparameter.

[0059] Step 806: By optimizing the target of the policy network, the policy parameter θ is updated using the gradient ascent method, and the iteration is performed until the number of iterations is satisfied, and the collaborative calculation strategy π is output. θ ;

[0060] In each update, the PPO algorithm calculates the action probability and corresponding reward in each state. Then, it calculates the policy gradient based on the importance sampling ratio and threshold, and uses this gradient to update the policy parameters;

[0061] The advantages of the present invention are:

[0062] 1) The present invention provides a method for improving the reliability of an IoT-aware communication network in a power transmission channel scenario. CNN is used in a proximal strategy optimization algorithm. Compared with a fully connected network, CNN can better capture local information and utilize the relationship between data elements. In the considered scenario, the satellites are connected by star links, which makes them not independent of each other. Therefore, convolution operations are used to extract local relationships between neighboring satellites. Secondly, the giant LEO satellite constellation considered has a Euclidean data structure, and each node has the same number of neighbors. In this case, CNN is more suitable for feature extraction than other networks (such as neural networks and recursive neural networks).

[0063] 2) The present invention provides a method for improving the reliability of the Internet of Things perception communication network in the transmission channel scenario. By introducing a convolutional neural network, it can not only better capture local information and utilize the relationship between data elements to improve the quality of understanding and algorithm efficiency, but also reduce the computational complexity and better perform feature extraction, thereby enhancing the algorithm's extensive search capability when exploring the solution space and more effectively finding the optimal solution. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 It is a flow chart of a method for improving the reliability of an IoT sensing communication network in a power transmission channel scenario of the present invention;

[0065] Figure 2 It is a schematic diagram of a power transmission channel scenario of an edge computing network constructed by the present invention;

[0066] Figure 3 It is a comparison chart of the effects of the method of the present invention and two control experiments of BE and RE;

[0067] Figure 4 is the end-to-end delay of a single user within 50 seconds when the present invention uses the PPO algorithm;

[0068] Figure 5 is the end-to-end delay of a single user within 50 seconds when the present invention uses the BE algorithm;

[0069] Figure 6 It is the end-to-end delay of a single user within 50 seconds when the present invention uses the RE algorithm. DETAILED DESCRIPTION

[0070] In order to facilitate those skilled in the art to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Obviously, the described embodiments are only partial embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the scope of protection of the present invention.

[0071] The present invention provides a method for improving the reliability of IoT sensing communication networks in power transmission channel scenarios. It adopts a deep reinforcement learning algorithm optimized by convolution-proximal strategy, and shows good efficiency and accuracy in finding the optimal solution of mixed integer nonlinear optimization problems. The algorithm is implemented in edge computing scenarios to efficiently manage and allocate computing resources, thereby improving the overall performance of the system and ensuring the stability and response speed of the service.

[0072] The present invention integrates satellite communication technology with the ground network to form a space-ground integrated communication system. Satellite links are used as an auxiliary means of ground communication to ensure that data can still be transmitted to the central control system through satellite links in areas where ground networks are limited or unavailable. At the same time, a dynamic network management algorithm is designed to intelligently adjust the data transmission path and resource allocation according to the real-time monitored network status and environmental changes to optimize network performance. At the same time, a proximal strategy optimization algorithm is used to initialize the strategy network parameter θ and the value network parameter ω through the known constellation information and the position of each user, and finally the collaborative computing strategy between satellites can be obtained.

[0073] Compared with the existing Q-learning and DQN algorithms, the PPO algorithm used in the present invention has good stability, high data utilization, and is easier to adjust and implement. In order to solve the impact of the rapid mobility of low-orbit satellites and the complex satellite environment on the communication link, a user coverage model based on continuous time slots is designed, which can describe the connection mechanism and corresponding connection strategy between satellites and users in detail. In addition, in view of the problem that the traditional DRL algorithm has slow convergence speed and is easy to fall into local optimality, the PPO algorithm is used to solve the proposed dynamic optimization problem, and the simulation results verify its superiority.

[0074] The method for improving the reliability of the IoT sensing communication network in the power transmission channel scenario is as follows: Figure 1 As shown, the specific steps are as follows:

[0075] Step 1: Build a satellite edge computing network in the power transmission channel scenario;

[0076] The network includes H satellites, each of which is surrounded by several edge servers, each of which is surrounded by a base station and a resource scheduler;

[0077] Each satellite has different computing resources, including CPU and memory. When computing tasks are offloaded to the satellite, they will be allocated according to the computing resources available to the satellite.

[0078] Step 2: At time slot t, user u enters the satellite edge computing network and uploads the task, which is then offloaded to the low-orbit satellite h associated with user u via a wireless link.

[0079] Step 3: Determine whether to offload the task of user u from satellite h to satellite h'; if so, calculate the offloading delay of the task from satellite h and satellite h' Otherwise, go to step 4;

[0080]

[0081] Decision variables Indicates whether to offload the task of user u from satellite h to satellite h'; if Indicates that satellite h will offload the task of user u to satellite h', indicates that the task of user u is calculated by satellite h, that is, the satellite covering user u; H is the set of all satellites; U t is the set of all users; T is the set of all time slots.

[0082] is the offloading ratio of user u’s tasks from satellite h to satellite h’; is the distance from user u to satellite h'; v c is the speed at which light travels in a vacuum; The computational delay calculated by satellite h for user u's task is:

[0083]

[0084] is the available computing resource allocated to users by satellite h at time slot t, and represents the number of CPU cycles that the edge server can provide per second. It obeys Gaussian distribution, denoted as where μ and σ 2 are the mean and variance respectively. The number of CPU cycles per second required for a given task.

[0085] Step 4: Calculate the transmission delay of user u to satellite h for offloading tasks and distance propagation delay

[0086] Transmission delay The calculation formula is:

[0087]

[0088] is the task size of user u; is the propagation rate from user u to the satellite, and its calculation formula is:

[0089]

[0090] M is the number of orthogonal subcarriers that the Ka band is divided into; B is the bandwidth of the subcarrier; σ2 is the Gaussian noise power; G I and G R are the gains of the transmitting antenna and the receiving antenna respectively; is the transmission power of user u at time slot t; The channel gain between user u and satellite h at time slot t;

[0091] Distance Propagation Delay The calculation formula is:

[0092]

[0093] is the distance from user u to satellite h;

[0094] Step 5: Calculate the end-to-end delay of user u in time slot t by combining the calculated delay, transmission delay and distance propagation delay.

[0095]

[0096] Step 6: Determine end-to-end delay With threshold N u The relationship between , and thus calculate the total QoE value of user u:

[0097]

[0098] gather is the set of unloading decisions between satellites, is the satellite unloading ratio set,

[0099] Step 7: Maximize the total QoE value of all users in all time slots under energy constraints as the objective function:

[0100]

[0101] γ t-1 is the discount factor;

[0102] Step 8: Solve the objective function based on the deep reinforcement learning algorithm optimized by convolution-proximal strategy to obtain the collaborative computing strategy π between satellites θ , thereby improving the reliability of the communication network.

[0103] The specific steps are:

[0104] Step 801: Initialize the structure and learning rate of the neural network, set the number and size of convolution kernels, set the convolution stride; and initialize the policy network parameter θ and value network parameter ω, constellation information and the position of each user.

[0105] Step 802: Execute the current strategy in the environment to obtain interaction data (a t ,s t ,r t ,s t+1 ), including state a t , actions t , reward r t and possible next states s t+1 ;

[0106] Step 803: Input the state matrix into the neural network. The convolution layer extracts features from the state matrix and flattens it before inputting it into the fully connected layers of the policy network and the value network. The value network gives an evaluation function.

[0107] The height and width of the state matrix are determined by the height and width of the constellation, and the number of channels is determined by the dimension of the state space, that is, the distance between the user and the satellite, and the available computing resources of the satellite.

[0108] The value network gives the evaluation function: The expression of L(ω) is as follows:

[0109]

[0110] where r t is the reward given by the environment at time slot t, γ is the discount factor, V ω (s t ) is the state value function given by the value network;

[0111] Step 804: Using the evaluation function of the value network, use the generalized advantage estimation (GAE) to estimate the advantage function:

[0112]

[0113] Among them, ψ is a parameter used to adjust the bias-variance trade-off, δ t As the TD-error identification parameter of time slot t, its calculation formula is:

[0114] δ t =r t +γV ω (s t+1 )-V ω (s t )

[0115] Step 805: Calculate the target of the policy network using the advantage function, as shown below:

[0116] max E t [min(r t (θ)A t ,clip(rt (θ),1-∈,1+∈)A t )]

[0117] is the importance sampling ratio, which is used to measure the relative performance of the new strategy and the old strategy. The clip function converts r t The value of (θ) is restricted to the range [1-∈, 1+∈], where ∈ is a hyperparameter.

[0118] Step 806: By optimizing the target of the policy network, the policy parameter θ is updated using the gradient ascent method, and the iteration is performed until the number of iterations is satisfied, and the collaborative calculation strategy π is output. θ ;

[0119] In each update, the PPO algorithm calculates the action probability and corresponding reward in each state. Then, it calculates the policy gradient based on the importance sampling ratio and threshold, and uses this gradient to update the policy parameters;

[0120] Example:

[0121] (1) Simulation settings

[0122] In order to verify the performance of the proposed solution, this embodiment uses PyCharm for scene construction and numerical simulation. Consider a set of 25 satellites (5×5), which are part of the Walker Star constellation and fly over a small ground area of ​​1km×1km at an altitude of 500km. The task computing power requirements in this network will be adjusted according to the dynamic changes in the number of instances. Each satellite has different computing power resources, including CPU and memory. When computing tasks are offloaded to the satellite, they will be allocated according to the computing resources available to the satellite.

[0123] As the demand for network services changes, computing tasks need to be rescheduled in the satellite. The PPO algorithm is used to obtain the corresponding optimal strategy that meets each optimization goal, and then the appropriate solution is selected as the best scheduling solution for the edge computing network according to network demand. Finally, the collaborative computing method is used to implement the solution.

[0124] (1) Scenario Model

[0125] In the satellite edge computing network environment, it is crucial to ensure the balanced distribution of network resources as the satellite changes dynamically. To address this problem, the IoT perception technology model proposed in this paper involves timely scaling or migration of microservice instances, while placing special emphasis on maintaining the continuous availability of microservices during the migration process. Figure 2As shown in Figure 1, the edge computing network includes multiple edge servers deployed on satellites and a resource scheduler responsible for rescheduling. Each edge server is equipped with a base station (BS) to provide it with necessary network support.

[0126] The problem is modeled as a continuous time slot T = {1, 2, ..., T}. Assume that there are H satellites H = {1, 2, ..., H} in the system, and each satellite is equipped with a MEC server. and are the number of users entering the network at time slot t and the number of users leaving the network at the end of time slot t. t represents the users in the network at the beginning of time slot t. represents the task size of user u at time slot t. The transmission delay of user u unloading the task to the satellite at time slot t can be expressed as Since the user is far away from the associated satellite, the round-trip propagation delay between the user and the satellite needs to be considered, where the distance from the user to the satellite is expressed as The propagation delay is expressed as To express.

[0127] In order to meet the latency requirements of each user, it is necessary to use the computing resources on the satellite to help users process tasks. First, the computing tasks are offloaded to the low-orbit satellite associated with the user through a wireless link, and then partially offloaded by the low-orbit satellite to another satellite for processing, realizing task scheduling and edge collaboration in the low-orbit satellite network. In the scenario considered, one satellite can serve many user requests, and user requests are random. Therefore, the available computing resources of the satellite vary randomly.

[0128] set up is the available computing resources of satellite s at time t, indicating the number of CPU cycles per second that the edge server can provide. Assume It obeys Gaussian distribution, denoted as where μ and σ 2 are the mean and variance respectively. Given a task requires CPU cycles per second The computation delay can be written as Will As a decision variable, it indicates whether to offload the task of user u from satellite h to satellite h'. Satellite h will offload the task of user u to satellite h', otherwise = indicates that the task of user u is calculated by satellite h, which is the satellite covering user u. A satellite can only offload a user's task to one satellite. Next, we discuss the delay in the calculation of the user's task on the satellite. Since the satellite can partially offload the task offloaded by the user again, As the partial unloading ratio, is the Manhattan distance between satellite h and satellite h' at time slot t, and the calculated delay is In summary, the end-to-end delay of user u in time slot t can be expressed as:

[0129]

[0130] A QoE utility parameter is defined to measure the user's total QoE value. In order to provide users with a smooth experience, the actual end-to-end delay of user u needs to be lower than its maximum tolerable delay. The constructed Q value function is as follows:

[0131]

[0132] Among them, the collection is the set of unloading decisions between satellites, Unload the satellite ratio set; the goal is to maximize the total user long-term QoE value under energy constraints by developing inter-satellite cooperative computing strategies and appropriate resource allocation schemes. Therefore, we are more concerned about the long-term performance of the wireless network over a continuous period of time rather than the instantaneous performance.

[0133] The goal of designing a solution is to maximize the cumulative QoE value of the total number of users. The optimization problem can be expressed as:

[0134]

[0135] (2) Based on the convolution-proximal strategy optimization algorithm

[0136] A deep reinforcement learning algorithm based on convolution-proximal policy optimization is proposed to solve the above problems. The proximal policy optimization algorithm (PPO) is a widely used deep reinforcement learning algorithm that shows good efficiency and accuracy in finding the optimal solution to mixed integer nonlinear optimization problems. With the addition of neural networks, PPO can handle problems with high-dimensional action space and state space. However, the deep neural network in the PPO algorithm uses a fully connected network, which is not as effective as the convolutional neural network (CNN).

[0137] First, compared with fully connected networks, convolutional neural networks can better capture local information and exploit the relationship between data elements. In the scenario considered, satellites are connected through Starlink, which makes them not independent of each other. Therefore, convolution operations are used to extract local relationships between neighboring satellites. In order to overcome the shortcomings of the prior art, this example proposes an improved PPO algorithm. By improving the neural network in the PPO algorithm, it can better perform feature extraction and make the algorithm more effective.

[0138] The PPO algorithm uses an Actor-Critic framework, where the Actor generates operations and the Critic evaluates operations. An important drawback of the ordinary Actor-Critic framework is its low sample effectiveness, which means that it requires a large number of interactions with the environment before converging. To solve this problem, PPO allows the target policy π θ (a t |s t ) and behavioral strategy π θold (a t |s t ) there is a slight difference between t and t are the observed state and executed action of the agent at time t.

[0139] The core idea of ​​the PPO algorithm is to use an importance sampling ratio to measure the relative performance between the new and old strategies, and then update the policy parameters based on this ratio. In each update, the PPO algorithm collects a segment of trajectory data and calculates the action probability and corresponding reward for each state on the trajectory. Then, the policy gradient is calculated based on the importance sampling ratio and the threshold parameter, and this gradient is used to update the policy parameters. It uses the clip function to constrain the difference between the new and old strategies. When the change in the policy exceeds a certain threshold, the clip function prevents it from growing further. The PPO algorithm ensures that the policy update is not too large, thereby improving the stability of the algorithm.

[0140] In PPO, the state matrix will first be feature extracted by the convolutional layer. Next, the extracted features are flattened and input into the fully connected layers of the policy network and the value network. The purpose of the value network is to give the agent an evaluation, which is similar to TD-target. Next, let δ t As TD-error, where ψ is used to adjust the bias-variance trade-off. In practice, the advantage function can be estimated using the generalized advantage estimator (GAE).

[0141]

[0142] Combined with the given advantage function, the goal of the policy network is as follows

[0143] max E t [min(r t (θ)A t ,clip(r t (θ),1-∈,1+∈)A t )]

[0144] θ is the parameter of the policy network, is the importance sampling ratio, which is used to measure the relative performance of the new strategy and the old strategy. The clip function converts r t The value of (θ) is restricted to the range [1-∈, 1+∈], where ∈ is a hyperparameter.

[0145] By introducing convolutional neural networks, the present invention can not only better capture local information and utilize the relationship between data elements to improve the quality of solution and algorithm efficiency, but also reduce the computational complexity and better perform feature extraction, thereby enhancing the algorithm's extensive search capability when exploring the solution space and more effectively finding the optimal solution.

[0146] (2) Simulation results

[0147] like Figure 3 As shown in the figure, the control experiment of the two groups of this simulation experiment is shown, including BE (calculating in the associated satellite without task migration) and RE (random task migration strategy). It can be clearly seen from the figure that the PPO algorithm shows a low latency at the beginning of the round, and as the number of rounds increases, the latency of the PPO algorithm remains at a low and relatively stable level. This shows that the PPO algorithm can quickly adapt to the environment and quickly learn an effective task offloading strategy.

[0148] In contrast, the delays of the RE and BE strategies are higher at the beginning of the episode, and the delay of the RE strategy fluctuates as the episode increases, but is still higher than the PPO algorithm overall. The delay of the BE strategy remains at a high level throughout the simulation and does not drop significantly. This may be due to the lack of an effective task migration mechanism, which makes it impossible to fully utilize the resources of the satellite network. The curve trend in the figure further confirms the effectiveness of the PPO algorithm in reducing the end-to-end delay of the task. The PPO algorithm optimizes resource allocation by dynamically adjusting task offloading decisions, thereby achieving lower latency.

[0149] The end-to-end delay of a single user within 50 seconds when using different algorithms is as follows Figure 4 , Figure 5 and Figure 6As shown in the figure, the black solid line represents the maximum tolerable end-to-end delay, that is, T_max. It can be seen that when the PPO algorithm is used, most points fall below the T_max line, and only four values ​​do not meet the delay requirements. In contrast, the BE algorithm and the RE algorithm are much worse than the PPO algorithm in terms of end-to-end delay. Among them, the RE algorithm is much better than the BE algorithm because it can better utilize the resources of the satellite network. This shows that the PPO algorithm can be adjusted dynamically.

[0150] Through simulation experiments, it is concluded that the improved algorithm proposed in the present invention can significantly improve the adaptability and stability when facing different environments and task requirements, thereby providing a more efficient and reliable solution strategy.

Claims

1. A method for improving the reliability of IoT perception communication network in a power transmission channel scenario, characterized in that: The specific steps are as follows: Step 1: Build a satellite edge computing network in the power transmission channel scenario; Step 2: At time slot t, user u enters the satellite edge computing network and uploads the task, which is then offloaded to the low-orbit satellite h associated with user u via a wireless link. Step 3: Determine whether to offload the task of user u from satellite h to satellite h'; If yes, calculate the offloading delay of the task from satellite h and satellite h' Otherwise, go to step 4; Decision variables Indicates whether to offload the task of user u from satellite h to satellite h'; if Indicates that satellite h will offload the task of user u to satellite h', indicates that the task of user u is calculated by satellite h, that is, the satellite covering user u; H is the set of all satellites; U t is the set of all users; T is the set of all time slots; is the offloading ratio of user u’s tasks from satellite h to satellite h’; is the distance from user u to satellite h'; v c is the speed at which light travels in a vacuum; The computational delay calculated by satellite h for user u's task is: Step 4: Calculate the transmission delay of user u to satellite h for offloading tasks and distance propagation delay Transmission delay The calculation formula is: is the task size of user u; is the propagation rate from user u to the satellite; Distance Propagation Delay The calculation formula is: is the distance from user u to satellite h; Step 5: Calculate the end-to-end delay of user u in time slot t by combining the calculated delay, transmission delay and distance propagation delay. Step 6: Determine end-to-end delay With threshold N u The relationship between , and thus calculate the total QoE value of user u: gather is the set of unloading decisions between satellites, is the satellite unloading ratio set, Step 7: Maximize the total QoE value of all users in all time slots under energy constraints as the objective function: γ t-1 is the discount factor; Step 8: Solve the objective function based on the deep reinforcement learning algorithm optimized by convolution-proximal strategy to obtain the collaborative computing strategy π between satellites θ , thereby improving the reliability of the communication network.

2. The method for improving the reliability of the IoT sensing communication network in a power transmission channel scenario according to claim 1, characterized in that: The satellite edge computing network constructed in step 1 includes H satellites, and a number of edge servers are deployed around each satellite. Each edge server is equipped with a base station and a resource scheduler around it; Each satellite has different computing resources, including CPU and memory. When computing tasks are offloaded to the satellite, they will be allocated according to the computing resources available to the satellite.

3. The method for improving the reliability of the IoT sensing communication network in a power transmission channel scenario according to claim 1, characterized in that: The delay is calculated in step 3 The calculation formula is: is the available computing resource allocated to users by satellite h in time slot t, which represents the number of CPU cycles that the edge server can provide per second; It obeys Gaussian distribution, denoted as where μ and σ 2 are the mean and variance respectively; The number of CPU cycles per second required for a given task.

4. The method for improving the reliability of the IoT sensing communication network in a power transmission channel scenario according to claim 1, characterized in that: In step 4, the propagation rate The calculation formula is: M is the number of orthogonal subcarriers that the Ka band is divided into; B is the bandwidth of the subcarrier; σ 2 is the Gaussian noise power; G I and G R are the gains of the transmitting antenna and the receiving antenna respectively; is the transmission power of user u at time slot t; Channel gain between user u and satellite h at time slot t.

5. The method for improving the reliability of the IoT sensing communication network in a power transmission channel scenario according to claim 1, characterized in that: The specific steps of step eight are: Step 801: Initialize the structure and learning rate of the neural network, set the number and size of convolution kernels, and set the convolution stride; and initialize the policy network parameter θ and value network parameter ω, constellation information, and the position of each user; Step 802: Execute the current strategy in the environment to obtain interaction data (a t ,s t ,r t ,s t+1 ), including state a t , actions t , reward r t and possible next states s t+1 ; Step 803: Input the state matrix into the neural network. The convolution layer extracts features from the state matrix and flattens it before inputting it into the fully connected layers of the policy network and the value network. The value network gives an evaluation function. The height and width of the state matrix are determined by the height and width of the constellation, and the number of channels is determined by the dimension of the state space, i.e., the distance between the user and the satellite, and the available computing resources of the satellite; The value network gives the evaluation function: The expression of L(ω) is as follows: where r t is the reward given by the environment at time slot t, γ is the discount factor, V ω (s t ) is the state value function given by the value network; Step 804: Using the evaluation function of the value network, use the generalized advantage estimation (GAE) to estimate the advantage function: Among them, ψ is a parameter used to adjust the bias-variance trade-off, δ t As the TD-error identification parameter of time slot t, its calculation formula is: δ t =r t +γV ω (s t+1 )-V ω (s t ) Step 805: Calculate the target of the policy network using the advantage function, as shown below: max E t [min(r t (i)A t ,clip(r t (θ),1-∈,1+∈)A t )] is the importance sampling ratio, which is used to measure the relative performance of the new strategy and the old strategy. The clip function converts r t The value of (θ) is restricted to the range [1-∈, 1+∈], where ∈ is a hyperparameter; Step 806: By optimizing the target of the policy network, the policy parameter θ is updated using the gradient ascent method, and the iteration is performed until the number of iterations is satisfied, and the collaborative calculation strategy π is output. θ .

Images