Electronic health network computing offloading method based on DDPG algorithm

Abstract

The application discloses an electronic health network computing offloading method based on a DDPG algorithm. The method comprises the following steps: determining an electronic health network architecture, and initializing necessary network parameters; then, converting an optimal computing offloading strategy problem into a Markov decision problem; finally, training a computing offloading model by using the DDPG algorithm, obtaining a communication connection and an offloading strategy, and enabling each user to obtain appropriate communication resources and computing resources in a dynamic environment. The application not only solves the problem that local computing resources of a mobile device are limited, but also can significantly reduce the time delay and energy consumption of a system in a scene where multiple WBAN users and multiple MECs are present, and can be widely applied to occasions such as home medical treatment, sports fitness, disease prevention and control and medical monitoring.

Description

Technical Field

[0001] This invention belongs to the field of wireless communication network technology, and in particular to a user task computation offloading method based on the DDPG algorithm. Background Technology

[0002] Thanks to advancements in microelectromechanical systems (MEMS), communication technologies, and information technologies, mobile devices, including wearable devices, smartphones, and PDAs, have driven progress in the Internet of Things (IoT) field. Among these, eHealth Networks (eBANs) are a novel IoT technology that utilizes Wireless Body Area Networks (WBANs) for user disease monitoring and health management. In the 5G and 6G era, eHealth Network applications are becoming increasingly complex, and mobile data traffic is exploding. However, as a mobile device, the WBAN has very limited computing resources. If all computing tasks are run locally on the mobile device or entirely offloaded to the cloud, it is often difficult to achieve satisfactory service quality, which limits the application scope of eHealth Networks.

[0003] Mobile edge computing (MEC) is a concept proposed by the European Telecommunications Standards Institute (ETSI). Its basic idea is to extend cloud computing capabilities to the edge of cellular networks, providing solutions for running computationally intensive applications with the advantage of low latency. As a result, eHealth Networks based on Multi-Access Edge Computing (MEC) have emerged, perfectly solving the problem of limited local computing resources on mobile devices. Summary of the Invention

[0004] The purpose of this invention is to provide an electronic health network computation offloading method based on the DDPG algorithm, which enables each user to obtain appropriate communication and computing resources in a dynamic environment, solves the problem of limited local computing resources of mobile devices, and can significantly reduce system latency and energy consumption in scenarios with multiple WBAN users and multiple MECs.

[0005] The technical solution to achieve the purpose of this invention is: a computational offloading method for eHealth Networks based on the DDPG algorithm, the method comprising the following steps:

[0006] Step 1: Determine the electronic health network architecture and initialize network parameters;

[0007] Step 2: Transform the computational unloading problem into a Markov decision problem, and determine the state space, action space, and reward function in the Markov decision model;

[0008] Step 3: Train the computational unloading problem using the Deep Deterministic Policy Gradient (DDPG) algorithm, update the network parameters until the reward function converges throughout the entire operation, and output the trained decision network.

[0009] Step 4: Calculate the unloading based on the trained decision network.

[0010] Furthermore, the electronic health network architecture supporting blockchain transactions described in step 1 is divided into three layers: the user layer, the edge layer, and the cloud layer.

[0011] The user layer includes N mobile terminal device wireless body area network (WBAN) users. WBAN is used for wearable sensor devices that collect health data. WBAN users need to register with the blockchain network and use a blockchain account as the unique identifier to join the blockchain network and become legitimate participants. WBAN users can perform task calculations locally or offload some tasks to a multi-access edge computing (MEC) server or a cloud server for execution.

[0012] The edge layer includes macro base stations and M MEC servers. The controller module in the macro base station implements various control functions for the system network, including hash resource allocation strategies. The macro base station is used to monitor the blockchain network, and the MEC server provides real-time computing services for WBAN users, while participating in blockchain consensus and receiving corresponding rewards.

[0013] The cloud layer includes the cloud platform, where macro base stations in the edge layer schedule tasks to cloud servers for task offloading and computation; WBAN users purchase computing resources for data processing tasks through the blockchain, and transactions are executed by smart contracts, known as transaction contracts.

[0014] Furthermore, the parameters in the electronic health network architecture described in step 1 include several time slots t∈{0,1,2,…T-1} throughout the communication process, system bandwidth B, and spectrum bandwidth distributed to each base station, which is B. k The bandwidth W of the sub-channel For WBAN user collection, Location of MEC server set and WBAN users MEC Location For the computational tasks of WBAN user n, D n Indicates the data size of the task, q n Indicates the number of CPU cycles required for the task, K n Indicates task priority, τ n Indicates the expected completion time of the task and the average channel gain h. n,m (t), data rate r n,m The time T required for the task n (t) and energy consumption E n (t);

[0015] (1) The average channel gain h between WBAN user n and MEC server m in the current time slot t.n,m as follows:

[0016]

[0017] Where d n,m (t) represents the distance from the current WBAN to the MEC server, α represents the path loss exponent, and h0 represents the reference gain;

[0018] (2) Data rate r between WBANn and MEC server m n,m as follows:

[0019]

[0020] Where N0 represents the Gaussian noise variance. The fixed subcarrier power of WBANn;

[0021] (3) The time T required for WBAN user n to complete the task at time t n (t) is:

[0022]

[0023] Where q n D represents the number of CPU cycles required for a task. n Indicates the size of the task's data; r n Y is the transmission rate between the MEC server and the cloud server when user n offloads a task to the cloud layer through the edge layer; m X represents the number of CPU cores in MEC server m. n f is the number of CPU cores for WBAN user n; c Indicates the computing frequency of the remote cloud server; f u and These represent the CPU computing frequency of a single core in WBAN users and MEC, respectively, which is the number of CPU cycles per second. Indicates the return time for task processing; when o n When (t) = 0, the task is executed locally, T n This is equivalent to the local task computation time; when o n When (t) = 1, the task is offloaded to the MEC server, T n The sum of task unloading time, MEC computation time, and return time; when o n (t) = -1, the task is unloaded to a remote cloud server, T n The time for the task to be unloaded to the MEC server, the time for the task to be unloaded from the MEC server to the cloud server, and the cloud server computation time;

[0024] (4) Energy consumption of WBAN

[0025] Where e0 is the CPU energy consumption coefficient, o n (t)≠0 indicates that the WBANn task is offloaded to the MEC server or cloud server, and the energy consumption at this time is the transmission energy consumption; when o n When (t) = 0, the task is executed locally, and its energy consumption is the local computation energy consumption.

[0026] Furthermore, the Markov decision problem model described in step 2 includes a state space S(t), an action space A(t), and a reward function R(t). This computational unloading problem is viewed as finding the optimal task unloading strategy π. * This maximizes long-term cumulative rewards;

[0027] Step 2-1, calculate the state space as follows:

[0028]

[0029] Where the matrix Represents a set of N WBAN tasks; matrix Represents the set of locations of all WBAN users; matrix Represents the remaining task quantity for all WBANs, matrix R = [r n,m ] N×M To represent the set of communication rate indices between the MEC server and WBAN; matrix Represents the set of remaining energy states of the MEC server;

[0030] Step 2-2, calculate the motion space as follows:

[0031] A(t)={u n,m (t),o n (t)|n=1,2,…N; m=1,2,…M}

[0032] Where u n,m (t)∈{0,1} represents the connection action, o n (t)∈{0,-1,1} represents the unloading action; when u n,m When WBAN=1, WBANn connects to MEC server m; otherwise, u n,m =0; and Only when u n,m WBANn will only offload its tasks to the MEC server m when = 1; when o n When (t) = 0, the task is executed locally; while o n When (t) = 1, the task is offloaded to the MEC server; while o n When (t) = -1, the task is unloaded to a remote cloud server;

[0033] Steps 2-3: Calculate the reward function R(t), representing the system reward as follows: (The cost of all tasks in time slot t is used as the reward.)

[0034]

[0035] Among them, the task cost C n The processing delay T for the task n And WBAN energy consumption E n linear combination, λ represents the time and energy costs required to execute this mission. t and λ e These are the delay factor and the energy consumption factor, respectively.

[0036] Furthermore, step 3, the training of the DDPG algorithm, includes the following steps:

[0037] Step 3-1: Initialize algorithm parameters, WBAN user locations, task information, and remaining energy;

[0038] Step 3-2: Train the decision network parameters. In the previous training step, the μ output parameter μ(s) of the Actor main network... i Add motion noise n i Obtain the actual output action a i =μ(s) i )+n i ,in μ e and σ e,i These are the mean and standard deviation, respectively.

[0039] Step 3-3: In an environment with multiple edge servers and multiple WBAN users, the agent selects action a. i Observe the next state s i+1 and instant rewards i Then the sequence (s) i ,a i ,s i+1 ,r i Store the data in the experience replay pool. Determine if the experience replay pool is full. If it is, proceed to the next step; otherwise, return to step 3-2.

[0040] Steps 3-4 involve updating the Actor main network and Critic main network by selecting K batch sizes of sequences from the experience replay pool. The Actor target network outputs μ'(s). i+1 The Critic target network calculates the target value y based on the output of the Actor target network. i ;

[0041] Steps 3-5: Update the parameters of the decision network over multiple training cycles. If the user device runs out of energy or the task ends, terminate the training cycle early. After each training cycle, determine whether the reward function of the decision network has converged. If it has, end the training; otherwise, return to step 3-4.

[0042] Furthermore, step 4, which involves calculating the unloading based on the trained decision network, specifically involves: setting the current state s... i (t) Input the decision network and output action a. i (t), which refers to the appropriate user connection strategy and task offloading strategy.

[0043] A computational offloading system for an eHealth network based on the DDPG algorithm, the system comprising:

[0044] The network construction module is used to determine the eHealth network architecture and initialize network parameters;

[0045] The model building module is used to transform the computational offloading problem into a Markov decision problem and determine the state space, action space and reward function in the Markov decision model.

[0046] The training module is used to train the computational offloading problem using the Deep Deterministic Policy Gradient (DDPG) algorithm, update the network parameters, and output the trained decision network until the reward function converges during the entire operation.

[0047] The unloading module is used to perform unloading calculations based on the trained decision network.

[0048] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps:

[0049] Step 1: Determine the electronic health network architecture and initialize network parameters;

[0050] Step 2: Transform the computational unloading problem into a Markov decision problem, and determine the state space, action space, and reward function in the Markov decision model;

[0051] Step 3: Train the computational unloading problem using the Deep Deterministic Policy Gradient (DDPG) algorithm, update the network parameters until the reward function converges throughout the entire operation, and output the trained decision network.

[0052] Step 4: Calculate the unloading based on the trained decision network.

[0053] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, performs the following steps:

[0054] Step 1: Determine the electronic health network architecture and initialize network parameters;

[0055] Step 2: Transform the computational unloading problem into a Markov decision problem, and determine the state space, action space, and reward function in the Markov decision model;

[0056] Step 3: Train the computational unloading problem using the Deep Deterministic Policy Gradient (DDPG) algorithm, update the network parameters until the reward function converges throughout the entire operation, and output the trained decision network.

[0057] Step 4: Calculate the unloading based on the trained decision network.

[0058] Compared with the prior art, the significant advantages of this invention are: 1) It can reasonably allocate communication and computing resources during the offloading process, so that the task execution time and energy consumption in the system are minimized; 2) Applying the DDPG algorithm to the task offloading problem of edge networks can make the strategy optimization more efficient and obtain the optimal solution faster.

[0059] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0060] Figure 1 This is a flowchart of the electronic health network computation offloading method based on the DDPG algorithm of the present invention.

[0061] Figure 2 This is a diagram of the electronic health network architecture in an embodiment of the present invention.

[0062] Figure 3 This is a comparison chart of task costs under different schemes in the embodiments of the present invention. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] In one embodiment, combined Figure 1 A computational offloading method for eHealth Networks based on the DDPG algorithm is provided, the method comprising the following steps:

[0065] Step 1: Determine the electronic health network architecture and initialize network parameters;

[0066] Step 2: Transform the computational unloading problem into a Markov decision problem, and determine the state space, action space, and reward function in the Markov decision model;

[0067] Step 3: Train the computational unloading problem using the Deep Deterministic Policy Gradient (DDPG) algorithm, update the network parameters until the reward function converges throughout the entire operation, and output the trained decision network.

[0068] Step 4: Calculate the unloading based on the trained decision network.

[0069] Furthermore, in one embodiment, the electronic health network architecture supporting blockchain transactions described in step 1 is divided into three layers: a user layer, an edge layer, and a cloud layer. The user layer consists of N mobile terminal device wireless body area network (WBAN) users, where WBAN is a wearable sensor device used to collect health data. WBAN users can perform task computation locally or offload some tasks to a multi-access edge computing (MEC) server or a cloud server. The edge layer consists of macro base stations and M MEC servers. The controller module in the macro base station can implement various control functions for the system network, such as connection and offloading strategies. The macro base station can also communicate with servers in the cloud platform. The MEC servers in the network are typically placed near roadside small base stations to provide them with certain wireless communication capabilities. The MEC servers have sufficient computing resources, including high-frequency CPU cores and large storage capacity, enabling them to provide real-time computing services to WBAN users. The cloud platform in the cloud layer has greater computing power than the MEC servers, and tasks are scheduled to the cloud servers by the macro base stations in the edge layer for task offloading computation.

[0070] Furthermore, in one embodiment, the parameters in the electronic health network architecture described in step 1 include several time slots t∈{0,1,2,…T-1} throughout the communication process, system bandwidth B, and spectrum bandwidth distributed to each base station of B. k The bandwidth W of the sub-channel For WBAN user collection, Location of MEC server set and WBAN users MEC Location For the computational tasks of WBAN user n, D n Indicates the data size of the task, q n Indicates the number of CPU cycles required for the task, K n Indicates task priority, τ n Indicates the expected completion time of the task and the average channel gain h. n,m (t), data rate r n,m The time T required for the task n (t) and energy consumption E n (t);

[0071] (1) The average channel gain h between WBAN user n and MEC server m in the current time slot t. n,m as follows:

[0072]

[0073] Where d n,m (t) represents the distance from the current WBAN to the MEC server, α represents the path loss exponent, and h0 represents the reference gain;

[0074] (2) Data rate r between WBANn and MEC server m n,m as follows:

[0075]

[0076] Where N0 represents the Gaussian noise variance. The fixed subcarrier power of WBANn;

[0077] (3) The time T required for WBAN user n to complete the task at time t n (t) is:

[0078]

[0079] Where q n D represents the number of CPU cycles required for a task. n Indicates the size of the task's data; r n Y is the transmission rate between the MEC server and the cloud server when user n offloads a task to the cloud layer through the edge layer; m X represents the number of CPU cores in MEC server m. n f is the number of CPU cores for WBAN user n; c Indicates the computing frequency of the remote cloud server; f u and These represent the CPU computing frequency of a single core in WBAN users and MEC, respectively, which is the number of CPU cycles per second. Indicates the return time for task processing; when o n When (t) = 0, the task is executed locally, T n This is equivalent to the local task computation time; when o n When (t) = 1, the task is offloaded to the MEC server, T n The sum of task unloading time, MEC computation time, and return time; when o n (t) = -1, the task is unloaded to a remote cloud server, T n The time for the task to be unloaded to the MEC server, the time for the task to be unloaded from the MEC server to the cloud server, and the cloud server computation time;

[0080] (4) Energy consumption of WBAN

[0081] Where e0 is the CPU energy consumption coefficient, o n (t)≠0 indicates that the WBANn task is offloaded to the MEC server or cloud server, and the energy consumption at this time is the transmission energy consumption; when o n When (t) = 0, the task is executed locally, and its energy consumption is the local computation energy consumption.

[0082] Furthermore, in one embodiment, the Markov decision problem model described in step 2 includes a state space S(t), an action space A(t), and a reward function R(t). This computational unloading problem is viewed as finding the optimal task unloading strategy π. * This maximizes long-term cumulative rewards;

[0083] Step 2-1, calculate the state space as follows:

[0084]

[0085] Where the matrix Represents a set of N WBAN tasks; matrix Represents the set of locations of all WBAN users; matrix Represents the remaining task quantity for all WBANs, matrix R = [r n,m ] N×M To represent the set of communication rate indices between the MEC server and WBAN; matrix Represents the set of remaining energy states of the MEC server;

[0086] Step 2-2, calculate the motion space as follows:

[0087] A(t)={u n,m (t),o n (t)|n=1,2,…N; m=1,2,…M}

[0088] Where u n,m (t)∈{0,1} represents the connection action, o n (t)∈{0,-1,1} represents the unloading action; when u n,m When WBAN=1, WBANn connects to MEC server m; otherwise, u n,m =0; and Only when u n,m WBANn will only offload its tasks to the MEC server m when = 1; when o n When (t) = 0, the task is executed locally; while o n When (t) = 1, the task is offloaded to the MEC server; while o nWhen (t) = -1, the task is unloaded to a remote cloud server;

[0089] Steps 2-3 involve calculating the reward function R(t). The goal of the edge network task offloading problem is to minimize the overall execution time and energy of WBAN users by effectively utilizing network communication and computing resources. This means selecting connection and offloading actions in a given state s(t) to optimize system performance. The reward for all tasks in time slot t is represented as follows:

[0090]

[0091] Among them, the task cost C n The processing delay T for the task n And WBAN energy consumption E n linear combination, λ represents the time and energy costs required to execute this mission. t and λ e These are the delay factor and the energy consumption factor, respectively.

[0092] Furthermore, in one embodiment, step 3 of training the DDPG algorithm includes the following steps:

[0093] Step 3-1: Initialize algorithm parameters, WBAN user locations, task information, and remaining energy;

[0094] Step 3-2: Train the decision network parameters. In the previous training step, the μ output parameter μ(s) of the Actor main network... i Since the DDPG algorithm is an offline learning strategy, action noise n is added to facilitate more effective action space exploration. i Obtain the actual output action a i =μ(s) i )+n i ,in μ e and σ e,i These are the mean and standard deviation, respectively.

[0095] Step 3-3: In an environment with multiple edge servers and multiple WBAN users, the agent selects action a. i Observe the next state s i+1 and instant rewards i Then the sequence (s) i ,a i ,s i+1 ,r i Store the data in the experience replay pool. Determine if the experience replay pool is full. If it is, proceed to the next step; otherwise, return to step 3-2.

[0096] Steps 3-4 involve updating the Actor main network and Critic main network by selecting K batch sizes of sequences from the experience replay pool. The Actor target network outputs μ'(s). i+1 The Critic target network calculates the target value y based on the output of the Actor target network. i ;

[0097] Steps 3-5: Update the parameters of the decision network over multiple training cycles. If the user device runs out of energy or the task ends, terminate the training cycle early. After each training cycle, determine whether the reward function of the decision network has converged. If it has, end the training; otherwise, return to step 3-4.

[0098] Furthermore, in one embodiment, step 4, which involves calculating the unloading based on the trained decision network, specifically involves: setting the current state s... i (t) Input the decision network and output action a. i (t), which refers to the appropriate user connection strategy and task offloading strategy.

[0099] In one embodiment, an eHealth Network computation offloading system based on the DDPG algorithm is provided, the system comprising:

[0100] The network construction module is used to determine the eHealth network architecture and initialize network parameters.

[0101] The model building module is used to transform the computational offloading problem into a Markov decision problem and determine the state space, action space and reward function in the Markov decision model.

[0102] The training module is used to train the computational offloading problem using the Deep Deterministic Policy Gradient (DDPG) algorithm, update the network parameters, and output the trained decision network until the reward function converges during the entire operation.

[0103] The unloading module is used to perform unloading calculations based on the trained decision network.

[0104] Specific limitations regarding the DDPG-based eHealth Network computation offloading system can be found in the above section on the limitations of the DDPG-based eHealth Network computation offloading method, and will not be repeated here. Each module in the aforementioned DDPG-based eHealth Network computation offloading system can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the computer device's memory as software, so that the processor can call and execute the corresponding operations of each module.

[0105] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0106] Step 1: Determine the electronic health network architecture and initialize network parameters;

[0107] Step 2: Transform the computational unloading problem into a Markov decision problem, and determine the state space, action space, and reward function in the Markov decision model;

[0108] Step 3: Train the computational unloading problem using the Deep Deterministic Policy Gradient (DDPG) algorithm, update the network parameters until the reward function converges throughout the entire operation, and output the trained decision network.

[0109] Step 4: Calculate the unloading based on the trained decision network.

[0110] For specific limitations on each step, please refer to the limitations on the electronic health network computation offloading method based on the DDPG algorithm mentioned above, which will not be repeated here.

[0111] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0112] Step 1: Determine the electronic health network architecture and initialize network parameters;

[0113] Step 2: Transform the computational unloading problem into a Markov decision problem, and determine the state space, action space, and reward function in the Markov decision model;

[0114] Step 3: Train the computational unloading problem using the Deep Deterministic Policy Gradient (DDPG) algorithm, update the network parameters until the reward function converges throughout the entire operation, and output the trained decision network.

[0115] Step 4: Calculate the unloading based on the trained decision network.

[0116] For specific limitations on each step, please refer to the limitations on computational offloading of the eHealth Network based on the DDPG algorithm mentioned above, which will not be repeated here.

[0117] As a specific example, the invention will be further verified and illustrated in one embodiment.

[0118] according to Figure 2An electronic health network is established using the following architecture. Within a 500m × 500m space, the locations of WBAN users (N = 100) and MEC edge servers (M = 10) are randomly determined. The locations of the WBAN users follow an RGMM model and move, while the locations of the MEC servers are fixed. The transmit power of the WBAN users is 1W, the additive white Gaussian noise power is N0 = -100dBm, and the channel reference gain is h0 = 10. -4 The path loss exponent is set to α = 2. The uplink task data size is d. n Following a uniform distribution between 1 and 10 Mb, the return data volume is typically very small and negligible. Task priority K n It follows a uniform distribution between 1 and 7. The number of CPU cycles q required per unit task. n The range is 1000 to 10000.

[0119] Based on step 2, the state s(t), action a(t), and reward function R(t) of the unloading problem in the embodiment are modeled. Then, according to step 3, the decision network is trained using the DDPG algorithm. The parameters in the DDPG algorithm are set as follows: the capacity of the experience recall pool is designed to be D = 10000, the experience samples K = 64, the discount factor γ = 0.001, and the learning rate lr of the Actor and Critic. a and lr c The values ​​are 0.001 and 0.002 respectively.

[0120] During the training phase, the system state vector for each transmission is calculated, including the task set. Location set Remaining task quantity and the set of remaining energy states It is input into the decision network, outputs appropriate user connection and offloading strategies, records the state, actions, and reward values, and puts them into the experience memory pool D. n When D n When the capacity is full, randomly replace the experience samples in the smallest batch with the experience samples, calculate the target Q value, update the parameters of the decision network according to the gradient method, and record the average reward until the task ends.

[0121] Figure 3This diagram compares the task costs under different schemes in this embodiment, namely the local execution strategy, the random execution strategy, the AC learning strategy, and the DDPG learning strategy. The deep network algorithm is trained for a total of 300 episodes. In the task offloading problem of edge networks, the cost of the local execution strategy is consistently much higher than that of the other three strategies. In the initial stage, the cost of the random strategy is basically the same as that of the learning strategy (DDPG, AC), indicating that this offloading strategy algorithm cannot effectively utilize computational and communication resources, resulting in a higher cost than the learning strategy algorithm. Similar to DDPG, the framework of the AC algorithm consists of a policy gradient algorithm and a value function estimation. As can be seen from the figure, AC fails to converge with the increase of the number of training epochs, while DDPG converges around 150 episodes. This is because the Critic network in AC is inherently difficult to converge, and if it is updated together with the Actor network, it will be even more difficult to converge; while DDPG uses a dual network structure of evaluation network and target network, which can reduce the correlation of data, fully explore the action space, and obtain a near-optimal connection policy and offloading policy.

[0122] In summary, the method of this invention can rationally allocate communication and computing resources, solving the problem of limited local computing resources in mobile devices. This method can significantly reduce system latency and energy consumption, and can be widely applied in home healthcare, fitness, disease prevention and control, and medical monitoring.

[0123] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the invention without departing from its spirit and scope, and all such changes and modifications fall within the scope of the claimed invention.

Claims

1. A computational offloading method for electronic health networks based on the DDPG algorithm, characterized in that, The method includes the following steps: Step 1: Determine the electronic health network architecture and initialize network parameters; Step 2: Transform the computational unloading problem into a Markov decision problem, and determine the state space, action space, and reward function in the Markov decision model; Step 3: Train the computational unloading problem using the Deep Deterministic Policy Gradient (DDPG) algorithm, update the network parameters until the reward function converges throughout the entire operation, and output the trained decision network. Step 4: Calculate the unloading based on the trained decision network; The eHealth Network architecture described in Step 1 consists of three layers: the user layer, the edge layer, and the cloud layer. The user layer includes A mobile terminal device wireless body area network (WBAN) user, WBAN is used for wearable sensor devices to collect health data; WBAN users need to register with the blockchain network, and use a blockchain account as the unique identifier to join the blockchain network and become a legitimate participant; WBAN users can perform task calculations locally, or they can offload some tasks to a multi-access edge computing (MEC) server or a cloud server for execution. The edge layer includes macro base stations and Each MEC server contains a macro base station whose controller module implements various control functions for the system network, including hash resource allocation strategy, communication connection strategy, and task offloading strategy. The macro base station is used to monitor the blockchain network, and the MEC server provides real-time computing services to WBAN users, while also participating in blockchain consensus and receiving corresponding rewards. The cloud layer includes the cloud platform, where macro base stations in the edge layer schedule tasks to cloud servers for task offloading and computation; WBAN users purchase computing resources for data processing tasks through the blockchain, and transactions are executed by smart contracts, known as transaction contracts; The Markov decision model described in step 2 includes a state space. Action space Reward function This computational unloading problem is viewed as finding the optimal task unloading strategy. This maximizes long-term cumulative rewards; Step 2-1, calculate the state space as follows: Where the matrix Represents a set of N WBAN tasks; matrix Represents the set of locations of all WBAN users; matrix The matrix represents the remaining task quantity for all WBANs. To represent the set of communication rate indices between the MEC server and WBAN; matrix Represents the set of remaining energy states of the MEC server; Step 2-2, calculate the motion space as follows: in Indicates a connection action. Indicates the uninstallation action; when At that time, WBAN Connect to MEC server ,on the contrary ;and Only when At that time, WBAN Only then will its tasks be offloaded to the MEC server. Above; when At that time, the task is executed locally; while At that time, the task was offloaded to the MEC server; and At that time, the task was unloaded to a remote cloud server; Steps 2-3: Calculate the reward function , time slot The total cost of all tasks is represented as a reward for the system as follows: Among them, task cost For task processing delay and WBAN energy consumption linear combination, This indicates the time and energy costs required to execute this mission. and These are the delay factor and the energy consumption factor, respectively. Step 3, the DDPG algorithm training, includes the following steps: Step 3-1: Initialize algorithm parameters, WBAN user locations, task information, and remaining energy; Step 3-2: Train the decision network parameters. In the previous training step, the Actor main network... Output parameters Add motion noise Obtain the actual output action ,in , and These are the mean and standard deviation, respectively. Step 3-3: In an environment with multiple edge servers and multiple WBAN users, the agent selects an action. Observe the next state and instant rewards Then the sequence Store the data in the experience replay pool. Determine if the experience replay pool is full. If it is, proceed to the next step; otherwise, return to step 3-2. Steps 3-4: Select from the experience replay pool The Actor main network and Critic main network are updated with a batch size of sequences, and the Actor target network outputs... The Critic target network calculates the target value based on the output of the Actor target network. ; Steps 3-5: Update the parameters of the decision network over multiple training cycles. If the user device runs out of energy or the task ends, terminate the training cycle early. After each training cycle, determine whether the reward function of the decision network has converged. If it has, end the training; otherwise, return to step 3-4.

2. The electronic health network computational offloading method based on the DDPG algorithm according to claim 1, characterized in that, The parameters in the electronic health network architecture described in step 1 include several time slots throughout the communication process. System bandwidth B, and the spectrum bandwidth distributed to each base station are: Sub-channel bandwidth , For WBAN user collection, Location of MEC server set and WBAN users The location of MEC , For WBAN users Computational tasks Indicates the size of the task's data. Indicates the number of CPU cycles required for the task. Indicates task priority, Indicates the expected completion time of the task and the average channel gain. Data rate Time required for the task and energy consumption ; (1) Current time slot t, WBAN user and MEC server Average channel gain as follows: in This represents the distance from the current WBAN to the MEC server. This is the path loss index. Reference gain; (2) WBAN and MEC server Data rates between as follows: in This represents the variance of Gaussian noise. For WBAN Fixed subcarrier power; (3) WBAN users At any moment Time required to complete the task for: in This indicates the number of CPU cycles required for the task. Indicates the size of the task's data; When the user The transmission rate between the MEC server and the cloud server when tasks are offloaded to the cloud via the edge layer; MEC server The number of CPU cores, Is a WBAN user The number of CPU cores; Indicates the computing frequency of the remote cloud server; and These represent the CPU computing frequency of a single core in WBAN users and MEC, respectively, which is the number of CPU cycles per second. Indicates the return time for task processing; when At that time, the task is executed locally. This is equivalent to the local task computation time; when At that time, the task was unloaded to the MEC server. The sum of task unloading time, MEC computation time, and return time; when The task was unloaded to a remote cloud server. The time for the task to be unloaded to the MEC server, the time for the task to be unloaded from the MEC server to the cloud server, and the cloud server computation time; (4) Energy consumption of WBAN in CPU power consumption coefficient WBAN When a task is offloaded to an MEC server or cloud server, the energy consumption at this point is the transmission energy consumption; when When the task is executed locally, its energy consumption is the local computing energy consumption.

3. The electronic health network computational offloading method based on the DDPG algorithm according to claim 2, characterized in that, Step 4, which involves calculating the unloading based on the trained decision network, specifically involves: setting the current state... Input to a decision network, output as actions This refers to appropriate user connection strategies and task unloading strategies.

4. An eHealth Network computational offloading system based on the DDPG algorithm, implemented according to the method of any one of claims 1 to 3, characterized in that, The system includes: The network construction module is used to determine the eHealth network architecture and initialize network parameters; The model building module is used to transform the computational offloading problem into a Markov decision problem and determine the state space, action space, and reward function in the Markov decision model. The training module is used to train the computational offloading problem using the Deep Deterministic Policy Gradient (DDPG) algorithm, update the network parameters, and output the trained decision network until the reward function converges during the entire operation. The unloading module is used to perform unloading calculations based on the trained decision network.

5. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 3.

Images