Underwater computing task unloading method based on improved depth deterministic strategy gradient

Abstract

The invention discloses an underwater computing task unloading method based on an improved depth deterministic strategy gradient. The method comprises the following steps: firstly, initializing a state space of a task unloading problem, constructing an action space by using an unloading mode and an unloading quantity matrix plus noise, and designing a reward function by using a reciprocal of task unloading energy consumption; secondly, a depth deterministic strategy gradient algorithm is improved, a self-adaptive amplitude updating method is used in the algorithm parameter updating process, the updating amplitude is set to be a high value in the initial stage, and the updating amplitude is gradually reduced in the algorithm training process; and finally, solving a task unloading problem by improving a depth deterministic strategy gradient, inputting a Markov decision model of the task unloading problem, continuously training iteration until an average loss function is not reduced any more, and outputting a task unloading mode and an unloading quantity. According to the method, the convergence speed of the algorithm can be effectively increased, and the energy consumption of sensor node task unloading is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of edge computing technology, and specifically relates to an underwater computing task offloading method based on improved deep deterministic policy gradient. Background Art

[0002] The ocean occupies more than 71% of the earth's surface area. It is becoming increasingly important to build an Ocean of Things (OoT) to conduct three-dimensional observation of the ocean. The networking architecture of OoT generally includes two layers: sea surface and underwater. The sea surface layer is buoys, and the underwater layer is various sensor nodes. The sensor nodes transmit the collected data to the buoy nodes, and then the buoy nodes process the data and send it to the onshore monitoring center. With the development of OoT, the coverage area of ​​OoT is getting larger and larger, extending from the nearshore to the open ocean; at the same time, the computing services of sensor nodes in the network, such as data analysis and running network protocols, have increased sharply. Sensor nodes consume a lot of energy to process these computing services. The traditional OoT sea surface-underwater two-layer networking architecture can no longer meet the needs. Using satellite nodes to build a new space-sea surface-underwater three-layer OoT architecture, through the mobile edge computing (MEC) task offloading technology, the computing tasks generated by underwater sensor node data analysis and running network protocols are offloaded to buoys or satellite edge servers for processing, which has become a research hotspot.

[0003] Using traditional reinforcement learning algorithms to solve the problem of computing task offloading is a common method at present. In this method, the reinforcement learning model can be deployed on the buoy node for training. After the training is completed, the computing task offloading strategy is sent to the sensor nodes within the communication range of the buoy. The sensor nodes can offload their own computing tasks to the relevant edge servers according to the offloading strategy. However, the action space of the underwater computing task offloading problem cannot be expressed as a simple binary problem and needs to be further decomposed; and the impact of energy consumption is not considered in the design of the reward function of the reinforcement learning model, which easily leads to high energy consumption of computing task offloading. It should be noted that since satellite nodes and buoy nodes can usually be equipped with solar panels and charged by solar energy; underwater sensor nodes usually cannot directly replenish energy underwater under existing conditions, and they need to save power consumption as much as possible during work to maintain a longer working life cycle. Therefore, when performing underwater sensor task offloading, it is necessary to minimize energy consumption, including local computing energy consumption, energy consumption of sending tasks to the buoy node and receiving return results; when the task needs to be offloaded, the underwater sensor node will unload the data to the buoy node, and the buoy node will process the task and return the result; if the buoy node is busy and cannot process it, the calculation task will be further offloaded to the satellite node for processing.

[0004] The applicant Tongji University disclosed a method for edge computing collaborative task offloading based on multi-agent reinforcement learning in its patent application document "A method for edge computing collaborative task offloading based on multi-agent reinforcement learning" (application date: May 30, 2024, application number: 202410687523.1, application publication number: CN118585263 A, the content of this application can still be cited). This method for edge computing collaborative task offloading has the following shortcomings: the action space of its computing task offloading problem is represented as a simple 0 or 1 problem, that is, a binary problem, which cannot accurately describe the action of computing task offloading; and the design of the reward function does not consider the impact of energy consumption, resulting in high energy consumption of computing task offloading.

[0005] Using the Deep Deterministic Policy Gradient (DDPG) algorithm to solve the problem of computing task offloading is an effective method to reduce the energy consumption of task offloading in MEC systems. The applicant, Nanjing University of Science and Technology, disclosed a method for satellite edge computing task offloading and resource allocation based on deep reinforcement learning in its patent application document "Satellite edge computing task offloading and resource allocation method based on deep reinforcement learning" (application date: May 24, 2024, application number: 202410655864.0, application publication number: CN118250750A, the content of this application can still be cited). The satellite edge computing task offloading and resource allocation method has the following shortcomings: the deep reinforcement learning used is the DDPG algorithm, which is used to offload computing tasks. The DDPG algorithm uses a fixed value for the update amplitude during training, which results in a slow convergence of the algorithm when the amount of computing task offloading is large, and long-term training further exacerbates the energy consumption problem.

[0006] In summary, when traditional reinforcement learning-based methods are used in OoT, there are problems such as high energy consumption of computing task offloading and slow algorithm convergence. Summary of the invention

[0007] Purpose of the invention: The traditional reinforcement learning-based method solves the problem of computing task offloading. The action space is represented as a simple binary problem, which lacks randomness. The design of the reward function does not consider the impact of energy consumption, resulting in high energy consumption of computing task offloading. The DDPG algorithm uses a fixed update amplitude during training, resulting in a slow convergence speed. The present invention proposes an underwater computing task offloading method based on an improved deep deterministic policy gradient to make up for the shortcomings of the above technologies.

[0008] In order to achieve the above-mentioned purpose, the present invention adopts the following technical solutions: A method for offloading underwater computing tasks based on improved deep deterministic policy gradients, comprising the following steps: S1: Establishment of Markov decision model for computing task offloading: Describe the underwater computing task offloading problem as a Markov decision process, initialize the state space of the task offloading problem, construct the action space using the offloading mode and offloading quantity matrix, and design the reward function using the inverse of the computing task offloading energy consumption; S2: Improved deep deterministic policy gradient algorithm: First, replace the traditional minimization loss function with the minimization loss function for training, and then Parameters and targets of value functions In the process of updating the parameters of the value function, the adaptive update amplitude method is used to replace the traditional fixed update amplitude method to speed up the convergence of the algorithm; S3: Solve the computational task offloading problem by improving the deep deterministic policy gradient algorithm: Input the Markov decision model of the computational task offloading problem in step S1 into the improved deep deterministic policy gradient algorithm in S2, and continuously train the improved deep deterministic policy gradient algorithm in S2 until the average loss function no longer decreases, and output the computational task offloading mode and offloading quantity.

[0009] Furthermore, the step S1 is specifically as follows: S1-1: Initialize the state space of the task offloading problem: Deploy a three-layer network architecture of space-sea surface-underwater, including Low-orbit satellites, denoted as ; buoy nodes, denoted as ; Each buoy node is Underwater sensor nodes provide computing services, denoted as ;No. The buoy node and the The underwater sensor nodes are denoted as and ; Assume that both the low-orbit satellite and the buoy node have built-in edge servers that can perform data analysis and calculations. The buoy node is also equipped with an acoustic module and an electromagnetic wave module, which can communicate with the underwater sensor node and the satellite node at the same time. The underwater sensor node processes the computing tasks generated locally, or offloads the computing tasks to the edge server of the buoy node or the satellite node for calculation. The computing tasks generated by the underwater sensor node include data analysis and running network protocols. and They represent the total amount of computing tasks and the number of offloaded computing tasks owned by underwater sensor nodes respectively; When, representative The computing tasks need to be offloaded to the buoy node or satellite node, otherwise it means The computational tasks are processed locally by the underwater sensor nodes; define Indicates whether the task is offloaded to a buoy node or a satellite node, expressed as: (1) Underwater sensor nodes are networked underwater, and each underwater sensor node is regarded as an intelligent agent. In the initial stage, the intelligent body perceives the environment information, time slot The state at is composed of the states of all agents in the network, and the state space set It is expressed as: (2) , in, Representative Agents in time slot The state of being Contains the following sections: (3) , in, Represents link status information, Represents geographic location information, Represents resource allocation information, Represents the current computing task information, Represents computing resource information, Represents the remaining energy information; S1-2: Use the unloading pattern matrix and the unloading quantity matrix to add noise to construct the action space: set up R Pol =[ r 1 Pol ,..., r i Pol ,..., r I Pol ] Τ represents the uninstallation pattern matrix, and has r i Pol =[u n i,1 ,u n i,j ,...,u n i,J ] ; R Vol =[ r 1 Vol ,..., r i Vol ,..., r I Vol ] Τ represents the uninstall quantity matrix, and has r i Vol =[ g i,1 , g i,j ,..., g i,J ] ; will unload the matrix Defined as and Multiply the corresponding elements, then unload the matrix It is expressed as: (4) The action space is composed of the unloading matrix Composition, among which, instruct The computing task offloading mode, g i,j ∈ [0, G i,j ] instruct The number of computing tasks offloaded, and when When Tasks are processed locally, offloading the matrix Corresponding position elements Marked as negative value, i.e. , Indicates The buoy node manages The total amount of computing tasks that underwater sensor nodes have; time slot The original action space set It is expressed as: (5) in, Representative The buoy node manages indivual In time slot The optional actions are determined by the uninstall matrix Each row of It is expressed as: (6) To increase the randomness of the action space, add random noise to the original action space , the final action space It is expressed as: (7) in, represents the original action space, represents random noise; S1-3: Design reward function using the inverse of computing task offloading energy consumption: The inverse of the energy consumption of computing task offloading is used as a reward to guide the agent to gradually reduce the energy consumption of computing task offloading. The calculation formula of the reward function is: (8) , in, Indicates time slot Inside The energy consumption of local processing computing tasks, Indicates time slot Internal buoy node The energy consumption of the processing task, Indicates time slot Internal Satellite Node The energy consumption of the processing task, and Represents the additional rewards for buoy node business and edge server business respectively. When business is busy, The value is negative, otherwise The value is positive; when the edge server When business is busy, The value is negative, otherwise The value is a positive number.

[0010] Furthermore, the step S2 is specifically as follows: S2-1: Calculate the current state of the improved deep deterministic policy gradient Value function value, algorithm training is performed by minimizing the average loss function: Improving Deep Deterministic Policy Gradients Current The calculation formula of the value function is: (9) in, Represents the current The parameters of the value function; represents the discount factor, The value range is between 0 and 1. The closer the value is to 1, the more attention is paid to the future. value; Represents the current policy function; represents the reward function; The current state representing the next state Value function, target The calculation formula of the value function is: (10) in, Representative target The parameters of the value function; represents the target policy function; The target for the next state Value function; algorithm training is performed by minimizing the average loss function, the average loss function The calculation formula is: (11) , in, Represents the number of all agents in the network; S2-2: Improve DDPG algorithm parameters and The adaptive update amplitude method is used in the update process. The update amplitude is set to a larger value in the initial stage and gradually reduced during the training process of the improved DDPG algorithm: Update the parameters in step S2-1 using the adaptive update amplitude method and , the calculation formula is: (12) , in, is the update amplitude, between 0 and 1, The smaller the value, the slower the algorithm updates. In order to stabilize the training process and ensure that the improved DDPG algorithm can be updated stably while achieving faster convergence, the update amplitude is set to Set it to a larger value and gradually reduce it during the training process of the improved DDPG algorithm. The calculation formula is: (13) in, and Represent the initial and later update amplitudes respectively. According to formula (13), the adaptive update amplitude The value will be from Gradually changes to Once in the process of change or , then immediately switch to ; It is the update amplitude control threshold. After many experiments, it is determined The value range is between 0.5 and 2.5. The smaller the value, the smaller the initial update amplitude; Represents the switching time threshold. After many experiments, it is determined The value range is between 1 and 10. A larger value means a longer update amplitude switching time.

[0011] Furthermore, in step S3, an improved deep deterministic policy gradient algorithm is used to solve the problem of computing task offloading, as follows: The computational task offloading problem is solved by the improved deep deterministic policy gradient algorithm in S2, and the Markov decision model of the computational task offloading problem in S1 is input; according to the current state and the current policy function Select Action , calculates the current value of the deep deterministic policy gradient algorithm Value function value; according to the current Value Function Value and Objective Value function value calculates the average loss function and updates the parameters and ; Repeat the above process to continuously reduce the average loss function And the cumulative reward function , until the average loss function no longer decreases, the output calculation unloading mode and calculate the number of uninstalls .

[0012] The advantages and technical effects of the present invention are as follows: After simulation verification, the improved DDPG algorithm in the present invention has a significantly faster convergence speed than the method with fixed update amplitude in the traditional DDPG algorithm; compared with the deep Q learning (Deep Q-Network, DQN) algorithm, the energy consumption of underwater sensor node computing task offloading is significantly reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is an overall flow chart of an embodiment of the present invention; Figure 2 This is a space-sea surface-underwater network architecture diagram of an embodiment of the present invention; Figure 3 is a schematic diagram of classification of underwater computing task offloading modes according to an embodiment of the present invention; Figure 4 This is an underwater computing task offloading architecture diagram of an embodiment of the present invention; Figure 5 This is a simulation diagram comparing the loss function of the method of an embodiment of the present invention and the traditional fixed update amplitude method with the change of training rounds; Figure 6 This is a simulation diagram comparing the changes in task amount in calculating task offloading energy consumption using the method and the DQN method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0015] This embodiment proposes an underwater computing task offloading method based on improved deep deterministic policy gradient. The overall flow chart is as follows: Figure 1 As shown, the following steps are included: S1: Establishment of Markov decision model for computing task offloading: Describe the underwater computing task offloading problem as a Markov decision process, initialize the state space of the task offloading problem, use the offloading mode and offloading quantity matrix to add noise to construct the action space, and use the inverse of the computing task offloading energy consumption to design the reward function. The specific steps are as follows: S1-1: Initialize the state space of the task offloading problem: like Figure 2 As shown in Figure 1, a three-layer network architecture of space-sea surface-underwater is deployed, including two satellite nodes, denoted as , 3 buoy nodes, denoted as , each buoy node provides computing services for three underwater sensor nodes, denoted as , all underwater sensor nodes are evenly distributed in 1000×1000×1000 m 3 In the region, The buoy node and the The underwater sensor nodes are denoted as and ,in and ; Both satellite nodes and buoy nodes have built-in edge servers that can perform data analysis and calculations. Buoy nodes are equipped with acoustic modules and electromagnetic wave modules, which can communicate with underwater sensor nodes and satellite nodes at the same time; The classification of underwater computing task offloading modes is shown as follows Figure 3 As shown in Figure 1, underwater sensor nodes can process the generated computing tasks, such as data analysis and running network protocols, locally, or offload the computing tasks to edge servers of buoy nodes or satellite nodes for computing. and They represent the total amount of computing tasks and the number of offloaded computing tasks owned by underwater sensor nodes respectively; When, representative The computing tasks need to be offloaded to the buoy node or satellite node, otherwise it means The computational tasks are processed locally by the underwater sensor nodes; define Indicates whether the task is offloaded to a buoy node or a satellite node, expressed as: (1) Note that when When it is always 1, the task will only be unloaded to the buoy node. At this time, the three-layer network architecture of space-sea surface-underwater becomes a two-layer network architecture of sea surface-underwater. Therefore, the present invention is also applicable to the two-layer architecture of sea surface-underwater.

[0016] Underwater sensor nodes are networked underwater, and each underwater sensor node is regarded as an intelligent agent. In the initial stage, the intelligent body perceives the environment information, time slot The state at is composed of the states of all agents in the network, and the state space set It is expressed as: (2) in, Representative Agents in time slot The state of being Contains the following sections: (3) , in, Represents link status information, Represents geographic location information, Represents resource allocation information, Represents the current computing task information, Represents computing resource information, Represents the remaining energy information; S1-2: Use the unloading pattern matrix and the unloading quantity matrix to add noise to construct the action space: set up R Pol =[ r 1 Pol ,..., r i Pol ,..., r I Pol ] Τ represents the uninstallation pattern matrix, and has r i Pol =[u n i,1 ,u n i,j ,...,u n i,J ] ; R Vol =[ r 1 Vol ,..., r i Vol ,..., r I Vol ] Τ represents the uninstall quantity matrix, and has r i Vol =[ g i,1 , g i,j ,..., g i,J ] ; will unload the matrix Defined as and Multiply the corresponding elements, then unload the matrix It is expressed as: (4) The action space is composed of the unloading matrix Composition, among which, instruct The computing task offloading mode, g i,j ∈ [0, G i,j ] instruct The number of computing tasks offloaded, and when When Tasks are processed locally, offloading the matrix Corresponding position elements Marked as negative value, i.e. , Indicates The buoy node manages The total amount of computing tasks that underwater sensor nodes have; time slot The original action space set It is expressed as: (5) in, Representative The buoy node manages indivual In time slot The optional actions are determined by the uninstall matrix Each row of It is expressed as: (6) To increase the randomness of the action space, add random noise to the original action space , the final action space It is expressed as: (7) in, represents the original action space, represents random noise.

[0017] S1-3: Design reward function using the inverse of computing task offloading energy consumption: The inverse of the energy consumption of computing task offloading is used as a reward to guide the agent to gradually reduce the energy consumption of computing task offloading. The calculation formula of the reward function is: (8) , in, Indicates time slot Inside The energy consumption of local processing computing tasks, Indicates time slot Internal buoy node The energy consumption of the processing task, Indicates time slot Internal Satellite Node The energy consumption of the processing task, and Represents the additional rewards for buoy node business and edge server business respectively. When business is busy, The value is negative, otherwise The value is positive; when the edge server When business is busy, The value is negative, otherwise The value is a positive number.

[0018] S2: Improved deep deterministic policy gradient algorithm: First, the training is performed by replacing the traditional minimization loss function with the minimization average loss function. Secondly, the adaptive update amplitude method is used to replace the traditional fixed update amplitude method in the process of updating the parameters of the current value function and the target value function to speed up the convergence of the algorithm. The specific steps are as follows: S2-1: Calculate the current state of the improved deep deterministic policy gradient Value function value, algorithm training is performed by minimizing the average loss function: Improving Deep Deterministic Policy Gradients Current The calculation formula of the value function is: (9) in, Represents the current The parameters of the value function; represents the discount factor, The value range is between 0 and 1. The closer the value is to 1, the more attention is paid to the future. value; Represents the current policy function; represents the reward function; The current state representing the next state Value function, target The calculation formula of the value function is: (10) in, Representative target The parameters of the value function; represents the target policy function; The target for the next state Value function; algorithm training is performed by minimizing the average loss function, the average loss function The calculation formula is: (11) , in, Represents the number of all agents in the network.

[0019] S2-2: Improve DDPG algorithm parameters and The adaptive update amplitude method is used in the update process. The update amplitude is set to a higher value in the initial stage and gradually reduced during the training process of the improved DDPG algorithm: Update the parameters in step S2-1 using the adaptive update amplitude method and , the calculation formula is: (12) , in, is the update amplitude, between 0 and 1, The smaller the value, the slower the algorithm updates. In order to stabilize the training process and ensure that the improved DDPG algorithm can be updated stably while achieving faster convergence, the update amplitude is set to Set it to a larger value and gradually reduce it during the training process of the improved DDPG algorithm. The calculation formula is: (13) in, and Represent the initial and later update amplitudes respectively. According to formula (13), the adaptive update amplitude The value will be from Gradually changes to Once in the process of change or , then immediately switch to ; It is the update amplitude control threshold. After many experiments, it is determined The value range is between 0.5 and 2.5. The smaller the value, the smaller the initial update amplitude. ; Represents the switching time threshold. After many experiments, it is determined The value range is between 1 and 10. The larger the value, the longer the update amplitude switching time. Hour.

[0020] S3: Solve the computing task offloading problem by improving the deep deterministic policy gradient algorithm: Input the Markov decision model of the computing task offloading problem in step S1 into the improved deep deterministic policy gradient algorithm in S2, and continuously train the improved deep deterministic policy gradient algorithm in S2 until the average loss function no longer decreases, and output the computing task offloading mode and offloading quantity. The specific steps are as follows: The computational task offloading problem is solved by the improved deep deterministic policy gradient algorithm in S2. The Markov decision model of the computational task offloading problem in S1 is input; according to the current state and the current policy function Select Action , calculates the current value of the deep deterministic policy gradient algorithm Value function value; according to the current Value Function Value and Objective Value function value calculates the average loss function and updates the parameters and ; Repeat the above process to continuously reduce the average loss function And the cumulative reward function , until the average loss function no longer decreases, the output calculation unloading mode and calculate the number of uninstalls .

[0021] The simulation results of loss function with the number of training steps using the method provided by this embodiment and the traditional method with fixed update amplitude are compared. Figure 5 As shown, this method uses an adaptive update amplitude, the initial update amplitude and later update range Take separately and , calculate the update amplitude according to formula (13) in the embodiment , adaptive update amplitude The value will gradually change from 0.7 to 0.1. or hours, then switch immediately to The two methods compared are the traditional fixed update amplitude method and the fixed update amplitude method. No change, compared with method 2 fixed The simulation of this embodiment is completed in Matlab R2021a, the computer memory used is 32GB, and the x64-based Intel Core i5-12400F CPU processor is used. The specific parameters of all simulations in this embodiment are listed in Table 1.

[0022] Table 1 Simulation parameters

[0023] from Figure 5 The simulation results show that using a fixed The algorithm converges slowly when the value is used. This method is better than the fixed value, because in the early stage of network training, a larger update amplitude can approach the optimal solution faster; as the training progresses, reducing the update amplitude value can make the algorithm converge more stably. When using a variable update amplitude, Figure 5 The curve in the figure represents that this method reaches convergence after 450 rounds of training, and the final loss value is relatively stable; compared with method 1 with a fixed update amplitude The curve converges after 1000 rounds of training; compared with method 2 with fixed update amplitude The curve converges after 600 rounds of training, and the final loss value oscillates back and forth. This method is compared with the fixed update amplitude of the comparison method 1. Accelerated the convergence speed by 55%; compared with the fixed update amplitude of comparison method 2 The convergence speed has been accelerated by 25%, which can achieve faster convergence while ensuring stable network updates.

[0024] Comparison of simulation results of network offloading delay with task volume using the present invention and traditional reinforcement learning method (DQN) Figure 6 As shown in Figure 2, the DQN algorithm is a reinforcement learning algorithm combined with deep learning, and is often used to solve the problem of high energy consumption in underwater computing task offloading.

[0025] from Figure 6 It can be seen from the simulation results that with the increase of task size, the computing task offloading energy consumption of the two methods shows an increasing trend, and the method provided in this embodiment is less affected. When the task size is 800 KB, the method provided in this embodiment reduces the computing task offloading energy consumption by about 19.2% compared with the DQN method.

[0026] In summary, the present invention can effectively accelerate the convergence speed of the algorithm and reduce the energy consumption of sensor node task offloading.

[0027] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for a person skilled in the art to modify the technical solutions described in the aforementioned embodiments, or to replace some of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions claimed to be protected by the present invention.

Claims

1. A method for offloading underwater computing tasks based on improved deep deterministic policy gradient, characterized in that: The method comprises the following steps: S1: Establishment of Markov decision model for computing task offloading: Describe the underwater computing task offloading problem as a Markov decision process, initialize the state space of the task offloading problem, construct the action space using the offloading mode and offloading quantity matrix, and design the reward function using the inverse of the computing task offloading energy consumption; S2: Improved deep deterministic policy gradient algorithm: First, replace the traditional minimization loss function with the minimization loss function for training, and then Parameters and targets of value functions In the process of updating the parameters of the value function, the adaptive update amplitude method is used to replace the traditional fixed update amplitude method to speed up the convergence of the algorithm; S3: Solve the computational task offloading problem by improving the deep deterministic policy gradient algorithm: Input the Markov decision model of the computational task offloading problem in step S1 into the improved deep deterministic policy gradient algorithm in S2, and continuously train the improved deep deterministic policy gradient algorithm in S2 until the average loss function no longer decreases, and output the computational task offloading mode and offloading quantity.

2. The underwater computing task offloading method based on improved deep deterministic policy gradient according to claim 1, characterized in that: The step S1 is specifically as follows: S1-1: Initialize the state space of the task offloading problem: Deploy a three-layer network architecture of space-sea surface-underwater, including Low-orbit satellites, denoted as ; buoy nodes, denoted as ; Each buoy node is Underwater sensor nodes provide computing services, denoted as ;No. The buoy node and the The underwater sensor nodes are denoted as and ; Assume that both the low-orbit satellite and the buoy node have built-in edge servers that can perform data analysis and calculations, and the buoy node is equipped with both an acoustic module and an electromagnetic wave module, which can communicate with the underwater sensor node and the satellite node at the same time; The underwater sensor nodes process the computing tasks generated locally, or offload the computing tasks to the edge servers of the buoy nodes or satellite nodes for computing. The computing tasks generated by the underwater sensor nodes include data analysis and running network protocols. and They represent the total amount of computing tasks and the number of offloaded computing tasks owned by underwater sensor nodes respectively; When, representative The computing tasks need to be offloaded to the buoy node or satellite node, otherwise it means The computational tasks are processed locally by the underwater sensor nodes; define Indicates whether the task is offloaded to a buoy node or a satellite node, expressed as: (1)， Underwater sensor nodes are networked underwater, and each underwater sensor node is regarded as an intelligent agent. In the initial stage, the intelligent body perceives the environment information, time slot The state at is composed of the states of all agents in the network, and the state space set It is expressed as: (2) ， in, Representative Agents in time slot The state of being Contains the following sections: (3) ， in, Represents link status information, Represents geographic location information, Represents resource allocation information, Represents the current computing task information, Represents computing resource information, Represents the remaining energy information; S1-2: Use the unloading pattern matrix and the unloading quantity matrix to add noise to construct the action space: set up R Pol =[ r 1 Pol ,..., r i Pol ,..., r I Pol ] Τ represents the uninstallation pattern matrix, and has r i Pol =[u n i,1 ,u n i,j ,...,u n i,J ] ; R Vol =[ r 1 Vol ,..., r i Vol ,..., r I Vol ] Τ represents the uninstall quantity matrix, and has r i Vol =[ g i,1 , g i,j ,..., g i,J ] ; will unload the matrix Defined as and Multiply the corresponding elements, then unload the matrix It is expressed as: R= u n 1,1 g 1,1 u n 1,2 g 1,2 ⋯ u n 1,J g 2,J u n 2,1 g 2,1 u n 2,2 g 2,2 ⋯ u n 2,J g 2,J ⋮ ⋮ ⋱ ⋮ u n I,1 g I,1 u n I,2 g I,2 ⋯ u n I,J g I,J = a ̃ 1,1 a ̃ 1,2 ⋯ a ̃ 1,J a ̃ 2,1 a ̃ 2,2 ⋯ a ̃ 2,J ⋮ ⋮ ⋱ ⋮ a ̃ I,1 a ̃ I,2 ⋯ a ̃ I,J (4)， The action space is composed of the unloading matrix Composition, among which, instruct The computing task offloading mode, g i,j ∈ [0, G i,j ] instruct The number of computing tasks offloaded, and when When Tasks are processed locally, offloading the matrix Corresponding position elements Marked as negative value, i.e. , Indicates The buoy node manages The total amount of computing tasks that underwater sensor nodes have; time slot The original action space set It is expressed as: (5)， in, Representative The buoy node manages indivual In time slot The optional actions are determined by the uninstall matrix Each row of It is expressed as: (6)， To increase the randomness of the action space, add random noise to the original action space , the final action space It is expressed as: (7)， in, represents the original action space, represents random noise; S1-3: Design reward function using the inverse of computing task offloading energy consumption: The inverse of the energy consumption of computing task offloading is used as a reward to guide the agent to gradually reduce the energy consumption of computing task offloading. The calculation formula of the reward function is: (8) ， in, Indicates time slot Inside The energy consumption of local processing computing tasks, Indicates time slot Internal buoy node The energy consumption of the processing task, Indicates time slot Internal Satellite Node The energy consumption of the processing task, and Represents the additional rewards for buoy node business and edge server business respectively. When business is busy, The value is negative, otherwise The value is a positive number; when the edge server When business is busy, The value is negative, otherwise The value is a positive number.

3. The underwater computing task offloading method based on improved deep deterministic policy gradient according to claim 1, characterized in that: The step S2 is specifically as follows: S2-1: Calculate the current state of the improved deep deterministic policy gradient Value function value, algorithm training is performed by minimizing the average loss function: Improving Deep Deterministic Policy Gradients Current The calculation formula of the value function is: (9)， in, Represents the current The parameters of the value function; represents the discount factor, The value range is between 0 and 1. The closer the value is to 1, the more attention is paid to the future. value; Represents the current policy function; represents the reward function; The current state representing the next state Value function, target The calculation formula of the value function is: (10)， in, Representative target The parameters of the value function; represents the target policy function; The target for the next state Value function; algorithm training is performed by minimizing the average loss function, the average loss function The calculation formula is: (11)， in, Represents the number of all agents in the network; S2-2: Improve DDPG algorithm parameters and The adaptive update amplitude method is used in the update process. The update amplitude is set to the maximum value in the initial stage and gradually reduced during the training process of the improved DDPG algorithm: Update the parameters in step S2-1 using the adaptive update amplitude method and , the calculation formula is: (12)， in, is the update amplitude, between 0 and 1, The smaller the value, the slower the algorithm updates. In order to stabilize the training process and ensure that the improved DDPG algorithm can be updated stably while achieving faster convergence, the update amplitude is set to Set the maximum value and gradually reduce it during the training process of the improved DDPG algorithm. The calculation formula is: (13)， in, and Represent the initial and later update amplitudes respectively. According to formula (13), the adaptive update amplitude The value will be from Gradually changes to , once in the process of change or , then immediately switch to ; is the update amplitude control threshold, The value range is between 0.5 and 2.

5. The smaller the value, the smaller the initial update amplitude; represents the switching time threshold, The value range is between 1 and 10. A larger value means a longer update amplitude switching time.

4. The underwater computing task offloading method based on improved deep deterministic policy gradient according to claim 1, characterized in that: In step S3, an improved deep deterministic policy gradient algorithm is used to solve the problem of computing task offloading, as follows: The computational task offloading problem is solved by the improved deep deterministic policy gradient algorithm in S2, and the Markov decision model of the computational task offloading problem in S1 is input; according to the current state and the current policy function Select Action , calculates the current value of the deep deterministic policy gradient algorithm Value function value; according to the current Value Function Value and Objective Value function value calculates the average loss function and updates the parameters and ; Repeat the above process to continuously reduce the average loss function And the cumulative reward function , until the average loss function no longer decreases, the output calculation unloading mode and calculate the number of uninstalls .

Images