Edge computing task processing method, device and electronic equipment

Abstract

The embodiment of the present invention proposes an edge computing task processing method, device and electronic device, which belongs to the field of edge computing. The method includes: according to a preset first reinforcement learning network about multiple mobile devices, obtaining unloading strategy experience of various edge computing environments, and putting each group of unloading strategy experience into an experience pool, so that for the pending task of each mobile device, according to the environmental state of the mobile device, the target experience matching the environmental state is retrieved from the experience pool, and then according to the environmental state and target experience of the mobile device, the preset decision model is used to obtain the unloading decision of the pending task of the mobile device, so as to realize the use of different target experiences and environmental states as considerations for unloading decisions for edge computing environments of different mobile devices, thereby improving the problem that the existing task computing offloading method is usually only applicable to a certain type of edge computing environment and has poor applicability to different edge computing environments.

Description

Technical Field

[0001] The present invention relates to the field of edge computing, and in particular to an edge computing task processing method, device and electronic device. Background Art

[0002] Mobile edge computing provides powerful computing power and energy resources for mobile devices by setting up edge servers in edge computing networks. Mobile devices can choose to offload computing-intensive tasks or latency-critical tasks to edge servers to reduce task execution latency and save battery energy consumed by local devices.

[0003] One of the key issues that need to be addressed in mobile edge computing networks is the computation offloading problem, which is the choice of whether a mobile device should offload computing tasks to a nearby MEC server or execute them locally. Currently, the task computation offloading methods used to handle computation offloading problems are usually only applicable to a certain type of edge computing environment, and have poor applicability to different edge computing environments. Summary of the invention

[0004] The purpose of the present invention is to provide an edge computing task processing method, device and electronic device, which can improve the problem that the existing task computing offloading methods for handling computing offloading problems are usually only applicable to a certain type of edge computing environment and have poor applicability to different edge computing environments.

[0005] In order to achieve the above objectives, the technical solutions adopted by the embodiments of the present invention are as follows.

[0006] In a first aspect, an embodiment of the present invention provides an edge computing task processing method, the method comprising:

[0007] According to a preset first reinforcement learning network for multiple mobile devices, offloading strategy experiences of various edge computing environments are obtained, and each group of offloading strategy experiences is put into an experience pool;

[0008] For each task to be processed of the mobile device, according to the environmental state of the mobile device, a target experience matching the environmental state is retrieved from the experience pool;

[0009] According to the environmental state of the mobile device and the target experience, a preset decision model is used to obtain an offloading decision of the pending task of the mobile device.

[0010] Furthermore, the step of obtaining a decision action for a task to be processed of the mobile device by using a preset decision model according to the environmental state of the mobile device and the target experience includes:

[0011] Based on the target experience and the environmental state of the mobile device, using a second reinforcement learning network, obtaining a first state transition tuple;

[0012] The first state transition tuple and the environment state are used as inputs of the decision model to obtain an offloading decision for the task to be processed.

[0013] Further, the decision model includes a first encoder, a second encoder, an evaluator and a first decoder;

[0014] The step of taking the first state transition tuple and the environment state as inputs of the decision model to obtain the decision action of the task to be processed includes:

[0015] Taking the environment state as input of the first encoder, obtaining a strategy vector;

[0016] Taking the first state transition tuple and the environment state as inputs of the second encoder, obtaining a state transition vector;

[0017] Using the strategy vector and the state transition vector as inputs of the evaluator to obtain an optimized strategy vector;

[0018] The optimized strategy vector is used as the input of the first decoder to obtain the offloading decision of the task to be processed.

[0019] Furthermore, the method further comprises the step of training a supervised learning model to obtain a decision model, wherein the supervised learning model comprises a first encoder, a first decoder, a second encoder, a second decoder and an evaluator;

[0020] The step of training the supervised learning model to obtain the decision model includes:

[0021] Based on the experience of the unloading strategy, multiple groups of state actions are obtained, each group of state actions includes an environment state and a decision action;

[0022] Iteratively updating the parameters of the first encoder and the first decoder using the multiple sets of the state actions until a first end condition is met;

[0023] Based on the second reinforcement learning network, using the offloading strategy experience, obtaining a second state transition tuple about the environment interaction of each edge computing environment;

[0024] Iteratively updating the parameters of the second encoder and the second decoder using the second state transition tuple until a second end condition is met;

[0025] Using the trained output of the first encoder and the trained output of the second encoder as inputs of the evaluator, and iteratively updating parameters of the evaluator until a third termination condition is met;

[0026] Based on the trained first encoder, first decoder, second encoder and evaluator, a decision model is obtained.

[0027] Furthermore, the step of using the trained output of the first encoder and the output of the second encoder as inputs of the evaluator and iteratively updating the parameters of the evaluator until a third termination condition is met includes:

[0028] Based on all the state actions and the offloading strategy experience, calculate the average action value of each state of each type of edge computing environment;

[0029] In each iteration, the output of the first encoder and the output of the second encoder that have been trained are used as inputs of the evaluator to obtain an average benefit value predicted by the evaluator, where the average benefit value represents the average action value of multiple states of the current edge computing environment;

[0030] The loss value between the average benefit value and the average action value is calculated. If the loss value satisfies the third termination condition, the iteration is terminated; otherwise, the iteration is continued until the loss value satisfies the third termination condition.

[0031] Furthermore, the step of retrieving target experience matching the environmental state from the experience pool according to the environmental state of the mobile device includes:

[0032] Using a clustering algorithm, the unloading strategy experience in the experience pool is divided into a plurality of strategy families;

[0033] Calculating the distance between the environment state and each of the strategy families, and taking the strategy family with the smallest distance among the multiple strategy families as the target strategy family;

[0034] An uninstallation strategy experience is selected from the target strategy family as a target experience.

[0035] Furthermore, the method further comprises the step of training a first reinforcement learning network, the step comprising:

[0036] Construct an edge computing network system model including multiple mobile devices;

[0037] Based on the edge computing network system model, construct an optimization objective function for computing cost;

[0038] Based on the mobile edge computing network system model, construct a first reinforcement learning network, wherein the first reinforcement learning network includes a sequentially connected graph neural network, an actor network, a critic network, and a hybrid network;

[0039] The first reinforcement learning network is trained in combination with the optimization objective function and the environmental state of the edge computing environment of each mobile device.

[0040] Furthermore, the optimization objective function includes a cost optimization function, an offloading mode constraint, an energy consumption constraint, a delay constraint, and a transmission power constraint;

[0041] The step of training the first reinforcement learning network in combination with the optimization objective function and the environmental state of the edge computing environment of each mobile device includes:

[0042] Based on the optimization objective function, construct a loss function;

[0043] The environmental state of the edge computing environment of each of the mobile devices is used as an input of the first reinforcement learning network, and the first reinforcement learning network is trained in combination with the loss function.

[0044] In a second aspect, an embodiment of the present invention provides an edge computing task processing device, including a first preparation module, a second preparation module and a processing module;

[0045] The first preparation module is used to obtain the offloading strategy experience of various edge computing environments according to the preset first reinforcement learning network about multiple mobile devices, and put each group of the offloading strategy experience into the experience pool;

[0046] The second preparation module is used to retrieve, for each task to be processed of the mobile device, target experience matching the environmental state from the experience pool according to the environmental state of the mobile device;

[0047] The processing module is used to obtain an offloading decision of the to-be-processed task of the mobile device by using the decision model according to the environmental state of the mobile device and the experience pool.

[0048] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the edge computing task processing method as described in the first aspect.

[0049] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the edge computing task processing method as described in the first aspect is implemented.

[0050] The edge computing task processing method, device and electronic device provided by the embodiments of the present invention obtain the unloading strategy experience of various edge computing environments of various mobile devices for computing tasks through the first reinforcement learning network, so that for each task to be processed of each mobile device, the target experience is extracted from the experience pool containing the unloading strategy experience, and then according to the target experience and the environmental state of the mobile device, the decision model is used to obtain the unloading decision of the task to be processed, that is, for the edge computing environments of different mobile devices, different target experiences and environmental states are used as considerations for the unloading decision, so as to achieve the adaptation between the unloading decision and the different edge computing environments, so as to improve the problem that the existing task computing offloading method is usually only applicable to a certain type of edge computing environment and has poor applicability to different edge computing environments.

[0051] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0053] Figure 1 A block diagram of an edge computing task processing system provided by an embodiment of the present invention is shown.

[0054] Figure 2 One of the flow charts of the edge computing task processing method provided by an embodiment of the present invention is shown.

[0055] Figure 3 The second flowchart of the edge computing task processing method provided by an embodiment of the present invention is shown.

[0056] Figure 4 A schematic diagram of the structure of a first reinforcement learning network provided by an embodiment of the present invention is shown.

[0057] Figure 5 Shows Figure 2 Schematic diagram of the process of some sub-steps of step S102.

[0058] Figure 6 The third flowchart of the edge computing task processing method provided by an embodiment of the present invention is shown.

[0059] Figure 7A schematic diagram of training a decision model provided by an embodiment of the present invention is shown.

[0060] Figure 8 Shows Figure 2 Schematic diagram of the process of some sub-steps of step S106.

[0061] Fig. 9 Shows Figure 8 Schematic diagram of the process of some sub-steps of step S106-2.

[0062] Fig.10 A block diagram of an edge computing task processing device provided by an embodiment of the present invention is shown.

[0063] Fig.11 A block diagram of an electronic device provided by an embodiment of the present invention is shown.

[0064] Icons: 100 - edge computing task processing system; 110 - base station; 120 - edge server; 130 - mobile device; 140 - edge computing task processing device; 150 - first preparation module; 160 - second preparation module; 170 - processing module; 180 - electronic device. DETAILED DESCRIPTION

[0065] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0066] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0067] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0068] In recent years, with the rapid development of wireless communication technology and the popularity of smart mobile devices, the number of various mobile applications has shown an explosive growth trend. For example, facial recognition payment systems, online cloud games, and virtual / augmented reality are computationally intensive and latency-critical applications. However, mobile devices that run these computationally intensive and latency-critical applications usually have limited computing power and battery power. Therefore, mobile edge computing came into being.

[0069] One of the key issues that needs to be addressed in mobile edge computing networks is the computation offloading problem, that is, whether mobile devices choose to offload computing tasks to nearby MEC servers or execute them locally, and how to allocate resources (e.g., computing resources and energy consumption resources) for tasks offloaded to the server.

[0070] A good computation offloading strategy can improve the overall computing power of wireless mobile devices, shorten the system processing task latency, reduce the overall energy consumption of wireless mobile devices, and enhance the performance of mobile edge computing systems. At present, the task computation offloading methods used to deal with computation offloading problems are usually only applicable to a certain type of edge computing environment. If certain environmental parameters of the current edge computing environment (e.g., the amount of data for processing tasks, network bandwidth, computing power of edge servers, number of mobile device communications, etc.) change, the strategies provided by these methods are less applicable when facing the new environment, that is, the strategies provided by these methods have poor portability and are difficult to apply to real environments.

[0071] Based on the above considerations, an embodiment of the present invention provides a processing solution for edge computing tasks, which can improve the current task computing offloading method, which is usually applicable to a certain type of edge computing environment and has the problems of poor applicability and poor portability. The following will be introduced in detail.

[0072] The edge computing task processing method provided by the embodiment of the present invention can be applied to Figure 1In the edge computing task processing system 100 shown, the edge computing task processing system 100 includes a base station 110, an edge server 120 and a plurality of mobile devices 130, each of which is connected to the edge server 120 on the base station 110 through a network. Due to the use of online payment systems, online cloud games and virtual / augmented reality services, new computing tasks are often generated on the mobile device 130, and the computing tasks are tasks to be processed. When the mobile device 130 generates a task to be processed, it extracts target experience matching the environment state from the experience pool containing the experience of uninstallation strategies of various edge computing environments according to the observed environment state of the edge computing environment in which it is located, so as to obtain the uninstallation decision of the task to be processed by using the decision model according to the environment state and the target experience.

[0073] If the offloading decision is local computing, the mobile device 130 executes the task to be processed locally; if the offloading decision is offloading computing, the task to be processed is offloaded to the edge network of the edge server 120 for execution.

[0074] The mobile device 130 includes but is not limited to: a mobile phone, an iPad, a laptop computer, a game console, and a personal wearable device.

[0075] In one embodiment, referring to Figure 2 , provides an edge computing task processing method, comprising the following steps. In this embodiment, the edge computing task processing method is applied to Figure 1 Take the mobile device 130 as an example.

[0076] S102, obtaining offloading strategy experiences of various edge computing environments according to a preset first reinforcement learning network for multiple mobile devices, and putting each set of offloading strategy experiences into an experience pool.

[0077] The edge computing environment refers to the interactive environment between the mobile device 130 and the edge server 120, and the influencing factors include but are not limited to: the data volume of the task processed by the mobile device 130, the data volume of the task processed by the edge server 120, the network bandwidth, the computing power of the edge server 120, the number of communications of the mobile device 130, and the speed and range of the mobile device 130. Any change in any influencing factor will form a new edge computing environment.

[0078] There are multiple groups of unloading strategy experiences in the experience pool. Each group of unloading strategy experiences includes parameters of the neural network initialized with different random seeds. These first reinforcement learning networks are trained separately to obtain various unloading strategy experiences. The unloading strategy experience actually refers to the parameter values ​​of the first reinforcement learning network at different stages of the training of the first reinforcement learning network. Therefore, the unloading strategy experience represents the decision logic of the first reinforcement learning network for different edge computing environments.

[0079] Among them, step S102 can be executed by any server or terminal device, and the experience pool is stored in the mobile device 130. That is, the execution subject of step S102 may not be the same as the execution subject of steps S104 and S106.

[0080] S104, for each task to be processed of the mobile device, according to the environmental state of the mobile device, retrieve the target experience matching the environmental state from the experience pool.

[0081] The environmental state of the mobile device 130 is the state of the edge computing environment observed by the mobile device 130 at this time, that is, it describes the state of the edge computing environment of the mobile device 130.

[0082] The environmental state includes the channel gain between the mobile device 130 and the base station 110, the tasks to be processed (i.e., the computing tasks generated), the available energy of the mobile device 130 at the beginning of time slice t, the queuing delay of the tasks to be processed when they are locally calculated on the mobile device 130, and the queuing delay of the tasks to be processed on the edge server 120 (i.e., the queuing delay when the tasks to be processed are offloaded).

[0083] The target experience extracted from the experience pool has the highest degree of adaptability to the environmental state of the mobile device 130 .

[0084] S106, obtaining an offloading decision of the pending task of the mobile device according to the environmental state and target experience of the mobile device by using a preset decision model.

[0085] The decision model is a pre-built and trained model. The decision model uses the target experience as a parameter and the environment state as an output to output the offloading decision of the task to be processed.

[0086] The principle of the above-mentioned edge computing task processing method is: through the first reinforcement learning network, the unloading strategy experience of various edge computing environments of each mobile device 130 for computing tasks is obtained, so that for each task to be processed of each mobile device 130, the mobile device 130 extracts the target experience from the experience pool containing the unloading strategy experience, and then obtains the unloading decision of the task to be processed based on the target experience and the environmental status of the mobile device 130 using the decision model.

[0087] For different edge computing environments of mobile devices 130, different target experiences and environmental states are used as considerations for offloading decisions to achieve adaptation between offloading decisions and different edge computing environments, so as to improve the problem that existing task computing offloading methods are usually only applicable to a certain type of edge computing environment and have poor applicability to different edge computing environments.

[0088] For the first reinforcement learning model in step S102, in this embodiment, the edge computing task processing method further includes the step of training the first reinforcement learning model, referring to Figure 3 , this step includes the following steps.

[0089] S201, construct an edge computing network system model including multiple mobile devices.

[0090] The edge computing network system model includes the system model, signal transmission model, communication model and computing task model. The following is a detailed introduction.

[0091] 1. System Model

[0092] In order to make the edge computing task processing method provided by the present invention applicable to different edge computing environments, that is, edge computing environments with different influencing factors, without loss of generality, the system model provided is as follows Figure 1 The edge computing task processing system 100 shown in the figure includes an edge server 120 with a wireless access point (access point, AP) and Mobile devices 130, wherein

[0093] Each mobile device 130 is randomly distributed within a certain range of the base station 110. Each mobile device 130 is equipped with a wireless transmission antenna and can transmit data with the base station 110. The time of the entire edge computing network system is divided into multiple constant time slices, represented as t∈{1,2,...}, and the length of each time slice is T seconds. Each mobile device 130 will move with the time slice, assuming that its moving speed V and moving direction θ satisfy the uniform distribution V~U(0,V max ) and θ~U(0,2π).

[0094] Each mobile device 130 generates a computationally intensive task, i.e., a pending task, in time slice t. Each mobile device 130 can choose to remotely execute the pending task on the edge server 120 by means of computational offloading, or can choose to execute the pending task locally on the device. At the same time, due to the failure of task execution caused by heavy workload or undesirable communication conditions that may occur in the network, the edge computing network system uses a first-in-first-out (FIFO) queue as a buffer to store tasks that cannot be completed in the current time slice.

[0095] 2. Signal transmission model

[0096] The signal transmission model includes a channel gain calculation formula and a distance calculation formula between the mobile device 130 and the base station 110 within a time slice t.

[0097] At each time slice t, the channel gain between the edge base station 110 and the mobile device n is It can be calculated using the channel gain calculation formula.

[0098] The channel gain calculation formula includes:

[0099] in, represents the small-scale Rayleigh fading component, represents the large-scale fading component, Remains unchanged within a time slice.

[0100] The calculation method is:

[0101] The lognormal random variable z follows

[0102] In this embodiment, a three-dimensional Cartesian rectangular coordinate system is used to represent the positions of the base station 110 and the mobile device 130 in the edge computing network. Therefore, the distance between the mobile device n and the base station 110 in time slice t is The calculation method is:

[0103] in, and represent the coordinates of the mobile device n and the base station 110 respectively.

[0104] 3. Communication model

[0105] The communication model includes an uplink signal-to-noise ratio from the mobile device 130 to the base station 110 and an uplink data transmission rate of the mobile device 130 in a time slice t.

[0106] use Denotes the transmit power of mobile device n in time slice t. The signal-to-noise ratio (SINR) of the uplink from mobile device n to base station 110 can be calculated as:

[0107] Among them, σ 2 is the additional noise power, Indicates the interference between mobile devices during uninstallation. Denotes the uninstall decision of mobile device n.

[0108] Given the network bandwidth W of the wireless channel, the uplink data transmission rate of mobile device n in time slice t is It can be expressed as:

[0109] 4. Computational Task Model

[0110] The computing task model includes local computing model and edge computing model.

[0111] In this embodiment, the task to be processed (i.e., the computing task) generated by the nth mobile device 130 within the time slice t is Expressed as

[0112] in, Indicates the task The amount of data (in bits), Indicates the number of clock cycles required for the CPU to process 1 bit of data. is the maximum allowable delay of the task. The execution time exceeds will be discarded.

[0113] When a mobile device 130 generates a task to be processed, the task to be processed will be buffered in the FIFO queue of the mobile device 130, waiting to be scheduled for calculation. In this embodiment, the edge computing task processing system 100 provided by the present invention uses a complete offloading method, that is, the task to be processed is either completely executed locally on the mobile device 130, or remotely executed on the edge server 120 through calculation offloading.

[0114] represents the unloading decision variable of mobile device n at time slice t. Indicates pending tasks is offloaded to the edge server 120, Indicates local computing tasks to be processed

[0115] After obtaining the energy transmitted from the wireless access point of the edge server 120, each mobile device 130 needs to decide whether to offload the task to be processed to the edge server 120 or execute it locally, that is, edge computing or local computing, so as to optimize the scheduling and reduce the overall task delay and energy consumption. The local computing model and the edge computing model are described below.

[0116] 1) Local computing model

[0117] The local computation model includes local computation delay, mobile device 130 energy consumption, and mobile device 130 computation cost.

[0118] At time slice t, the local computation frequency of mobile device n is expressed as Pending tasks Local computation latency It can be expressed as:

[0119] in, represents the queuing delay of mobile device n in time slice t, that is, the queuing delay of the task when it is calculated locally on mobile device n, which is calculated by the delay of the previous time slot. and the length of the time slice T, that is:

[0120] Local computing tasks to be processed Energy consumed It can be expressed as: in, The energy consumption of one CPU cycle of mobile device n is expressed as:

[0121] Given and Use their weighted sum to represent the local computing tasks to be processed The computational cost is: ω1+ω2=1.

[0122] Among them, ω1 and ω2 represent the weights of time cost and energy cost respectively, and satisfy 0≤ω1,ω2≤1.

[0123] 2) Edge computing model

[0124] The edge computing model includes the offloading delay, the energy consumption of the mobile device 130, the execution time of the task to be processed on the edge server 120, the waiting energy consumption of the mobile device 130 and the offloading computing cost.

[0125] If the mobile device 130 selects to process the task If the computation is offloaded to the edge server 120 for remote execution, the computation offloading process can be divided into three parts.

[0126] First, the mobile device n unloads the task data of the task to be processed to the edge server 120 via wireless transmission. The required unloading delay is expressed as:

[0127] In pending tasks During the upload process, the energy consumed by mobile device n is expressed as:

[0128] Second, the edge server 120 allocates computing resources to the unloaded pending tasks to complete the computation. Similar to the local computing queue, the edge server 120 on the base station 110 also sets up a task queue. At time slice t, The unloaded tasks (to-be-processed tasks) are stored in the FIFO queue according to the time of arrival at the edge server 120. In Chinese, that is:

[0129] Among them, i represents the task to be processed The index number of the .

[0130] Pending tasks The execution time on the edge server 120 of the base station 110 is expressed as:

[0131] in, Indicates that the edge server 120 is assigned to the task to be processed of computing resources, Indicates pending tasks The queuing delay on the edge server 120 is the waiting time to be executed by the edge server 120 .

[0132] The queuing delay calculation method is expressed as:

[0133] in, Is a pending task In FIFO queue The index in is the time that the edge server 120 on the base station 110 processes the offload task in the previous time slice, which can be expressed as:

[0134] The edge server 120 remotely executes the pending tasks During this period, the energy consumed by mobile device n while waiting can be expressed as:

[0135] in, Represents the power consumption of mobile device n in idle state.

[0136] Third, the calculation results of the tasks to be processed are sent back to the corresponding mobile device 130 by wireless transmission. Since the amount of calculation results is much smaller than the amount of task data, the present invention ignores the transmission delay and energy consumption caused by downloading the calculation results.

[0137] On this basis, the computational cost of offloading can be:

[0138]

[0139] Among them, ω3 and ω4 satisfy 0≤ω3, ω4≤1 and ω3+ω4=1.

[0140] S202: Based on the edge computing network system model, construct an optimization objective function for computing cost.

[0141] By jointly optimizing task offloading and resource allocation for tasks to be processed, an optimization objective function is proposed. The purpose of the optimization objective function is to minimize the total cost of the edge computing network system.

[0142] The optimization objective function can be described as:

[0143]

[0144]

[0145]

[0146]

[0147]

[0148] in, represents the binary offloading decision vector of all mobile devices 130 in the edge computing network system, is the transmit power of each mobile device 130 .

[0149] If the mobile device 130 chooses to execute the pending task locally Then the edge server 120 will not allocate computing resources to it, that is, When, then

[0150] Problem (P) (i.e., function (P)) is a cost optimization function. Constraint (a) is an offloading model constraint, which indicates that the edge computing network system uses a 0 / 1 offloading mode. Constraint (b) is an energy consumption constraint, which ensures that the energy of each mobile device 130 should not be greater than the maximum energy that the device can provide, or the energy of the mobile device 130 should not be exhausted to less than 0. Constraint (c) is a delay constraint, which indicates that the processing time of a task to be processed cannot exceed the maximum acceptable delay. Violating this delay will introduce a penalty Ω that is much larger than the task cost. Constraint (d) is a transmission function constraint, which ensures that the transmission power of each mobile device 130 should not exceed the maximum power it can emit.

[0151] S203: construct a first reinforcement learning network based on the mobile edge computing network system model.

[0152] Among them, the first reinforcement learning network includes a sequentially connected graph neural network, an actor network, a critic network and a hybrid network.

[0153] In detail, based on the above-mentioned edge computing network system model and optimization objective function, in order to enable the learned computing offloading strategy to be migrated to a network environment with different numbers of communication devices (communication devices and mobile devices 130). The present invention first converts the purpose of optimizing the objective function into a non-central partially observable Markov decision process (Dec-POMDP) ​​problem, and defines the state, action and reward of the edge computing network (MEC) system model.

[0154] Due to the signal interference between different mobile devices 130, the offloading decision and resource allocation strategy between the mobile devices 130 are tightly coupled together, which will affect the performance of the entire edge computing network system. Therefore, the present invention regards each mobile device 130 as an independent intelligent agent, and makes decision actions based on partial observations of the current environment, interacts with the environment to gain experience and continuously update the neural network parameters. The present invention adopts a centralized training with decentralized execution (CTDE) mode. Each mobile device 130 only takes its own action observation history (environmental state) as input and uses the learned strategy to make decisions. In this embodiment, the state observation, action and reward are set as follows.

[0155] State observation: At time slice t, since a single agent (mobile device 130) cannot obtain the global environment state, agent n, i.e. mobile device n, can only observe its own partial cognition of the edge computing environment. In this embodiment, this cognition is called the environment state. The environmental status specifically includes: the channel gain between the mobile device 130 and the base station 110, the tasks to be processed (computation tasks) generated, the available energy of the mobile device at the beginning of time slice t, the queuing delay of the tasks to be processed when they are calculated locally on the mobile device 130, and the time of the tasks to be processed The queuing delay on the edge server 120. Therefore, It can be expressed as

[0156] Action: Based on the problem (P) in the optimization objective function, the decision action needs to be determined The decision action includes the calculation offloading vector and the transmit power resource allocation vector for each mobile device 130, namely Based on the corresponding environmental status The first reinforcement learning algorithm (i.e., the offloading algorithm) can obtain the state transition probability P(s′|s,u) of the mobile device 130 from the environment state To decision action The approximately optimal mapping of .

[0157] Reward function: known decision action The purpose of optimizing the objective function is to minimize the sum of system costs, so the goal of the first reinforcement learning network (i.e., the unloading algorithm) is to obtain the maximum reward. In order to meet the constraints of the problem (P), in this embodiment, various penalty terms are introduced to ensure that the first reinforcement learning network can learn a reasonable strategy.

[0158] For each mobile device 130, the penalty term for energy exhaustion is:

[0159] The penalty for task execution timeout is:

[0160] Among them, the indicator function 1{a} means that the value of this item is 1 when condition a is met.

[0161] Then, for each mobile device 130, its corresponding reward function can be expressed as: represents the reward function of the mobile device 130.

[0162] In order to enable the first reinforcement learning network of each mobile device 130 to learn the policy more effectively, the difference in reward function between the current policy and the task completely local calculation policy can be expressed as:

[0163] in, Represents the reward that can be obtained by local computing with maximum computing power.

[0164] Since the present invention belongs to a cooperative mobile edge computing scenario based on multi-agents, different agents need to cooperate with each other to improve the performance of the entire system, rather than acting greedily for their own interests. Therefore, it is necessary to calculate the average reward of all agents to represent the overall performance of the system. Therefore, the final reward function can be expressed as:

[0165] On this basis, the present invention realizes the calculation of the global value expectation function (Q function) based on the self-attention mechanism, in order to realize the environmental state of different numbers of devices For each time slice t, the environmental state of mobile device n Mapped into a semantic embedding, represented as

[0166] Due to the need for computational offloading and resource allocation in the mobile edge computing environment, decision actions contain both discrete values ​​and continuous values. Therefore, in this embodiment, the first reinforcement learning network considers strategy learning based on the "actor-critic" architecture. The action value function {Q1,...,Q n} can be expressed as:

[0167] in, The embedding mapping function representing the environmental state of mobile device n, is the decision action output by mobile device n at time slice t according to the actor network, φ n and ψ n Represent the parameters of the actor network and the critic network respectively. Represents the temporal hidden state of the previous time slice. This variable is introduced because of the DEC-POMDP property of the mobile edge computing environment. It describes the action-observation history information of the wireless mobile device n.

[0168] The present invention will be the timing hidden state As a global time unit to represent the action history information of agent interaction, that is, it is fed into the actor network as a single input, the global time unit ensures the stability of multi-agent algorithm strategy learning in a simple way.

[0169] In order to ensure efficient communication between different devices in the mobile edge computing environment and reduce network congestion caused by unnecessary information interaction between devices, the multi-agent algorithm of the present invention does not introduce a communication mechanism between agents. However, in order to learn excellent offloading strategies in the multi-agent-based algorithm, the present invention considers introducing a self-attention mechanism to allow the mobile device 130 to notice the environmental status and decision-making actions of other devices, thereby making decisions that are beneficial to the overall system. In detail, the present invention embeds the semantics of each mobile device 130 into the As a token, the self-attention mechanism uses Q, K, and V to represent the query, key, and value vectors of each token respectively. Accordingly, the attention can be expressed as:

[0170] Among them, d k Indicates the scaling factor of the key vector dimension. The present invention uses the self-attention mechanism to learn the relationship between the environmental state embedding of different agents and the characteristics of global historical information. In order to learn their own strategies in decentralized multi-agent reinforcement learning, the present invention defines their own Q for each mobile device 130 n ,K n ,V n Vector, the neural network that implements the self-attention mechanism is called the Transformer network.

[0171] The Transformer network contains multiple layers, and the calculation process of the lth layer can be expressed as: as well as

[0172] in, Indicates the calculation of Q n ,K n ,V n A linear fully connected network of vectors, To include embedded information and temporal hidden state The vector of Represents the environment state embedding feature.

[0173] Embed the environment state of the last layer of the Transformer network into features Projection to the action-value function Q n The output space of the feature map is projected using a linear function P, which is expressed as:

[0174] After obtaining the action value function of each user device, the present invention uses a centralized gradient predictor to learn the cooperative task of continuous actions by updating each critic network.

[0175] When the number or dimension of decision actions of agents is large, it may be difficult and impractical to learn a centralized critic network conditioned on global state and joint actions. Therefore, the present invention adopts the idea of ​​action value function decomposition based on the multi-agent actor-critic framework, which can avoid the problem of dimensionality explosion when the number of agents increases, and can effectively train a centralized critic network in the Dec-POMDPs environment. The overall joint action value function Q of all agents is tot It can be expressed as:

[0176] Among them, ψ and ψ n They are the overall joint action value function Q tot And each agent's action value function Q n The critic network parameters, o and o n They are the overall joint action value function Q tot and each agent’s action-value function Q n The environmental state embedding features, u and u n They are the overall joint action value function Q tot and each agent’s action-value function Q n The decision action, φ n are the parameters of the actor network, s is the state of the environment, g represents a nonlinear monotonic mixing function, and its network parameter η.

[0177] To train the evaluation strategy, the critic networks are trained centrally using the following loss function:

[0178] Among them, y tot =r+γQ tot (o′,μ(o′;φ′);ψ′,η′), y totrepresents the expected reward value of the target network, and D represents the experience pool. φ′, ψ′, and η′ are the parameters of the target actor network, the target critic network, and the target hybrid network, respectively. o′ is the embedded feature of the environment state, r is the reward function value, and γ is the discount factor, which can be set as needed.

[0179] In order to update the individual strategies of each agent, i.e. the parameters of the actor network, the present invention designs a centralized gradient updater to achieve better cooperation between agents. It can optimize the entire joint action space instead of optimizing the actions of each agent separately as in the existing methods. In addition, when calculating the policy gradient, the present invention evaluates Q tot When , we select actions from the current strategies μ of all agents. Therefore, the centralized policy gradient of the present invention can be expressed as:

[0180] Among them, μ={o1,μ1(o1;φ1),...,o n ,μ n (o n ; φ n )} is the set of current strategies of all agents. The actor network structure of all agents is the same, and the parameter is φ n .

[0181] The centralized gradient estimator of the present invention uses a deterministic strategy to optimize over the entire joint action space, taking full advantage of the centralized critic network update.

[0182] Based on the above, refer to Figure 4 The first reinforcement learning network (i.e., the target network) finally constructed includes a graph neural network, an actor network, a critic network, and a hybrid network connected in sequence, i.e., a transformer network, an actor network, a critic network, and a hybrid network connected in sequence. In addition, the graph neural network, the actor network, and the critic network are multiple, and a graph neural network, an actor network, and a critic network connected in sequence constitute an offloading algorithm, each offloading algorithm corresponds to an intelligent agent (i.e., a mobile device 130), and the critic network of each offloading algorithm is connected to the hybrid network.

[0183] The actor network decides which respective action to take, i.e., local computation or offloaded computation. The critic network estimates the Q of each agent. n Then, all Q n The values ​​are combined into a joint action value function Q tot .Q totA centralized gradient estimator is used for parameter updates to help the actor network learn an offloading policy.

[0184] S204: Train a first reinforcement learning network in combination with the optimization objective function and the environmental state of the edge computing environment of each mobile device.

[0185] Based on the above content, it can be known that the reward function is related to the optimization objective function, and the loss function is related to the reward function. Therefore, the first reinforcement learning network is updated according to the loss function iteration root until the end condition is reached, and the final first reinforcement learning network, that is, the first reinforcement learning network preset in S102, can be obtained.

[0186] In one embodiment, the first reinforcement learning network can be trained in the following manner: construct a loss function based on the optimization objective function; use the environmental state of the edge computing environment of each mobile device 130 as the input of the first reinforcement learning network, and train the first reinforcement learning network in combination with the loss function.

[0187] During the training process of the first reinforcement learning network, the parameters of the first reinforcement learning network at each step (each iteration) are used as the unloading strategy experience to obtain the experience pool.

[0188] Further, refer to Figure 5 , which is a flowchart of some sub-steps of step S104, the following steps can be used to retrieve target experience matching the environmental state from the experience pool.

[0189] S104-1, using a clustering algorithm, divide the unloading strategy experience in the experience pool into multiple strategy families.

[0190] The clustering algorithm can be flexibly selected and is not limited to a single one. In one implementation, it can be a k-means algorithm.

[0191] S104-2, calculating the distance between the environment state and each strategy family, and taking the strategy family with the smallest distance among the multiple strategy families as the target strategy family.

[0192] The distances (which may be Euclidean distances) between the environment state observed by the mobile device 130 generating the task to be processed and each strategy family are calculated, and the target strategy family with the smallest distance (the smallest distance means the highest correlation) is selected.

[0193] S104-3, selecting an uninstallation policy experience from the target policy family as the target experience.

[0194] Through the above steps S104 - 1 to S104 - 3 , a target experience matching the environmental state of the mobile device 130 may be selected.

[0195] In order to introduce step S106 in more detail, a method for obtaining a decision model is first provided, that is, the edge computing task processing method provided by the present invention also includes a step of training a supervised learning model to obtain a decision model.

[0196] The supervised learning model includes a first encoder, a first decoder, a second encoder, a second decoder and an evaluator. The output of the first encoder is the input of the first decoder, and the first decoder and the second decoder are used to obtain the unloading decision. The output of the second encoder is the input of the second decoder. The output of the first encoder and the output of the second encoder after training are the input of the evaluator, and the evaluator is used to optimize the output of the first encoder.

[0197] For more details, see Figure 6 and Figure 7 , which can be achieved by following the steps below.

[0198] S301, based on the experience of the uninstallation strategy, multiple groups of state actions are obtained.

[0199] Each set of state-action includes environment state and decision action. t ,u t )} means, s t Indicates the environmental state, u t Indicates a decision action.

[0200] The unloading strategy experience is used as the parameter of the second reinforcement learning network, and the environment state is input into the first reinforcement learning network to obtain the decision action and the state action. The second reinforcement learning network can be obtained after the parameters of the first reinforcement learning are initialized.

[0201] S302: Use multiple groups of state actions to iteratively update parameters of the first encoder and the first decoder until a first end condition is met.

[0202] See also Figure 7 (a) At each iteration, the state action is used as the input of the first encoder, and the first encoder outputs the policy embedding vector e μ To the first decoder, the first decoder predicts the decision action It can be expressed as: μ =E μ ({(s t ,u t )};θ μ ),

[0203] Among them, θ μ is the parameter of the first encoder, δ μ are the parameters of the first decoder.

[0204] The first end condition may be that the loss value satisfies a threshold for minimizing the loss.

[0205] After each iteration, the predicted decision action and the actual u are calculated t When the loss value reaches the preset threshold for minimizing the loss, the iteration ends. The training of the first encoder and the first decoder is completed.

[0206] S303, based on the second reinforcement learning network and using the offloading strategy experience, obtain a second state transition tuple regarding the environment interaction of each edge computing environment.

[0207] By using the unloading strategy experience as the parameter of the second reinforcement learning network and inputting the environmental state into the first reinforcement learning network, the decision action can be obtained to obtain the interaction trajectory of the q previous k interactions between the mobile device 130 and the edge computing network, that is, the second state transition tuple.

[0208] The second state transition tuple can be expressed as {(s t ,u t ,s t+1 )}.

[0209] S304: Use the second state transition tuple to iteratively update parameters of the second encoder and the second decoder until a second end condition is met.

[0210] See also Figure 7 (b) In each iteration, the second state transition tuple is used as the input of the second encoder to obtain the state transition embedding vector e output by the second encoder t , the state transfer embedding vector e t Input the second decoder to get the next environment state predicted by the second decoder It can be expressed as: t =E t ({(s t ,u t ,s t+1 )};θ t ),and

[0211] Among them, θ t and δ t are the parameters of the second encoder and the second decoder respectively.

[0212] The second end condition can be that the loss value satisfies the minimization and t+1 The loss error between .

[0213] After each iteration, calculate and t+1When the loss error reaches the second end condition, the iteration ends and the training of the second encoder and the second decoder is completed.

[0214] S305 , using the trained output of the first encoder and the trained output of the second encoder as inputs of an evaluator, and iteratively updating parameters of the evaluator until a third termination condition is met.

[0215] See also Figure 7 (c) In more detail:

[0216] S1: Based on all state actions and offloading strategy experience, calculate the average action value of each state for various edge computing environments.

[0217] Using the above Get the average action value G.

[0218] S2: At each iteration, the output of the first encoder and the output of the second encoder are used as the input of the evaluator to obtain the average return value predicted by the evaluator.

[0219] Among them, the average benefit value represents the average action value of multiple states of the current edge computing environment.

[0220] The evaluator W is actually a policy embedding vector e μ The evaluation function can also be understood as an optimization function. Given a given environment, the state transfer embedding vector e t Under the condition that the estimated average action value G is allowed to μ Update optimization without interacting with known network environments.

[0221] The estimator can be expressed as: Among them, the matrix A(s0,e t ) is the initial environment state s0 and the dynamic embedding e t A function of can be subjected to singular value decomposition A = USV T , calculate the optimal strategy embedding vector for the current new network environment

[0222] Using Encoder E μ To infer the policy embedding vector e μ , predicted by the evaluator W, the average return Finally, given the initial state s0, the policy embedding vector e μ and the state transition embedding vector e t , using supervised learning to minimize the average return G and The loss function is used to train the evaluator W.

[0223] S306, obtaining a decision model based on the trained first encoder, first decoder, second encoder and evaluator.

[0224] Through the above steps S301 to S306, the trained decision model can quickly adapt to the new edge computing environment.

[0225] In one embodiment, reference Figure 8 , step S106 can be implemented by the following steps.

[0226] S106-1, based on the target experience and the environmental state of the mobile device, using a second reinforcement learning network, obtain a first state transition tuple.

[0227] The second reinforcement learning network can be obtained after initializing parameters of the first reinforcement learning network.

[0228] The parameters of the second reinforcement learning network are updated to use the target experience as the parameters of the second reinforcement learning network. At this time, the updated second reinforcement learning model can learn the unloading strategy of the target experience.

[0229] The environmental state of the mobile device 130 is input into the updated second reinforcement learning model to obtain a first state transition tuple.

[0230] S106-2, using the first state transition tuple and the environment state as inputs of the decision model to obtain an offloading decision for the task to be processed.

[0231] Furthermore, the decision model includes a first encoder, a second encoder, an evaluator and a first decoder. The output of the first encoder and the output of the second encoder are input to the evaluator, the output of the evaluator is input to the first decoder, and the first decoder outputs an unloading decision.

[0232] For more details, see Fig. 9 , S106-2 may include the following steps.

[0233] S401, using the environment state as the input of the first encoder to obtain a strategy vector.

[0234] When a task to be processed is generated, the mobile device 130 inputs the observed environment state into the first encoder to obtain a strategy vector output by the first encoder.

[0235] S402, using the first state transition tuple and the environment state as inputs of the second encoder to obtain a state transition vector.

[0236] S403, using the strategy vector and the state transition vector as inputs of the evaluator to obtain an optimized strategy vector.

[0237] S404: Using the optimized strategy vector as input to the first decoder to obtain an offloading decision for the task to be processed.

[0238] Among them, the offloading decision includes local computing or offloading computing.

[0239] Furthermore, through the above steps S401 to S404 , the offloading decision of the to-be-processed task of each mobile device 130 can be obtained.

[0240] In the edge computing task processing method provided by the present invention, the preset first reinforcement learning network is trained based on the optimization objective function for minimizing the total cost of the edge computing network system. Therefore, the unloading strategy experience of the first reinforcement learning network and the average action value of each state of various edge computing environments are considered, and the decision model for calculating the unloading decision according to the environmental state is trained. Therefore, the decision model has the ability of transfer learning, that is, it is still applicable in the face of different new edge computing environments.

[0241] Therefore, the edge computing task processing method provided by the present invention can adapt to different environmental states through a decision model to obtain an unloading decision, and the obtained unloading decision meets the condition of minimizing the total cost of the edge computing network system.

[0242] It should be understood that although Figure 2-Figure 9 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 2-Figure 9 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0243] Based on the concept of the above edge computing task processing method, in one embodiment, referring to Fig.10 , an edge computing task processing device 140, including a first preparation module 150, a second preparation module 160 and a processing module 170.

[0244] The first preparation module 150 is used to obtain the unloading strategy experience of various edge computing environments according to the preset first reinforcement learning network about the multiple mobile devices 130, and put each set of unloading strategy experience into the experience pool.

[0245] The second preparation module 160 is used to retrieve target experience matching the environmental state from the experience pool for each task to be processed of the mobile device 130 according to the environmental state of the mobile device 130 .

[0246] The processing module 170 is used to obtain an offloading decision of the to-be-processed task of the mobile device 130 by using a decision model according to the environmental state of the mobile device 130 and the experience pool.

[0247] The edge computing task processing device 140 can be applied to Figure 1 In the mobile device 130 shown, the target experience is prepared by the first preparation module 150 and the second preparation model, so that the processing module 170 obtains the offloading decision according to the experience pool and the environmental state. Using the decision model, for the edge computing environments of different mobile devices 130, different target experiences and environmental states are used as considerations for offloading decisions, so as to achieve the adaptation between offloading decisions and different edge computing environments, so as to improve the problem that the existing task computing offloading methods are usually only applicable to a certain type of edge computing environment and have poor applicability to different edge computing environments.

[0248] For the specific limitations of the edge computing task processing device 140, please refer to the limitations of the edge computing task processing method above, which will not be repeated here. The various modules in the above-mentioned edge computing task processing device 140 can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the edge computing task processing device 140 in the form of hardware, or can be stored in the memory in the edge computing task processing device 140 in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0249] In one embodiment, an electronic device 180 is provided, whose internal structure diagram can be as follows: Fig.11As shown. The electronic device 180 includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. Among them, the processor of the electronic device 180 is used to provide computing and control capabilities. The memory of the electronic device 180 includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the electronic device 180 is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, an operator network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, an edge computing task processing method is implemented. The display screen of the electronic device 180 can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device 180 can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the housing of the electronic device 180, or an external keyboard, touchpad or mouse.

[0250] Those skilled in the art will understand that Fig.11 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present invention, and does not constitute a limitation on the electronic device 180 to which the solution of the present invention is applied. The specific electronic device 180 may include Fig.11 More or fewer components may be shown, or certain components may be combined, or may have a different arrangement of components.

[0251] In one embodiment, the edge computing task processing method provided by the present invention can be implemented in the form of a computer program. The computer program can be Fig.11 The memory of the electronic device 180 may store various program modules constituting the edge computing task processing device 140, for example, Fig.10 The first preparation module 150, the second preparation module 160 and the processing module 170 are shown. The computer program composed of each program module enables the processor to execute the steps in the edge computing task processing method described in this specification.

[0252] For example, Fig.11 The electronic device 180 shown may be Fig.10 The first preparation in the edge computing task processing method shown is performed in step S102. The electronic device 180 can perform step S104 through the second preparation module 160. The electronic device 180 can perform step S106 through the processing module 170.

[0253] In one embodiment, an electronic device 180 is provided, including a memory and a processor, the memory storing a computer program, and the processor implementing the following steps when executing the computer program: obtaining unloading strategy experiences of various edge computing environments according to a preset first reinforcement learning network regarding multiple mobile devices 130, and placing each set of unloading strategy experiences into an experience pool; for each pending task of each mobile device 130, retrieving target experience matching the environmental state from the experience pool according to the environmental state of the mobile device 130; obtaining an unloading decision of the pending task of the mobile device 130 by using a preset decision model according to the environmental state and target experience of the mobile device 130.

[0254] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: according to a preset first reinforcement learning network regarding multiple mobile devices 130, unloading strategy experiences of various edge computing environments are obtained, and each group of unloading strategy experiences is placed into an experience pool; for each pending task of each mobile device 130, according to the environmental state of the mobile device 130, target experience matching the environmental state is retrieved from the experience pool; according to the environmental state and target experience of the mobile device 130, a preset decision model is used to obtain an unloading decision for the pending task of the mobile device 130.

[0255] In several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0256] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0257] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0258] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for processing edge computing tasks, characterized in that: The method comprises: According to a preset first reinforcement learning network for multiple mobile devices, offloading strategy experiences of various edge computing environments are obtained, and each group of offloading strategy experiences is put into an experience pool; For each task to be processed of the mobile device, according to the environmental state of the mobile device, a target experience matching the environmental state is retrieved from the experience pool; According to the environmental state of the mobile device and the target experience, using a preset decision model, obtaining an offloading decision of the pending task of the mobile device; The step of obtaining a decision action for a task to be processed of the mobile device by using a preset decision model according to the environmental state of the mobile device and the target experience comprises: Based on the target experience and the environmental state of the mobile device, using a second reinforcement learning network, obtaining a first state transition tuple; Taking the first state transition tuple and the environment state as inputs of the decision model, obtaining an offloading decision of the task to be processed; The decision model includes a first encoder, a second encoder, an evaluator and a first decoder; The step of taking the first state transition tuple and the environment state as inputs of the decision model to obtain the decision action of the task to be processed includes: Taking the environment state as input of the first encoder, obtaining a strategy vector; Taking the first state transition tuple and the environment state as inputs of the second encoder, obtaining a state transition vector; Using the strategy vector and the state transition vector as inputs of the evaluator to obtain an optimized strategy vector; The optimized strategy vector is used as the input of the first decoder to obtain the offloading decision of the task to be processed.

2. The method for processing edge computing tasks according to claim 1, characterized in that: The method further comprises the step of training a supervised learning model to obtain a decision model, wherein the supervised learning model comprises a first encoder, a first decoder, a second encoder, a second decoder and an evaluator; The step of training the supervised learning model to obtain the decision model includes: Based on the experience of the unloading strategy, multiple groups of state actions are obtained, each group of state actions includes an environment state and a decision action; Iteratively updating the parameters of the first encoder and the first decoder using the multiple sets of the state actions until a first end condition is met; Based on the second reinforcement learning network, using the offloading strategy experience, obtaining a second state transition tuple about the environment interaction of each edge computing environment; Iteratively updating the parameters of the second encoder and the second decoder using the second state transition tuple until a second end condition is met; Using the trained output of the first encoder and the trained output of the second encoder as inputs of the evaluator, and iteratively updating parameters of the evaluator until a third termination condition is met; Based on the trained first encoder, first decoder, second encoder and evaluator, a decision model is obtained.

3. The edge computing task processing method according to claim 2, characterized in that: The step of using the trained output of the first encoder and the output of the second encoder as inputs of the evaluator and iteratively updating the parameters of the evaluator until the third end condition is met comprises: Based on all the state actions and the offloading strategy experience, calculate the average action value of each state of each type of edge computing environment; In each iteration, the output of the first encoder and the output of the second encoder that have been trained are used as inputs of the evaluator to obtain an average benefit value predicted by the evaluator, where the average benefit value represents the average action value of multiple states of the current edge computing environment; The loss value between the average benefit value and the average action value is calculated. If the loss value satisfies the third termination condition, the iteration is terminated; otherwise, the iteration is continued until the loss value satisfies the third termination condition.

4. The edge computing task processing method according to claim 1, characterized in that: The step of retrieving target experience matching the environmental state from the experience pool according to the environmental state of the mobile device comprises: Using a clustering algorithm, the unloading strategy experience in the experience pool is divided into a plurality of strategy families; Calculating the distance between the environment state and each of the strategy families, and taking the strategy family with the smallest distance among the multiple strategy families as the target strategy family; An uninstallation strategy experience is selected from the target strategy family as a target experience.

5. The edge computing task processing method according to claim 1, characterized in that: The method further comprises the step of training a first reinforcement learning network, the step comprising: Construct an edge computing network system model including multiple mobile devices; Based on the edge computing network system model, construct an optimization objective function for computing cost; Based on the edge computing network system model, construct a first reinforcement learning network, wherein the first reinforcement learning network includes a sequentially connected graph neural network, an actor network, a critic network, and a hybrid network; The first reinforcement learning network is trained in combination with the optimization objective function and the environmental state of the edge computing environment of each mobile device.

6. The edge computing task processing method according to claim 5, characterized in that: The optimization objective function includes a cost optimization function, an offloading mode constraint, an energy consumption constraint, a delay constraint, and a transmission power constraint; The step of training the first reinforcement learning network in combination with the optimization objective function and the environmental state of the edge computing environment of each mobile device includes: Based on the optimization objective function, construct a loss function; The environmental state of the edge computing environment of each of the mobile devices is used as an input of the first reinforcement learning network, and the first reinforcement learning network is trained in combination with the loss function.

7. An edge computing task processing device, characterized in that: comprising a first preparation module, a second preparation module and a processing module, for implementing the edge computing task processing method as claimed in claim 1; The first preparation module is used to obtain the offloading strategy experience of various edge computing environments according to the preset first reinforcement learning network about multiple mobile devices, and put each group of the offloading strategy experience into the experience pool; The second preparation module is used to retrieve, for each task to be processed of the mobile device, target experience matching the environmental state from the experience pool according to the environmental state of the mobile device; The processing module is used to obtain an offloading decision of the to-be-processed task of the mobile device by using the decision model according to the environmental state of the mobile device and the experience pool.

8. An electronic device, characterized in that: It includes a processor and a memory, the memory stores machine executable instructions that can be executed by the processor, and the processor can execute the machine executable instructions to implement the edge computing task processing method as described in any one of claims 1 to 6.

Images