Method and device for obtaining calculation unloading strategy, electronic equipment and storage medium

By generating and inputting the environment state into the computational unloading decision model, combined with the optimization of the MADDPG algorithm, the problem that the simulated annealing algorithm cannot adapt to in a dynamic environment is solved, and the dynamic adaptability and performance improvement of the computational unloading strategy is achieved.

CN118939393APending Publication Date: 2024-11-12QINGDAO HAIER TECH +2
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410948117.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-16
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The existing simulated annealing algorithm cannot adjust the algorithm parameters in a timely manner in a dynamic environment, resulting in the computational unloading strategy being unable to adapt to environmental changes.

Method used

By obtaining the operation information and task information of the smart terminal, generating the environment status, and entering a preset computational offload decision model, a computational offload strategy suitable for the current environmental conditions is generated, and the decision model is optimized by the MADDPG algorithm to improve adaptability.

Benefits of technology

It realizes dynamic adaptability of the computing unloading strategy, reduces computing delay and energy consumption, and improves the computing performance of the smart terminal.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118939393A_ABST
    Figure CN118939393A_ABST
Patent Text Reader

Abstract

The invention discloses a method and device for obtaining a calculation unloading strategy, electronic equipment and a storage medium, and relates to the technical field of smart home, and the method for obtaining the calculation unloading strategy comprises the steps: obtaining operation information and task information; generating a first environment state according to the operation information and the task information; the first environment state is network information, computing power and a task state of the intelligent terminal; and inputting the first environment state into a preset calculation unloading decision model to generate a first calculation unloading strategy. In this way, the influence of environmental state factors on the calculation unloading strategy effect is fully considered, so that the obtained calculation unloading strategy can better adapt to the dynamically changing environmental state.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of smart home technology, and in particular to a method, device, electronic device and storage medium for obtaining a computing offload strategy. Background Art

[0002] Nowadays, in family life, the computing resources and processing capabilities of smart terminal devices are limited, while the amount of data generated by the interaction between devices is increasing, which poses a severe challenge to the processing capabilities of smart terminal devices. Computation offloading, as a technology to optimize the utilization of computing resources, can break through the bottleneck caused by the hardware functions of smart terminal devices by migrating computing tasks from smart terminal devices to edge servers or cloud platforms for processing, further improve the computing performance of smart terminal devices, and reduce energy consumption. In the process of computation offloading, it is crucial to choose an efficient computation offloading strategy to optimize latency and energy consumption.

[0003] Currently, the computing offloading strategy of intelligent terminal devices is mainly determined by random optimization algorithms, such as the simulated annealing algorithm. The simulated annealing algorithm is a random optimization algorithm that simulates the physical annealing process of solid materials. It finds the global optimal solution by random search and accepting new solutions by probability.

[0004] The optimal computational unloading scheme is obtained by using the simulated annealing algorithm. The specific process is as follows: first, the initial temperature, cooling coefficient and minimum temperature are set, the maximum number of iterations at a single temperature is set, and the annealing speed is set. The initial temperature is determined as the initial iteration temperature. The unloading decision is initialized, the initial unloading decision is obtained, and the initial fitness value is calculated. The initial unloading decision is determined as the optimal unloading decision, and the initial fitness value is determined as the fitness of the optimal unloading decision. During the iteration process, new unloading decisions are continuously obtained and new fitness values ​​are calculated. It is determined whether the fitness value of the new unloading decision is better than the fitness value of the optimal unloading decision. If yes, the new unloading decision is accepted, that is, the new unloading decision is determined as the optimal unloading decision; if no, it is determined whether to accept the new unloading decision based on the random value that changes with the iteration temperature. After completing the maximum number of iterations at a single temperature, the new iteration temperature is obtained according to the current iteration temperature and the cooling coefficient, and it is determined whether the new iteration temperature is less than or equal to the minimum temperature. If yes, the algorithm ends and the current optimal unloading decision is output; if no, it continues to iterate at the new iteration temperature until the iteration temperature is less than or equal to the minimum temperature.

[0005] In the process of implementing the embodiments of the present application, it is found that there are at least the following problems in the related technology:

[0006] In the prior art, the simulated annealing algorithm does not consider environmental factors during the iterative calculation process, so it is only applicable to static environments. The simulated annealing algorithm cannot adjust the algorithm parameters in a changing environment in a timely manner according to the changes in the environment, resulting in the calculation offloading strategy obtained being unable to adapt to the dynamically changing environment.

[0007] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present application, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0008] In order to provide a basic understanding of some aspects of the disclosed embodiments, a brief summary is given below. The summary is not an extensive review, nor is it intended to identify key / critical components or delineate the scope of protection of these embodiments, but rather serves as a prelude to the detailed description that follows.

[0009] Embodiments of the present application provide a method, device, electronic device, and storage medium for obtaining a computing offload strategy, so as to obtain a computing offload strategy that is more suitable for a dynamic environment.

[0010] In some embodiments, the method for obtaining a computing offloading strategy is applied to a smart terminal, and the method for obtaining a computing offloading strategy includes: obtaining operation information and task information; generating a first environmental state based on the operation information and the task information; the first environmental state is the network information, computing power and task status of the smart terminal; inputting the first environmental state into a preset computing offloading decision model to generate a first computing offloading strategy.

[0011] In some embodiments, the method for obtaining a computing offloading strategy is applied to a smart terminal, and the device for obtaining a computing offloading strategy includes: a first acquisition module, configured to obtain operation information and task information; a second acquisition module, configured to generate a first environmental state based on the operation information and the task information; the first environmental state is the network information, computing power and task status of the smart terminal; an execution module, configured to input the first environmental state into a preset computing offloading decision model to generate a first computing offloading strategy.

[0012] In some embodiments, the electronic device includes: a memory and a processor, the memory stores a computer program, and the processor is configured to execute the method for obtaining a computing offloading strategy as described above through the computer program.

[0013] In some embodiments, the computer-readable storage medium includes stored program instructions, and the program instructions are executed by a processor to implement the method for obtaining a computation offloading policy as described above.

[0014] The method, device, electronic device, and storage medium for obtaining a computing offloading strategy provided in the embodiments of the present application can achieve the following technical effects:

[0015] The present invention generates a first environmental state by acquiring operation information and task information, and inputs the first environmental state into a preset computing offloading decision model to generate a first computing offloading strategy. By inputting the environmental state into the computing offloading decision model, a computing offloading strategy suitable for the current environmental conditions can be obtained. The computing offloading decision model fully considers the influence of environmental state factors on the effect of computing offloading strategies, so that the acquired computing offloading strategies can better adapt to dynamically changing environmental states.

[0016] The above general description and the following description are exemplary and explanatory only and are not intended to limit the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0018] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0019] Figure 1 It is a hardware environment diagram of a method for obtaining a computing offloading strategy provided in an embodiment of the present application;

[0020] Figure 2 is a schematic diagram of a method for obtaining a computing offloading strategy provided by an embodiment of the present application;

[0021] Figure 3 This is a schematic diagram of a practical application scenario;

[0022] Figure 4 It is a schematic diagram of the architecture of a computing offloading decision model provided in an embodiment of the present application;

[0023] Figure 5 is a schematic diagram of a device for obtaining a computing offloading strategy provided by an embodiment of the present application;

[0024] Figure 6 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] According to one aspect of an embodiment of the present application, a method for obtaining a computing offloading strategy is provided. The method for obtaining a computing offloading strategy is widely used in smart home (Smart Home), smart home, smart home device ecology, smart residential (Intelligence House) ecology and other whole-house intelligent digital control application scenarios. Optionally, in this embodiment, the above-mentioned method for obtaining a computing offloading strategy can be applied to Figure 1 In the hardware environment composed of the intelligent terminal 102 and the server 104 shown in FIG. Figure 1 As shown, the server 104 is connected to the smart terminal 102 via a network, and can be used to provide services (such as application services, etc.) for the terminal or a client installed on the terminal. A database can be set on the server or independently of the server to provide data storage services for the server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data computing services for the server 104.

[0028] The above network may include but is not limited to at least one of the following: wired network, wireless network. The above wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the above wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The intelligent terminal may be but is not limited to PC, mobile phone, tablet computer, intelligent air conditioner, intelligent range hood, intelligent refrigerator, intelligent oven, intelligent stove, intelligent washing machine, intelligent water heater, intelligent washing equipment, intelligent dishwasher, intelligent projection equipment, smart TV, intelligent clothes drying rack, intelligent curtain, intelligent audio and video, intelligent socket, intelligent audio, intelligent speaker, intelligent fresh air equipment, intelligent kitchen and bathroom equipment, intelligent bathroom equipment, intelligent sweeping robot, intelligent window cleaning robot, intelligent mopping robot, intelligent air purification equipment, intelligent steamer, intelligent microwave oven, intelligent kitchen treasure, intelligent purifier, intelligent water dispenser, intelligent door lock, etc.

[0029] Combination Figure 2 As shown, an embodiment of the present application provides a method for obtaining a computing offloading strategy, which is applied to a smart terminal. The method for obtaining a computing offloading strategy includes:

[0030] Step S201: The intelligent terminal obtains operation information and task information, wherein the operation information is the operation information of the intelligent terminal and the task information is the computing task information of the intelligent terminal.

[0031] Step S202: The intelligent terminal generates a first environment state according to the operation information and the task information. The first environment state is the network information, computing capability and task state of the intelligent terminal.

[0032] Step S203: The intelligent terminal inputs the first environmental state into a preset computing offloading decision model to generate a first computing offloading strategy.

[0033] The method for obtaining a computing offloading strategy provided in an embodiment of the present application is adopted to generate a first environmental state by obtaining operation information and task information, and the first environmental state is input into a preset computing offloading decision model to generate a first computing offloading strategy. The computing offloading strategy is obtained by inputting the environmental state into the computing offloading decision model to obtain a computing offloading strategy suitable for the current environmental conditions. The computing offloading decision model fully considers the influence of environmental state factors on the effect of the computing offloading strategy, so that the computing offloading strategy obtained can better adapt to the dynamically changing environmental state.

[0034] The computing offloading strategy is used to divide the proportion of computing tasks generated by the smart terminal to be processed locally in the smart terminal and offloaded to the edge server for processing. The first computing offloading strategy is a computing offloading strategy obtained by a preset computing offloading decision model. The preset computing offloading decision model is a model based on the MADDPG algorithm.

[0035] Optionally, the operation information of the intelligent terminal includes the computing power and available channel bandwidth of the intelligent terminal. The computing task information of the intelligent terminal includes the data volume of the computing task, the CPU cycle required to process the computing task, the maximum acceptable delay of the intelligent terminal, and the maximum energy consumption capacity of the intelligent terminal. The available channel bandwidth includes the available uplink channel bandwidth and the available downlink channel bandwidth.

[0036] Optionally, after generating the first computing offloading strategy, the method further includes: the intelligent terminal allocating computing tasks to the edge server or the cloud platform according to the first computing offloading strategy.

[0037] Optionally, after generating the first computing offloading strategy, the method further includes: the intelligent terminal optimizing the computing offloading decision model. In this way, the computing offloading decision model can output the corresponding computing offloading strategy according to environmental changes more timely, so that the computing offloading decision model has stronger flexibility and adaptability.

[0038] Optionally, the intelligent terminal optimizes the calculation offloading decision model, including: the intelligent terminal obtains a second environmental state. The second environmental state is the network information, computing power and task state of the intelligent terminal after executing the second calculation offloading strategy. The second calculation offloading strategy is a calculation offloading strategy obtained by the calculation offloading decision model. The intelligent terminal adjusts the parameters of the calculation offloading decision model according to the training sample group stored in the preset experience playback buffer. The intelligent terminal inputs the second environmental state into the calculation offloading decision model after adjusting the parameters for training, and outputs a third calculation offloading strategy to optimize the calculation offloading decision model. The third calculation offloading strategy is a calculation offloading strategy obtained by the calculation offloading decision model after adjusting the parameters. Among them, the second calculation offloading strategy includes the first calculation offloading strategy and the third calculation offloading strategy. By obtaining the training sample group and the real-time environmental state, the calculation offloading decision model is optimized, and the environmental factors and past experience are fully considered to improve the adaptability of the calculation offloading decision model to the environment. And each time the calculation offloading decision model is adjusted and trained, a calculation offloading strategy will be output. In this way, while optimizing the calculation offloading decision model, a better calculation offloading strategy can also be obtained.

[0039] Optionally, after the intelligent terminal optimizes the computing offloading decision model, it also includes: the intelligent terminal determines the number of optimizations of the computing offloading decision model. When the number of optimizations reaches a preset number, the intelligent terminal determines the computing offloading decision model optimized for the preset number of times as the final computing offloading decision model. The final computing offloading decision model is obtained by iteratively optimizing the preset computing offloading decision model for a preset number of times, and the influence of environmental state factors on the effect of the computing offloading strategy is fully considered, so that the final computing offloading decision model obtained is more suitable for the current environmental conditions, thereby being more conducive to obtaining a computing offloading strategy that is more suitable for the current environmental conditions.

[0040] Optionally, after the intelligent terminal optimizes the computational offloading decision model, the method further includes: the intelligent terminal determines the number of optimizations of the computational offloading decision model. When the number of optimizations reaches a preset number, the intelligent terminal determines the computational offloading strategy obtained when optimizing the computational offloading decision model for the preset number of times as the final computational offloading strategy. By iteratively optimizing the preset computational offloading decision model for a preset number of times, the final computational offloading strategy is obtained when the computational offloading decision model fully considers the influence of the environmental state, so that the final computational offloading strategy has stronger adaptability to the current environmental conditions.

[0041] In some embodiments, the intelligent terminal inputs the first environmental state obtained into the above-mentioned calculation offloading decision model to obtain the first calculation offloading strategy. Then, the intelligent terminal obtains the second environmental state, and obtains the reward function according to the first calculation offloading strategy and the second environmental state. The first environmental state, the first calculation offloading strategy, the reward function and the second environmental state are determined as a training sample group and stored in the experience replay buffer. Then the intelligent terminal adjusts the parameters of the calculation offloading decision model according to the training sample group stored in the experience replay buffer. The calculation offloading decision model completes the first iteration cycle. The intelligent terminal inputs the second environmental state into the calculation offloading decision model training after the first parameter adjustment, and outputs the first third calculation offloading strategy. Then, the intelligent terminal obtains the fourth environmental state. The fourth environmental state is the network information, computing power and task state of the intelligent terminal after executing the first third calculation offloading strategy. And according to the first third calculation offloading strategy and the fourth environmental state, the reward function after the first parameter adjustment is obtained. The second environmental state, the first third calculation offloading strategy, the reward function after the first parameter adjustment and the fourth environmental state are determined as a training sample group and stored in the experience replay buffer. Then, the intelligent terminal optimizes and adjusts the parameters of the calculation offloading decision model again according to the training samples stored in the experience replay buffer. The calculation offloading decision model completes the second iteration cycle. The intelligent terminal inputs the fourth environmental state into the calculation offloading decision model training after the second parameter adjustment, and outputs the second third calculation offloading strategy. Then, the intelligent terminal obtains the fifth environmental state. The fifth environmental state is the network information, computing power and task state of the intelligent terminal after executing the second third calculation offloading strategy. And according to the second third calculation offloading strategy and the fifth environmental state, the reward function after the second parameter adjustment is obtained. The fourth environmental state, the second third calculation offloading strategy, the reward function after the second parameter adjustment and the fifth environmental state are determined as a training sample group and stored in the experience playback buffer. Then, the intelligent terminal optimizes and adjusts the parameters of the calculation offloading decision model according to the training samples stored in the experience playback buffer. The calculation offloading decision model completes the third iteration cycle. The intelligent terminal iterates the loop according to the above steps to optimize the calculation offloading decision model until the number of optimizations reaches the preset number, so that the intelligent terminal obtains a calculation offloading decision final model and obtains the corresponding final calculation offloading strategy.

[0042] Optionally, the experience replay buffer is obtained in the following manner: the intelligent terminal obtains a reward function according to the second calculation offloading strategy and the third environment state. The reward function is used to characterize the reward value obtained after executing the second calculation offloading strategy. The third environment state is the network information, computing power and task state of the intelligent terminal before executing the second calculation offloading strategy. The intelligent terminal jointly determines the third environment state, the second calculation offloading strategy, the reward function and the second environment state as a training sample group. The intelligent terminal stores the training sample group in the experience replay buffer. The purpose of setting the experience replay buffer is to utilize past experience to stabilize and accelerate the learning process of the calculation offloading decision model to improve the optimization efficiency of the calculation offloading decision model. For example, when the calculation offloading decision model is iterated for the first time, the first environment state is the third environment state. When the calculation offloading decision model is iterated for the second time, the second environment state is the third environment state. When the calculation offloading decision model is iterated for the third time, the fourth environment state is the third environment state. That is, the second environment state at the last iteration is the third environment state at the next iteration.

[0043] The capacity of the experience replay buffer is fixed. When a new training sample group is stored in the experience replay buffer, the old training sample group will be discarded to keep the capacity of the experience replay buffer unchanged. For example, if the capacity of the experience replay buffer is 100 training sample groups, when a new training sample group is stored in the experience replay buffer that already has 100 training sample groups stored, the training sample group with the longest storage time in the experience replay buffer will be discarded.

[0044] Optionally, the intelligent terminal obtains a reward function according to the second computing offloading strategy and the third environmental state, including: the intelligent terminal obtains a first computing delay and a first computing energy consumption according to the third environmental state. The intelligent terminal obtains a second computing delay and a second computing energy consumption according to the second computing offloading strategy. The intelligent terminal obtains a reward function according to the first computing delay, the first computing energy consumption, the second computing delay and the second computing energy consumption. Among them, the first computing delay is the delay caused by all computing tasks being processed locally in the intelligent terminal. The first computing energy consumption is the energy consumption caused by all computing tasks being processed locally in the intelligent terminal. The second computing delay is the delay caused by the intelligent terminal executing the second offloading strategy to process the computing task. The second computing energy consumption is the energy consumption caused by the intelligent terminal executing the second offloading strategy to process the computing task.

[0045] Optionally, the intelligent terminal adjusts the parameters of the computation offloading decision model according to the training sample groups stored in the preset experience replay buffer, including: the intelligent terminal randomly extracts multiple training sample groups from the experience replay buffer to obtain a set of training sample groups. The intelligent terminal adjusts the parameters of the computation offloading decision model according to the set of training sample groups. The computation offloading decision model is optimized by randomly sampling and reusing past training sample groups, so that the computation offloading decision model can learn more stably and effectively in a complex environment with multiple intelligent agents, thereby improving the optimization efficiency of the model.

[0046] Optionally, the intelligent terminal adjusts the parameters of the computation offloading decision model according to the training sample group set, including: the intelligent terminal obtains a loss function according to the training sample group set. The intelligent terminal obtains a policy gradient according to the training sample group set. The above policy gradient is the expected value of the long-term cumulative reward that the intelligent agent can obtain when executing the computation offloading policy output by the online actor network. The intelligent terminal adjusts the parameters of the computation offloading decision model according to the loss function and the policy gradient.

[0047] Optionally, the intelligent terminal adjusts the parameters of the computation offloading decision model according to the loss function and the policy gradient, including: the intelligent terminal updates the online critic network parameters according to the loss function. The intelligent terminal updates the online actor network parameters according to the policy gradient. The intelligent terminal adjusts the target critic network parameters and the target actor network parameters according to the updated online critic network parameters and the updated online actor network parameters.

[0048] Combination Figure 3 As shown, the embodiment of the present application models a smart home system consisting of M smart terminals and N edge servers in an IoT scenario, where N = {1, 2, ..., N} represents a set of edge servers, and M = {1, 2, ..., M} represents a set of smart terminals. Wherein, M and N are both positive integers. Smart terminals include UE1, UE2, ..., UE M The edge servers include ES1, ES2, ..., ES N . Assume that each smart terminal runs a smart home application, and the application generates a delay-sensitive and computationally intensive computing task in each time slot t. For example, the computing task generated by smart terminal UE1 is Task1, and the computing task generated by smart terminal UE2 is Task2. Among them, D is the number of time slots. The computing task generated by the i-th intelligent terminal can be represented by a four-tuple ψ i = {v i , c i , τ i , e i}, where v iThe amount of data generated for the computing task of the i-th intelligent terminal, c i is the CPU cycle required for the i-th intelligent terminal to process the computing task, τ i is the maximum acceptable delay of the i-th intelligent terminal, e i is the maximum energy consumption capacity of the ith intelligent terminal. The maximum energy consumption capacity is the remaining battery power of the ith intelligent terminal, 1≤i≤M, and M is the number of intelligent agents.

[0049] In order to improve the utilization of computing resources, each smart terminal divides the generated computing task into N+1 non-overlapping sub-computing tasks according to different division ratios, and each sub-computing task can be selected to be processed locally in the smart terminal, or to be offloaded to the edge server for processing. Each edge server allocates computing and communication resources according to the workload of each smart terminal. Assume that the i-th smart terminal is based on P i =(p i,0 , p i,1 , p i,2 , ..., pi,N ), and divide the generated computing task into N+1 sub-computing tasks. i is the computation offloading strategy of the i-th intelligent terminal, p i,0 is the division ratio of the first level task of the i-th intelligent terminal, p i,1 , p i,2 , ..., p i,N is the division ratio of the second sub-task of the i-th intelligent terminal, N is the number of edge servers, 1≤i≤M, M is the number of intelligent terminals. And, 0≤p i,j ≤1, p i,j is the division ratio of the second sub-task offloaded from the i-th smart terminal to the j-th second sub-task, 1≤j≤N, N is the number of edge servers. The first sub-task is a sub-computation task processed locally by the smart terminal. The second sub-task is a sub-computation task offloaded from the smart terminal to the edge server for processing. The i-th smart terminal will calculate the offload strategy P i The generated computing tasks are divided into sub-computing tasks according to the division ratio, and then the first-level tasks are left for local processing and the second-level tasks are offloaded to the corresponding edge server for processing.

[0050] Combining the local computing and edge computing models, the second computing delay of the i-th smart terminal is defined as Among them, T i local The delay caused by the i-th intelligent terminal processing the first primary task, is the delay caused by the i-th smart terminal offloading to the j-th edge server to process the second secondary task, and the max function is used to obtain the maximum value. The smart terminal processes the first secondary task and the edge server processes the corresponding second secondary task in parallel. Therefore, the computational delay of the smart terminal executing the computational offloading strategy is the longest delay caused by processing the first secondary task and the second secondary task. At the same time, the second computational energy consumption of the i-th smart terminal is defined as in, The energy consumption caused by the i-th intelligent terminal processing the first primary task, is the energy consumption caused by offloading the i-th smart terminal to the j-th edge server to process the second secondary task, 1≤j≤N, 1≤i≤M, N is the number of edge servers, and M is the number of smart terminals.

[0051] The purpose of this application is to reduce the computational latency and energy consumption of computing task offloading in dynamic scenarios of smart home IoT. In order to evaluate the effectiveness of the computational offloading strategy, the total cost of the i-th smart terminal is defined as y i =α·T i +β·E i Among them, T i is the second computing delay of the i-th intelligent terminal, E i is the second computing energy consumption of the ith intelligent terminal, α is the second computing delay weight, β is the second computing energy consumption weight, and α+β=1. It can be seen that the smaller the total cost of the ith intelligent terminal to execute the computing offloading strategy, the better the performance of the corresponding computing offloading strategy. It can be deduced that the long-term optimization goal of the computing offloading decision model is Among them, P i is the computation offloading strategy of the i-th intelligent terminal, v i is the sum of the weighted costs of delay and energy consumption of the ith intelligent terminal in all time slots, D is the number of time slots, y i is the total cost of executing the computation offloading strategy for the i-th intelligent terminal, γ is the attenuation factor, and the attenuation factor is the weight of the total cost, 0≤γ≤1. Therefore, the long-term optimization goal of the computation offloading decision model is to minimize the sum of the weighted total cost of computation delay and computation energy consumption of the intelligent terminal in all time slots.

[0052] The state space, action space and reward function are established in combination with the optimization objective, thereby obtaining the computational offloading decision model of the present application.

[0053] The intelligent terminal converts the state space S of time slot t into t Defined as the observed state σ of each agent i (t), the observed state of the agent is the state of the environment observed by the agent. The observed state of the i-th agent is σ i (t) = {fedge , f i local , ψ i (t), w up , w down}, where f edge is the computing power of all edge servers, f i local is the computing power of the ith intelligent terminal, ψ i (t) is the computing task generated by the i-th intelligent terminal in time slot t, w up is the available uplink channel bandwidth, w down is the available downlink channel bandwidth, 1≤t≤D, D is the number of time slots, 1≤i≤M, M is the number of agents. Therefore, the state space at time slot t is S t ={σ1(t),σ2(t),...,σ M (t)}, where σ1(t) is the state of the environment observed by the first agent, σ2(t) is the state of the environment observed by the second agent, and σ M (t) is the environmental state observed by the Mth agent.

[0054] In the computation offloading decision model proposed in this application, each intelligent agent will make a computation offloading decision based on the environmental state observed at the current moment. Therefore, the intelligent terminal will take the action space a of time slot t as t It is defined as consisting of the computation offloading decisions of each agent. The computation offloading strategy of the i-th agent in time slot t is a i (t) = {p i,0 (t), p i,1 (t), p i,2 (t), ..., p i,N (t)}, where p i,0 (t) is the division ratio of the first sub-task of the ith agent, p i,1 , p i,2 , ..., p i,N is the division ratio of the second sub-task of the i-th agent, N is the number of edge servers, 1≤i≤M, and M is the number of agents. Therefore, the action space in time slot t is a t ={a1(t), a2(t), ..., a M (t)}, where a1(t) is the computation offloading strategy of the first agent at time slot t, a2(t) is the computation offloading strategy of the second agent at time slot t, and a M (t) is the computation offloading strategy of the Mth agent at slot t. Then, the intelligent terminal sets the first environment state S 1 Input the preset computing offloading decision model to generate the corresponding first computing offloading strategy a1 .

[0055] At each time slot t, the environment will evaluate the pros and cons of the computation offloading strategy executed by each smart terminal and provide a corresponding reward value. This application defines the reward function of the i-th smart terminal at time slot t as Among them, λ t is the reward weight of the preset computing task delay, λ e is the reward weight of the preset computing task energy consumption, T i,local (t) is the first calculation delay of the i-th intelligent terminal, T i (t) is the second computing delay of the i-th intelligent terminal, E i,local (t) is the first computing energy consumption of the i-th intelligent terminal, E i (t) is the second computing energy consumption of the i-th intelligent terminal. Therefore, the global reward function at time slot t is Among them, 1≤i≤M, M is the number of intelligent terminals, and the global reward function includes the reward functions of all intelligent agents in time slot t.

[0056] In the traditional deep reinforcement learning environment, each agent is constantly learning and improving strategies without considering the evolutionary strategies of other agents, so that the training process cannot directly use the experience replay method. In addition, when the number of agents increases, the optimization method of estimating gradients may cause a large variance. Therefore, the present application applies the MADDPG algorithm to the computational offloading decision model to improve the performance of computational task offloading, and adopts a centralized training and decentralized execution mode to achieve mutual perception between multiple agents. The embodiment of the present application adopts the MADDPG algorithm to obtain a computational offloading decision model in the field of IoT smart home, optimizes the computational energy consumption and computational delay of computational offloading, thereby improving the efficiency of edge computing, and at the same time enables the acquired computational offloading strategy to better adapt to dynamically changing environmental conditions.

[0057] In this application, each IoT smart device is modeled as an intelligent agent with its own perception capability and decision model. Each intelligent agent includes an actor network responsible for outputting the computational offloading strategy and a critic network responsible for evaluating the computational offloading strategy. The actor network includes an online actor network and a target actor network, and the structure of the target actor network is the same as that of the online actor network. The critic network includes an online critic network and a target critic network, and the structure of the target critic network is the same as that of the online critic network.

[0058] The parameters of the online actor network are in, are the online actor network parameters of the first agent, are the online actor network parameters of the second agent, are the parameters of the online actor network of the Mth agent, where M is the number of agents. The parameters of the online critic network are in, are the parameters of the online critic network of the first agent, are the parameters of the online critic network of the second agent, are the network parameters of the online critic of the Mth agent. The network parameters of the target actor are in, is the target actor network parameter of the first agent, are the target actor network parameters of the second agent, is the target actor network parameter of the Mth agent. The target critic network parameter is in, are the target critic network parameters of the first agent, are the target critic network parameters of the second agent, are the target critic network parameters of the Mth agent.

[0059] Combination Figure 4 As shown, taking the i-th agent as an example, the online actor network parameters are The parameters of the online critic network are The target actor network parameters are The target critic network parameters are In the computation offloading decision model proposed in this application, the intelligent terminal takes the environmental state σ observed by the i-th intelligent agent at time slot t as i (t) Input the online actor network and obtain the computation offloading strategy of the i-th intelligent terminal in time slot t. Then, the state space S t and the action space a at time slot t t Input the online critic network and obtain the Q value to evaluate the performance of the computation offloading strategy. After the agent completes the computation offloading strategy, the agent will receive the reward function r t , and enter the next state space S t+1 At the same time, it will be (S t , a t , r t , S t+1 ) as the training sample group and store it in the experience replay buffer. In the process of training the computation offloading decision model, the agent randomly samples several training sample groups from the experience replay buffer, and determines the above several randomly sampled training sample groups as a training sample group set. The xth training sample group in the training sample group set is (S x , a x , r x , Sx +1 ). The intelligent terminal obtains the Q value of the target critic network and the Q value of the online critic network through the training sample set, thereby obtaining the loss function. The parameters of the online critic network of the i-th agent are updated through the loss function, and the parameters of the online actor network of the i-th agent are updated by calculating the policy gradient.

[0060] Then, the intelligent terminal calculates Get the Q value of the target critic network of the xth training sample group. x is the global reward function of the xth training sample group, and γ is the decay factor. is the Q value of the target critic network for the x+1th training sample group, S x+1 is the state space of the x+1th training sample group, is the computation offloading strategy for the first agent in the x+1th training sample group, The computation offloading strategy for the Mth agent in the x+1th training sample group. is the computation offloading strategy output by the target actor network for the i-th agent in the x+1-th training sample group, is the environmental state observed by the i-th agent in the x+1-th training sample group.

[0061] Therefore, the loss function of the ith agent is Where X is the number of training sample groups extracted from the experience playback buffer, y x is the Q value of the target critic network for the xth training sample group, is the Q value of the online critic network of the xth training sample group, which is used to evaluate the performance of the computational offloading strategy. x is the environmental state of the xth training sample group, is the computation offloading strategy for the first agent in the xth training sample group, is the computation offloading strategy for the Mth agent in the xth training sample group, 1≤x≤X.

[0062] At the same time, the i-th agent has a given state space S x and action space The online actor network parameters will be updated by calculating the policy gradient to maximize the accumulated reward value. The policy gradient of the intelligent terminal is That is, the expected value of the long-term cumulative reward that the ith agent can obtain when executing the computation offloading strategy output by the online actor network. π The function is used to calculate the expectation of long-term accumulated rewards. is the computation offloading strategy output by the online actor network of the ith agent in the xth training sample group, is the Q value of the i-th agent online critic network in the x-th training sample group. x is the environmental state of the xth training sample group, is the computation offloading strategy for the ith agent in the xth training sample group, The computation offloading strategy for the Mth agent in the xth training sample group. The computation offloading strategy obtained by the i-th agent in the x-th training sample group according to the observed environmental state. The online actor network parameters are updated by calculating the gradient to maximize the expected return of the long-term cumulative reward that the agent can obtain when executing the strategy output by the online actor network, thereby optimizing the computational energy consumption and computational latency of computation offloading, which helps to improve the efficiency of edge computing.

[0063] Finally, the intelligent terminal adjusts the target critic network parameters and the target actor network parameters according to the updated online critic network parameters and the updated online actor network parameters. The intelligent terminal calculates ω Q′ ←ξω Q +(1-ξ)ω Q′ , update the target critic network parameters. The intelligent terminal calculates θ π′ ←ξθ π +(1-ξ)θ π′ , update the target actor network parameters. Among them, ω Q′ is the target critic network parameter, ω Q is the updated online critic network parameter, θ π′ is the target actor network parameter, θ π are the updated online actor network parameters, ξ is the learning rate for soft update, which is usually set to 0.01.

[0064] After updating the target critic network parameters and the target actor network parameters, the intelligent terminal completes the optimization of the i-th agent in the above computation offloading decision model. The intelligent terminal repeats the above optimization steps of the computation offloading decision model until the number of executions reaches the preset number, obtains a computation offloading decision final model, and obtains a final computation offloading strategy.

[0065] Combination Figure 5As shown, an embodiment of the present application provides a device 500 for obtaining a computing offloading strategy, including: a first acquisition module 501, a second acquisition module 502 and an execution module 503. The first acquisition module 501 is configured to obtain operation information and task information. Among them, the operation information is the operation information of the smart terminal. The task information is the computing task information of the smart terminal. The second acquisition module 502 is configured to generate a first environmental state based on the operation information and the task information. The first environmental state is the network information, computing power and task status of the smart terminal. The execution module 503 is configured to input the first environmental state into a preset computing offloading decision model to generate a first computing offloading strategy.

[0066] The device for obtaining a computing offloading strategy provided by an embodiment of the present application is used to generate a first environmental state by obtaining the operation information and task information of the intelligent terminal, and the first environmental state is input into a preset computing offloading decision model to generate a first computing offloading strategy. By inputting the environmental state into the computing offloading decision model, a computing offloading strategy suitable for the current environmental conditions can be obtained. The computing offloading decision model fully considers the influence of environmental state factors on the effect of the computing offloading strategy, so that the acquired computing offloading strategy can better adapt to the dynamically changing environmental state.

[0067] Optionally, the device for obtaining a computing offloading strategy further includes an optimization module configured to optimize the computing offloading decision model.

[0068] Optionally, the optimization module is configured to optimize the computing offloading decision model in the following manner: obtaining a second environmental state. The second environmental state is the network information, computing power, and task state of the intelligent terminal after executing the second computing offloading strategy. The second computing offloading strategy is a computing offloading strategy obtained by the computing offloading decision model. According to the training sample group stored in the preset experience playback buffer, the parameters of the computing offloading decision model are adjusted. The computing offloading decision model after the second environmental state is input into the adjusted parameters is trained to optimize the computing offloading decision model.

[0069] Optionally, the optimization module is configured to adjust the parameters of the computation offloading decision model according to the training sample groups stored in the preset experience replay buffer in the following manner: randomly extract multiple training sample groups from the experience replay buffer to obtain a training sample group set. According to the training sample group set, the parameters of the computation offloading decision model are adjusted.

[0070] Optionally, the optimization module is further configured to obtain the experience replay buffer in the following manner: according to the second calculation offloading strategy and the third environment state, obtain a reward function. The reward function is used to characterize the reward value obtained after executing the second calculation offloading strategy. The third environment state is the network information, computing power and task state of the intelligent terminal before executing the second calculation offloading strategy. The third environment state, the second calculation offloading strategy, the reward function and the second environment state are jointly determined as a training sample group. The training sample group is stored in the experience replay buffer.

[0071] Optionally, the device for obtaining a computing offloading strategy further includes a first determination module configured to determine the number of optimizations of the computing offloading decision model. When the number of optimizations reaches a preset number, the computing offloading decision model optimized for the preset number of times is determined as the final computing offloading decision model.

[0072] Optionally, the device for obtaining a computing offloading strategy further includes a second determination module configured to determine the number of optimizations of the computing offloading decision model. When the number of optimizations reaches a preset number, the computing offloading strategy obtained when the computing offloading decision model is optimized for the preset number of times is determined as the final computing offloading strategy.

[0073] Combination Figure 6 As shown, an embodiment of the present application provides an electronic device 600, including a processor (processor) 601 and a memory (memory) 602. Optionally, the device may also include a communication interface (CommunicationInterface) 603 and a bus 604. Among them, the processor 601, the communication interface 603, and the memory 602 can communicate with each other through the bus 604. The communication interface 603 can be used for information transmission. The processor 601 can call the logic instructions in the memory 602 to execute the method for obtaining the calculation offloading strategy of the above embodiment. Among them, the electronic device is a smart terminal.

[0074] The electronic device provided in the embodiment of the present application is used to generate a first environmental state by acquiring operation information and task information, and the first environmental state is input into a preset computing offloading decision model to generate a first computing offloading strategy. By inputting the environmental state into the computing offloading decision model, a computing offloading strategy suitable for the current environmental conditions is obtained. The computing offloading decision model fully considers the influence of environmental state factors on the effect of the computing offloading strategy, so that the acquired computing offloading strategy can better adapt to the dynamically changing environmental state.

[0075] In addition, the logic instructions in the memory 602 described above may be implemented in the form of software functional units and when sold or used as independent products, may be stored in a computer-readable storage medium.

[0076] The memory 602, as a computer-readable storage medium, can be used to store software programs, computer executable programs, such as program instructions / modules corresponding to the methods in the embodiments of the present application.

[0077] The processor 601 executes the function application and data processing by running the program instructions / modules stored in the memory 602, that is, implements the method for obtaining the computing offloading strategy in the above embodiment.

[0078] The memory 602 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and an application required for at least one function; the data storage area may store data created according to the use of the smart terminal device, etc. In addition, the memory 602 may include a high-speed random access memory and may also include a non-volatile memory.

[0079] An embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium includes stored program instructions, and the program instructions are executed by a processor to implement the method for obtaining a computing offloading strategy as described above.

[0080] The computer-readable storage medium mentioned above may be a transient computer-readable storage medium or a non-transitory computer-readable storage medium.

[0081] The technical solution of the embodiment of the present application can be embodied in the form of a software product, which is stored in a storage medium and includes one or more instructions for a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the above method in the embodiment of the present application. The aforementioned storage medium may be a non-transient storage medium, including: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, and other media that can store program codes, or a transient storage medium.

[0082] The above description and accompanying drawings fully illustrate the present application embodiments so that those skilled in the art can practice them. Other embodiments may include structural, logical, electrical, process and other changes. The embodiments represent possible changes only. Unless explicitly required, separate components and functions are optional, and the order of operation may vary. The parts and features of some embodiments may be included in or replace the parts and features of other embodiments. Moreover, the words used in this application are only used to describe the embodiments and are not used to limit the claims. As used in the description of the embodiments and claims, unless the context clearly indicates, the singular forms of "a", "an" and "the" are intended to include plural forms as well. Similarly, the term "and / or" as used in this application refers to any and all possible combinations of listings containing one or more associated ones. In addition, when used in the present application, the term "comprise" and its variants "comprises" and / or comprising refer to the presence of stated features, wholes, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or groups thereof. In the absence of further restrictions, the elements defined by the sentence "comprising a ..." do not exclude the presence of other identical elements in the process, method or device comprising the elements. In this article, each embodiment may focus on the differences from other embodiments, and the same and similar parts between each embodiment may refer to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0083] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. The technicians may use different methods to implement the described functions for each specific application, but such implementations should not be considered to exceed the scope of the embodiments of the present application. The technicians may clearly understand that, for the convenience and simplicity of description, the specific working processes of the systems, devices and units described above may refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0084] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices, equipment, etc.) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present application may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.

[0085] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two continuous boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two continuous operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

[0086] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for obtaining a computing offloading strategy, characterized in that: Applied to a smart terminal, the method comprises: acquiring operation information and task information; Generate a first environment state according to the operation information and the task information; the first environment state is the network information, computing power and task state of the smart terminal; The first environmental state is input into a preset computing offloading decision model to generate a first computing offloading strategy.

2. The method for obtaining a computing offloading strategy according to claim 1, characterized in that: After the first computing offloading strategy is generated, the method further includes: Optimize the computation offloading decision model.

3. The method for obtaining a computing offloading strategy according to claim 2, characterized in that: Optimize the computing offloading decision model, including: Acquire a second environment state; the second environment state is the network information, computing power and task state of the intelligent terminal after executing the second computing offloading strategy; the second computing offloading strategy is a computing offloading strategy obtained through a computing offloading decision model; Adjusting the parameters of the computational offloading decision model according to the training sample group stored in the preset experience playback buffer; The computing offloading decision model after inputting the adjustment parameters of the second environmental state is trained to optimize the computing offloading decision model.

4. The method for obtaining a computing offloading strategy according to claim 3, characterized in that: According to the training sample group stored in the preset experience playback buffer, adjusting the parameters of the computation offloading decision model includes: Randomly extracting a plurality of training sample groups from the experience playback buffer to obtain a training sample group set; According to the training sample set, the parameters of the calculation offloading decision model are adjusted.

5. The method for obtaining a computing offloading strategy according to claim 3, characterized in that: The experience replay buffer is obtained in the following way: According to the second computing offloading strategy and the third environment state, a reward function is obtained; the reward function is used to characterize the reward value obtained after executing the second computing offloading strategy; the third environment state is the network information, computing power and task state of the smart terminal before executing the second computing offloading strategy; Determine the third environment state, the second computation offloading strategy, the reward function and the second environment state as a training sample group; The training sample group is stored in the experience playback buffer.

6. The method for obtaining a computing offloading strategy according to any one of claims 2 to 5, characterized in that: After optimizing the computation offloading decision model, it also includes: Determine the optimization times of calculating the uninstall decision model; When the number of optimizations reaches a preset number, the calculation offloading decision model after the preset optimization is determined as the final calculation offloading decision model.

7. The method for obtaining a computing offloading strategy according to any one of claims 2 to 5, characterized in that: After optimizing the computation offloading decision model, it also includes: Determine the optimization times of calculating the uninstall decision model; When the number of optimizations reaches a preset number, the calculation offloading strategy obtained when optimizing the calculation offloading decision model for the preset time is determined as the final calculation offloading strategy.

8. A device for obtaining a computing offloading strategy, characterized in that: Applied to smart terminals, including: A first acquisition module is configured to acquire operation information and task information; A second acquisition module is configured to generate a first environment state according to the operation information and the task information; The execution module is configured to input the first environmental state into a preset computing offloading decision model to generate a first computing offloading strategy.

9. An electronic device, comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method for obtaining a computation offloading strategy according to any one of claims 1 to 7 through the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes stored program instructions, and the program instructions are executed by a processor to implement the method for obtaining a computation offloading strategy according to any one of claims 1 to 7.

Citation Information

Cited By

  • Voice task scheduling method, device and system, and storage medium

    CN120091368A

  • Voice task scheduling method, device, system, and storage medium

    CN120091368B