Differentiated computing power pricing-oriented end-side collaborative reasoning task unloading method

Abstract

The embodiment of the invention discloses an end-side collaborative reasoning task unloading method for differentiated computing power pricing. The method comprises the steps of obtaining experience data in an experience pool; updating the reinforcement learning model according to the empirical data; constructing a state vector according to the channel gain at the current moment, the payment cost at the previous moment, the inference task break point and the edge server identifier, inputting the state vector into the updated reinforcement learning model, and obtaining a task allocation strategy of each inference task, each terminal device unloads the reasoning task to the edge server for reasoning calculation according to the task distribution strategy; wherein the empirical data at least comprises the payment cost of each terminal device and the service cost corresponding to the calculation resource quantity distributed by the characterization edge server for completing the remaining reasoning task of each terminal device. According to the embodiment of the invention, each terminal device can minimize the time delay of reasoning and the cost of using edge computing power.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of edge computing technology, and in particular to a method for offloading end-edge collaborative reasoning tasks for differentiated computing power pricing. Background Art

[0002] As deep learning is increasingly used in various smart devices, how to efficiently perform reasoning tasks in resource-constrained environments becomes a key issue.

[0003] To solve this problem, the end-edge collaborative reasoning technology was proposed. This technology divides the deep learning model between the terminal device and the edge server, so that part of the task runs on the terminal device and the other part is completed on the edge server. This makes full use of the computing resources of both parties and optimizes the entire reasoning process.

[0004] However, most of the existing edge-to-end collaborative reasoning optimization methods focus on the scenarios of a single terminal device and a single edge server, and fail to adapt to the complex environment of multiple terminals and multiple edge servers. In the prior art, although there are edge-to-end collaborative reasoning optimization methods based on game theory, such methods are mainly applicable to static environments and are difficult to cope with network fluctuations and device load changes under dynamic network conditions.

[0005] In addition, with the continuous growth of terminal devices, deep learning applications, and edge server deployment, it is becoming more common for multiple terminal devices to request computing resources at the same time. Due to the limited computing power, network bandwidth, and storage resources of edge servers, resource competition is easily triggered, resulting in a decrease in inference performance. Summary of the invention

[0006] Based on the above problems, an embodiment of the present application provides an end-edge collaborative reasoning task offloading method for differentiated computing power pricing, with the aim of optimizing reasoning task allocation and resource utilization efficiency in an edge computing environment.

[0007] In a first aspect, an embodiment of the present application provides a method for offloading end-to-end collaborative reasoning tasks for differentiated computing power pricing, including:

[0008] Get the experience data in the experience pool;

[0009] updating the reinforcement learning model according to the empirical data;

[0010] For each terminal device, a state vector is constructed according to the channel gain of the link between each edge server and the terminal device at the current moment, the payment fee at the previous moment, the reasoning task split point output at the previous moment, and the edge server identifier output at the previous moment, and the state vector is input into the updated reinforcement learning model to obtain the task allocation strategy of the reasoning task of each terminal device, so that each terminal device offloads the reasoning task to the edge server for reasoning calculation according to the task allocation strategy;

[0011] Among them, the empirical data includes the state vector, output strategy, and immediate benefit of each terminal device at each moment; the state vector includes at least payment fee, reasoning task split point and edge server identifier; the payment fee represents the service fee corresponding to the amount of computing resources allocated by the edge server to complete the remaining reasoning tasks of each terminal device; the output strategy includes the reasoning task split point and edge server identifier; the immediate benefit is used to represent the immediate income of each terminal device after unloading the reasoning task.

[0012] In one embodiment, the experience data in the experience pool is obtained in the following manner:

[0013] For each terminal device at each moment, a state vector is constructed according to the channel gain of the link between each edge server and the terminal device, the payment fee at the last moment, the reasoning task split point output at the last moment, and the edge server identifier output at the last moment;

[0014] Input the state vector into a pre-built reinforcement learning model to obtain an output action space; the action space includes the selection probability of each reasoning task split point and the selection probability of each edge server;

[0015] Determine an output strategy including the reasoning task split point and the edge server identifier according to the selection probability of each reasoning task split point and the selection probability of each edge server, so that the terminal device offloads the target reasoning task to the edge server corresponding to the edge server identifier for reasoning calculation according to the output strategy;

[0016] Allocating computing resources to the target reasoning task according to the output strategy, and calculating payment fees according to the amount of computing resources allocated to the target reasoning task;

[0017] Obtaining the inference time after the target inference task completes the inference calculation, and calculating the immediate benefit of each of the terminal devices according to the inference time and the payment fee;

[0018] The state vector, the output strategy and the immediate benefit at each moment are taken as the experience data at each moment and stored in the experience pool.

[0019] In one embodiment, determining the output strategy including the reasoning task split point and the edge server identifier according to the selection probability of each reasoning task split point and the selection probability of each edge server includes:

[0020] According to the selection probability of each reasoning task split point and the selection probability of each edge server, each reasoning task split point and each edge server are randomly selected respectively to obtain an output strategy including the reasoning task split point and the edge server identifier.

[0021] In one embodiment, allocating computing resources to the target reasoning task according to the output strategy, and calculating the payment fee according to the amount of computing resources allocated to the target reasoning task, includes:

[0022] Obtaining the remaining computational amount of the target reasoning task according to the reasoning task splitting point in the output strategy;

[0023] Determine the amount of computing resources allocated to the target reasoning task according to the remaining computing amount of the target reasoning task; the allocated computing resources are part of the available computing resources of the edge server corresponding to the edge server identifier in the output strategy;

[0024] The payment fee of the terminal device is calculated based on the amount of computing resources allocated to the target reasoning task and the preset computing resource usage pricing.

[0025] In one embodiment, determining the amount of computing resources allocated to the target reasoning task according to the remaining computing amount of the target reasoning task includes:

[0026] For the edge server corresponding to the edge server identifier in the output strategy, determine the sum of remaining computation amounts of all the inference tasks required for inference computation by the edge server;

[0027] The ratio between the total remaining computing amount and the remaining computing amount of the target reasoning task required for the reasoning calculation by the edge server is calculated to determine the amount of computing resources allocated to the target reasoning task.

[0028] In one embodiment, the calculating the immediate benefit of each terminal device according to the inference time and the payment fee includes:

[0029] The instant benefit of each terminal device is calculated by the following formula:

[0030]

[0031] in, is a benefit function corresponding to the instant benefit of the terminal device; is the total inference latency of the target inference task of the terminal device; is the payment fee of the terminal device; k is the time.

[0032] In one embodiment, updating the reinforcement learning model according to the empirical data includes:

[0033] At each moment, a sampled data set is obtained by sampling multiple experience data in the experience pool, and the sampled data set is used to update the weight parameters of the value network, the weight parameters of the average policy network, and the weight parameters of the policy network in the reinforcement learning model.

[0034] In a second aspect, the embodiment of the present application further provides a device-edge collaborative reasoning task offloading device for differentiated computing power pricing, including:

[0035] An acquisition unit, used for acquiring experience data in the experience pool;

[0036] A model updating unit, used for updating the reinforcement learning model according to the experience data;

[0037] A processing unit is used to construct a state vector for each terminal device according to the channel gain of the link between each edge server and the terminal device at the current moment, the payment fee at the previous moment, the reasoning task split point output at the previous moment, and the edge server identifier output at the previous moment, input the state vector into the updated reinforcement learning model, and obtain a task allocation strategy for the reasoning task of each terminal device, so that each terminal device offloads the reasoning task to the edge server for reasoning calculation according to the task allocation strategy;

[0038] Among them, the empirical data includes the state vector, output strategy, and immediate benefit of each terminal device at each moment; the state vector includes at least payment fee, reasoning task split point and edge server identifier; the payment fee represents the service fee corresponding to the amount of computing resources allocated by the edge server to complete the remaining reasoning tasks of each terminal device; the output strategy includes the reasoning task split point and edge server identifier; the immediate benefit is used to represent the immediate income of each terminal device after unloading the reasoning task.

[0039] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0040] CPU, memory, input and output interfaces;

[0041] The memory is a short-term storage memory or a persistent storage memory;

[0042] The central processing unit is configured to communicate with the memory and execute instruction operations in the memory to perform any one of the above-mentioned end-edge collaborative reasoning task offloading methods for differentiated computing power pricing.

[0043] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program runs on a computer, any one of the above-mentioned methods for end-edge collaborative reasoning task offloading for differentiated computing power pricing is executed.

[0044] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0045] By incorporating payment fees into the state vector, the reinforcement learning model can effectively optimize the amount of computing resources on the edge server side where the inference task is allocated, reducing the service fees that the terminal device needs to pay when completing the task.

[0046] Based on the channel gain of the link, the payment fee at the previous moment, and the inference task split point, and by linking the computing resources of the edge server with the payment fee of the terminal device, the reinforcement learning model can make the optimal task offloading decision, minimize the inference latency and the cost of using edge computing power. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0048] Figure 1 A schematic diagram of a process flow of a method for offloading end-to-end collaborative reasoning tasks for differentiated computing power pricing provided in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of reasoning task segmentation and reasoning provided in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of the structure of a device-edge collaborative reasoning task offloading device for differentiated computing power pricing provided in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0053] As deep learning is increasingly used in various smart devices, how to efficiently perform reasoning tasks in resource-constrained environments becomes a key issue.

[0054] To solve this problem, the end-edge collaborative reasoning technology was proposed. This technology divides the deep learning model between the terminal device and the edge server, so that part of the task runs on the terminal device and the other part is completed on the edge server. This makes full use of the computing resources of both parties and optimizes the entire reasoning process.

[0055] However, most of the existing edge-to-end collaborative reasoning optimization methods focus on the scenarios of a single terminal device and a single edge server, and fail to adapt to the complex environment of multiple terminals and multiple edge servers. In the prior art, although there are edge-to-end collaborative reasoning optimization methods based on game theory, such methods are mainly applicable to static environments and are difficult to cope with network fluctuations and device load changes under dynamic network conditions.

[0056] In addition, with the continuous growth of terminal devices, deep learning applications, and edge server deployment, it is becoming more common for multiple terminal devices to request computing resources at the same time. Due to the limited computing power, network bandwidth, and storage resources of edge servers, resource competition is easily triggered, resulting in a decrease in inference performance.

[0057] Based on this, various embodiments of the present application propose a distributed optimization method for end-edge collaborative reasoning for differentiated computing power pricing, so that each terminal device adopts a distributed reinforcement learning algorithm to decide the segmentation points of the deep learning model and the collaborative edge server, thereby weighing the efficiency of each terminal device in performing reasoning tasks and the cost of using edge computing power in a dynamic communication and computing environment.

[0058] The various embodiments of the present application are described in further detail below in conjunction with the accompanying drawings.

[0059] The present application embodiment provides a method for offloading edge collaborative reasoning tasks for differentiated computing power pricing, such as Figure 1 As shown, the method includes steps S101-S103.

[0060] S101: Acquire experience data in the experience pool.

[0061] Among them, the empirical data includes the state vector, output strategy, and immediate benefit of each terminal device at each moment; the state vector includes at least the payment fee, the reasoning task split point and the edge server identifier; the payment fee represents the service fee corresponding to the amount of computing resources allocated by the edge server to complete the remaining reasoning tasks of each terminal device; the output strategy includes the reasoning task split point and the edge server identifier; the immediate benefit is used to represent the immediate benefit of each terminal device after unloading the reasoning task.

[0062] In the embodiment of the present application, the empirical data is a data set used by the reinforcement learning model for learning and optimization. The state, action, and benefits of each terminal device at different times will be recorded for subsequent learning and decision-making of the reinforcement learning model.

[0063] The state vector is a collection of all possible variables used to describe the current state of the system. In the embodiment of the present application, the state vector may include payment fees, reasoning task split points (hereinafter referred to as "split points"), edge server identifiers, and may also include the channel gain of the link between the current terminal device and each edge server.

[0064] The payment fee refers to the service fee paid by the terminal device in order to allow the edge server to complete its remaining inference tasks. The payment fee is related to the amount of computing resources allocated by the edge server to the terminal device's inference task. Including the payment fee in the state vector can help weigh the cost when offloading the inference task of the terminal device.

[0065] The reasoning task split point refers to the point where the terminal device's reasoning task is divided between the terminal device and the edge server (for details, please refer to Figure 2 ), the role of the split point is to execute the part of the reasoning task before the split point on the terminal device, and the part of the reasoning task after the split point on the edge server. By choosing the split point, the reasoning task can be distributed between the terminal device and the edge server to achieve the purpose of balancing the computing load, reducing latency, and saving energy consumption.

[0066] The edge server ID is used to uniquely identify the selected edge server, helping the system determine which edge server to assign the current inference task to. Considering which edge server to select to complete the inference calculation of the remaining inference tasks is crucial to optimizing task allocation, because different edge servers may have different available computing resources and processing capabilities.

[0067] The immediate benefit is used to measure the performance of the strategy / task allocation strategy selected at the current moment. It indicates the immediate benefit obtained by each terminal device after offloading the inference task to the edge server. The immediate benefit is usually obtained by combining factors such as inference latency and resource usage cost.

[0068] By storing the above data as experience data in the experience pool, the process and results of historical reasoning task allocation are recorded to help the reinforcement learning model make better decisions in future task allocation. For example, the reinforcement learning model will randomly sample data from the experience pool for training, gradually improving the efficiency and reasoning performance of reasoning task allocation.

[0069] S102: Update the reinforcement learning model based on the empirical data.

[0070] The reinforcement learning model extracts these experience data from the experience pool, analyzes which output strategies have obtained higher benefits in the past and which strategies have led to poor benefits, and trains based on these experience data, so that when the reinforcement learning model encounters similar situations in the future, it can select task allocation strategies that can bring higher benefits (such as shorter inference latency or lower computing costs). By continuously learning from experience data, the task allocation strategy output by the system will be gradually optimized, and the inference efficiency of inference tasks and the computing resource utilization of edge servers will be improved.

[0071] S103: For each terminal device, a state vector is constructed according to the channel gain of the link between each edge server and the terminal device at the current moment, the payment fee at the previous moment, the inference task split point output at the previous moment, and the edge server identifier output at the previous moment, and the state vector is input into the updated reinforcement learning model to obtain the task allocation strategy of the inference task of each terminal device, so that each terminal device can offload the inference task to the edge server for inference calculation according to the task allocation strategy.

[0072] Channel gain is a parameter that measures the transmission quality of the edge computing network, reflecting the quality of communication between the terminal device and the edge server. Higher channel gain means better communication conditions, which may lead to lower transmission delays. The embodiment of the present application constructs a state vector (including channel gain, payment fees, split points, etc.) for each terminal device at each moment, and then inputs the state vector into the reinforcement learning model. The model will output the optimal task allocation strategy, including how to perform the split calculation at which layer of the original reasoning task model and which edge server to select for reasoning calculation. The terminal device will offload the task to the edge server according to the task allocation strategy to achieve efficient reasoning calculation.

[0073] In an embodiment of the present application, by incorporating payment fees into the state vector, the reinforcement learning model can effectively optimize the amount of computing resources on the edge server side of the reasoning task allocation, reducing the service fees that the terminal device needs to pay when completing the task.

[0074] Based on the channel gain of the link, the payment fee at the previous moment, and the inference task split point, and by linking the computing resources of the edge server with the payment fee of the terminal device, the reinforcement learning model can make the optimal task offloading decision, minimize the inference latency and the cost of using edge computing power.

[0075] In one embodiment, the experience data in the experience pool is obtained in the following manner: for each terminal device at each moment, a state vector is constructed according to the channel gain of the link between each edge server and the terminal device, the payment fee at the previous moment, the reasoning task segmentation point output at the previous moment, and the edge server identifier output at the previous moment; the state vector is input into a pre-constructed reinforcement learning model to obtain an output action space; the action space includes the selection probability of each reasoning task segmentation point and the selection probability of each edge server; according to the selection probability of each reasoning task segmentation point and the selection probability of each edge server, an output strategy including the reasoning task segmentation point and the edge server identifier is determined, so that the terminal device offloads the target reasoning task to the edge server corresponding to the edge server identifier for reasoning calculation according to the output strategy; computing resources are allocated to the target reasoning task according to the output strategy, and the payment fee is calculated according to the amount of computing resources allocated to the target reasoning task; the reasoning time after the target reasoning task completes the reasoning calculation is obtained, and the immediate benefit of each terminal device is calculated according to the reasoning time and the payment fee; the state vector, output strategy and immediate benefit at each moment are used as the experience data at each moment and stored in the experience pool.

[0076] For example, in practical applications, taking a single time k as an example, each terminal device m (1≤m≤M) receives a signal from an edge server, and then uses a channel estimation algorithm (such as minimum mean square error estimation, maximum likelihood estimation, etc.) to estimate the channel gain of each edge server based on parameters such as received signal strength, time delay, and frequency change.

[0077] Record the payment amount of the last moment k-1 The split point selected at the last moment and the edge server selected at the last moment Build Status

[0078] And the status Input to the policy network and get the policy probability distribution with output dimension LN Each element in the strategy probability distribution represents the probability of selecting the lth (1≤l≤L) segmentation point and the nth (1≤n≤N) edge server.

[0079] The output strategy is obtained by selecting from the strategy probability distribution. For example, a random selection can be performed, that is, any split point and any edge server can be randomly selected as the output strategy, or a split point with a higher probability and an edge server with a higher probability can be selected as the output strategy. By introducing a probability-based selection mechanism, this method can dynamically allocate task split points and edge servers according to the current network environment (such as channel gain) and the characteristics of the reasoning task. This dynamic allocation strategy makes task offloading more adaptive and helps improve resource utilization efficiency.

[0080] After the terminal device m inputs the target reasoning task to be reasoned from the input layer of the reinforcement learning model and obtains the output strategy, it The layer obtains the remaining reasoning tasks and transmits the remaining reasoning tasks to the edge server

[0081]

[0082] Allocate computing resources of edge servers to target reasoning tasks according to the output strategy, and calculate payment fees based on the amount of computing resources allocated to the target reasoning tasks Specifically, the payment fee can be calculated based on the amount of computing resources allocated and the pricing amount of the computing resources.

[0083] Get the inference time after the target inference task completes the inference calculation The inference time includes the inference time on the edge server side. Then, the immediate benefit of each terminal device is calculated based on the inference time and payment fee. The state vector, output strategy, and immediate benefit obtained according to the above steps are used as the empirical data at time k

[0084] Perform the above steps at each moment in the same way, so that the state vector at each moment can be obtained, and the strategy and immediate benefit can be output as the experience data at each moment. And the empirical data obtained at each moment Deposit into the experience pool with capacity D in a first-in-first-out manner , where t is any moment.

[0085] In one implementation, an output strategy including the reasoning task splitting point and the edge server identifier is determined based on the selection probability of each reasoning task splitting point and the selection probability of each edge server, including: randomly selecting each reasoning task splitting point and each edge server according to the selection probability of each reasoning task splitting point and the selection probability of each edge server, respectively, to obtain an output strategy including the reasoning task splitting point and the edge server identifier.

[0086] After the state vector is input into the reinforcement learning model, a probability distribution including the selection probability of each inference task split point and the selection probability of each edge server is obtained. Randomly sample in the probability distribution to obtain the sampling result x and get the randomly selected split point and edge servers The output strategy is

[0087] By randomly selecting task split points and edge server identifiers, the embodiment of the present application can flexibly respond to network environments with greater uncertainty, avoid system bottlenecks or excessive congestion caused by fixed allocation, and effectively achieve load balancing between edge servers, avoiding overloading of individual servers while other server resources are idle. In high-concurrency scenarios, it helps to reduce the overall computing delay of reasoning tasks, improve resource utilization and the response speed of reasoning tasks.

[0088] In one embodiment, computing resources are allocated to a target reasoning task according to an output strategy, and a payment fee is calculated based on the amount of computing resources allocated to the target reasoning task, including: obtaining the remaining computing amount of the target reasoning task according to the reasoning task split point in the output strategy; determining the amount of computing resources allocated to the target reasoning task according to the remaining computing amount of the target reasoning task; the allocated computing resources are part of the available computing resources of the edge server corresponding to the edge server identifier in the output strategy; and calculating the payment fee of the terminal device based on the amount of computing resources allocated to the target reasoning task and a preset computing resource usage pricing.

[0089] According to the reasoning task split point in the output strategy Get the remaining computational effort of the target reasoning task According to the remaining computation of the target reasoning task Determine the amount of computing resources allocated for the target reasoning task. For example, if terminal device A needs to perform a complex image recognition task, its remaining computing resources Terminal device B needs to process sensor data, and its remaining computing power The edge servers selected by terminal device A and terminal device B are both edge server S1, device B is allocated 20% of the resources of S1, and device A is allocated 70% of the computing resources of S1. It should be understood that the amount of computing resources allocated to each reasoning task is determined based on the edge server selected in the output strategy. Specifically, the amount of computing resources allocated is a portion of the computing resources currently available to the selected edge server. The available computing resources of the edge server are dynamic and may depend on the number and type of tasks currently being processed by the edge server. The amount of computing resources allocated to the current reasoning task is a portion of the computing resources currently available to the edge server, rather than all the available resources, which means that task allocation will take into account the remaining computing requirements of other reasoning tasks to ensure that the resource allocation of the edge server is rationalized and does not cause resource waste or server overload.

[0090] After the inference task is completed, obtain the inference result and the inference time on the edge server. The edge service provider calculates the amount that the terminal device m needs to pay based on the differentiated pricing standard. The function f is a monotonically increasing pricing function. For example, the pricing function can be The function can be expressed as This form, in which represents the amount of computing resources of edge server n allocated to the inference task issued by terminal device m at time k; 1 and v 2 are greater than 0, v 1 It can be used to represent the basic service fee or starting fee, representing the basic cost required by the edge server to provide computing resources (such as server maintenance, basic operating costs, etc.). 2 It can be used to characterize the variable cost coefficient that increases according to the amount of tasks or computing, and is used to calculate the cost of proportionally increasing the computing resources of the allocated edge server. 2 The edge server provider can adjust the charges according to the amount of computing tasks, thus achieving more flexible pricing. For example, if v 2 The higher the value, the faster the payment will increase with the amount of computation, which is suitable for situations where computing resources are tight or costly. 2 The lower the value, the slower the payment amount grows, which is suitable for situations with abundant resources or low costs. The edge server provider sends the required payment amount to each terminal device, and returns the inference result to the corresponding terminal device after receiving the payment.

[0091] The embodiment of the present application allocates the corresponding computing resources of the reasoning task by the remaining computing amount and the available computing resources of the edge server, which can ensure that each task obtains the most appropriate amount of computing resources and avoids the problem of waste or insufficient resources. This helps to improve the utilization of overall resources. At the same time, the differentiated pricing mechanism can be dynamically priced according to the actual reasoning time of the reasoning task and the computing power of the edge server, and can flexibly respond to the remaining computing requirements of different reasoning tasks. For tasks with higher computing requirements or longer time, higher fees will be charged accordingly, thereby ensuring the fairness and rationality of pricing. Through the above-mentioned embodiment, the reasoning results can be returned to the terminal device in time after the calculation is completed, and detailed cost information can be provided at the same time, so that the terminal device can not only quickly obtain the calculation results, but also clearly understand the resource consumption and cost expenditure of each task, which increases the transparency of the edge computing system.

[0092] In one embodiment, the amount of computing resources allocated to the target reasoning task is determined based on the remaining computing amount of the target reasoning task, including: determining the total remaining computing amount of all reasoning tasks required for reasoning calculations by the edge server corresponding to the edge server identifier in the output strategy; calculating the ratio between the total remaining computing amount and the remaining computing amount of the target reasoning task required for reasoning calculations by the edge server, and determining the amount of computing resources allocated to the target reasoning task.

[0093] In practical applications, edge service providers allocate computing power (i.e., the remaining computing power) required by each terminal device on each edge server. The remaining computing capacity is the part of the inference task that the terminal device unloads to the server at a specific moment that still requires inference calculation). The computing resources per unit time are proportionally allocated to the corresponding terminal device to perform the inference calculation of the inference task.

[0094] For edge server n, the calculation formula for the allocation ratio of computing resources per unit time is as follows:

[0095]

[0096] in, is the allocation ratio of computing resources per unit time for each reasoning task; represents the task allocation strategy of the mth (1≤m≤M) terminal device at time k, that is, whether edge server n is selected, is the indicator function, which is 1 if device m selects server n at time k, and 0 otherwise; is the remaining computational effort of the inference task.

[0097] It can be understood that the numerator of the formula represents the remaining computing amount of the inference task of a single terminal device that the edge server n needs to process at time k; the denominator represents the sum of the remaining computing amount of all inference tasks of all terminal devices that the edge server n needs to infer and calculate at time k.

[0098] In the embodiment of the present application, by allocating computing resources in proportion, it is possible to effectively cope with the situation where multiple terminal devices simultaneously offload their respective reasoning tasks to the same edge server. The computing resources obtained by each terminal device are proportional to its actual task requirements, ensuring balanced load distribution and avoiding the situation where terminal devices monopolize resources. This allows the system to flexibly adapt to the task complexity of the dynamic network and improves the concurrent processing capability and processing efficiency of the edge computing system. For terminal devices with different computing requirements, it can also reduce waiting time and speed up the feedback speed of reasoning calculations.

[0099] In one embodiment, the instant benefit of each terminal device is calculated according to the inference time and the payment fee, including: calculating the instant benefit of each terminal device by the following calculation formula:

[0100]

[0101] in, is the benefit function, corresponding to the immediate benefit of the terminal equipment; is the total inference delay of the target inference task of the terminal device; w is the weight parameter of the payment fee; is the payment fee of the terminal device; k is the time.

[0102] Immediate benefits The inference time and payment of fees The negative sign in the formula indicates that an increase in inference time and cost will reduce the immediate benefit of the terminal device. The weight parameter w controls the relative impact of cost on immediate benefit. For example, if w is large, the impact of payment costs on immediate benefit will be more significant. The weight parameter w can flexibly adjust the relative impact of inference time and payment costs on benefits, and can be adjusted according to actual application requirements. For example, in application scenarios that are very sensitive to latency, a higher weight can be given to inference time; and in cost-sensitive scenarios, the weight of costs can be increased. The weight parameter can be set according to actual task requirements and application scenarios, and is not limited here.

[0103] The embodiment of the present application provides a quantitative instant benefit evaluation method, which enables the system to evaluate the benefit of each terminal device based on a comprehensive consideration of inference delay and cost, and can more intuitively understand the overall benefit of the current task, and then adjust its behavior (such as determining the selection of split points and edge servers).

[0104] The embodiments of the present application enable the system to make adaptive adjustments based on real-time network conditions and task requirements. For example, when the network conditions are good and computing resources are abundant, the system can reduce inference delays, thereby improving efficiency; when resources are tight or costs are high, the system will give priority to cost factors to ensure maximum benefits. Through real-time benefit evaluation, terminal devices can obtain lower inference delays or pay less fees, which encourages reinforcement learning models to select optimized offloading strategies, and helps edge computing platforms allocate resources more reasonably and reduce unnecessary computing overhead.

[0105] In one embodiment, the reinforcement learning model is updated according to the experience data, including: for each moment, sampling a plurality of experience data in the experience pool to obtain a sampled data set, and using the sampled data set to update the weight parameters of the value network, the weight parameters of the average strategy network, and the weight parameters of the strategy network in the reinforcement learning model.

[0106] At each moment, a batch B consisting of B samples sampled from the experience pool is used to update the value network, average strategy network, and weight parameters of the strategy network. For details, please refer to the following formula:

[0107] The weight parameter update formula of the value network is:

[0108]

[0109] in, is the weight parameter of the value network at time k+1 after the update; is the weight parameter of the value network at time k after the update; α is the learning rate; B is the batch size, indicating that B samples are sampled from the experience pool; is the immediate benefit of the i-th sample terminal device m; γ is the discount factor; is the action space; is the state vector corresponding to the i+1th sample, is the state vector corresponding to the i-th sample; This is the predicted Q value at the current moment, indicating that in the current state Next, the terminal device selects an action The Q value that can be obtained; Indicates the state of the terminal device at the next moment Select the action x that brings the maximum reward and return the target Q value of the action, which reflects the cumulative reward in the future. It means that among all possible actions, the action x that brings the maximum Q value is selected, which belongs to the maximization principle in reinforcement learning. The square of the error between the target Q value and the predicted Q value. By minimizing this error, the reinforcement learning model can adjust the weight parameters so that the predicted Q value is closer to the actual Q value, thereby optimizing the strategy.

[0110] The weight parameter update formula of the average policy network is:

[0111]

[0112] in, is the weight parameter of the average strategy network at time k+1 after update; is the weight parameter of the average policy network at time k after the update; α is the learning rate; B is the batch size, indicating that B samples are sampled from the experience pool; X is the action space; is the state vector corresponding to the i-th sample; yes The number of historical visits, The more times a state has been accessed in the past, the The greater the impact on the update of the weight parameter, the more it represents the state Contains more historical experience; Characterizes the current average policy network in state The probability of selecting action x is based on the current weight That is, the probability distribution output by the policy network is used to represent the selection probability of each action in a given state. Characterizing the historical policy network in state The probability of selecting action x, based on the parameters of the current historical policy network Represents the decision-making basis of the past output strategy in the same state. By calculating the difference between the probability of the current strategy network output and the probability of the historical strategy output, the difference between the two is minimized, so that the system can gradually optimize the current strategy network and make better choices in future decisions.

[0113] The weight parameter update formula of the policy network is:

[0114]

[0115] in, is the weight parameter of the strategy network at time k+1 after update; is the weight parameter of the policy network at time k after the update; α is the learning rate; B is the batch size, indicating that B samples are sampled from the experience pool; is the action space; is the state vector corresponding to the i+1th sample, is the state vector corresponding to the i-th sample; is the predicted Q value at the current moment, indicating the Q value that the terminal device can obtain by selecting action x' in the current state; the exponential function exp() indicates the Q value that can be obtained in the future state. By calculating the expected rewards of possible future actions, the system can predict the rewards that can be obtained after performing a certain action in the next step. is the size of the action space, i.e. the total number of possible actions that can be taken in the system. Represents the number of actions after excluding an optimal action from the action space. By squaring the difference between the probability output by the current policy network and the probability of the optimal action and minimizing this difference, the system can continuously adjust the policy network to make better decisions in future task allocation.

[0116] In order to implement the end-edge collaborative reasoning task method for differentiated computing power pricing of the embodiment of the present application, the embodiment of the present application also provides an end-edge collaborative reasoning task unloading device for differentiated computing power pricing, such as Figure 3 As shown, the device comprises:

[0117] An acquisition unit 301 is used to acquire experience data in an experience pool;

[0118] A model updating unit 302, configured to update the reinforcement learning model according to the empirical data;

[0119] The processing unit 303 is used to construct a state vector for each terminal device according to the channel gain of the link between each edge server and the terminal device at the current moment, the payment fee at the previous moment, the reasoning task split point output at the previous moment, and the edge server identifier output at the previous moment, input the state vector into the updated reinforcement learning model, and obtain the task allocation strategy of the reasoning task of each terminal device, so that each terminal device offloads the reasoning task to the edge server for reasoning calculation according to the task allocation strategy;

[0120] Among them, the empirical data includes the state vector, output strategy, and immediate benefit of each terminal device at each moment; the state vector includes at least payment fee, reasoning task split point and edge server identifier; the payment fee represents the service fee corresponding to the amount of computing resources allocated by the edge server to complete the remaining reasoning tasks of each terminal device; the output strategy includes the reasoning task split point and edge server identifier; the immediate benefit is used to represent the immediate income of each terminal device after unloading the reasoning task.

[0121] In one embodiment, the processing unit 303 is further configured to construct a state vector for each terminal device at each moment according to the channel gain of the link between each edge server and the terminal device, the payment fee at the last moment, the reasoning task split point output at the last moment, and the edge server identifier output at the last moment;

[0122] The processing unit 303 is further used to input the state vector into a pre-built reinforcement learning model to obtain an output action space; the action space includes a selection probability of each reasoning task split point and a selection probability of each edge server;

[0123] A determination unit, configured to determine an output strategy including a reasoning task split point and an edge server identifier according to a selection probability of each reasoning task split point and a selection probability of each edge server, so that the terminal device offloads the target reasoning task to an edge server corresponding to the edge server identifier for reasoning calculation according to the output strategy;

[0124] The processing unit 303 is further used to allocate computing resources to the target reasoning task according to the output strategy, and calculate the payment fee according to the amount of computing resources allocated to the target reasoning task;

[0125] The processing unit 303 is further used to obtain the inference time after the target inference task completes the inference calculation, and calculate the immediate benefit of each of the terminal devices according to the inference time and the payment fee;

[0126] The processing unit 303 is further used to store the state vector, the output strategy and the immediate benefit at each moment as the experience data at each moment in the experience pool.

[0127] In one embodiment, the processing unit 303 is further used to randomly select each reasoning task splitting point and each edge server according to the selection probability of each reasoning task splitting point and the selection probability of each edge server, and obtain an output strategy including the reasoning task splitting point and the edge server identifier.

[0128] In one embodiment, the processing unit 303 is further configured to obtain the remaining computation amount of the target reasoning task according to the reasoning task segmentation point in the output strategy;

[0129] The processing unit 303 is further configured to determine the amount of computing resources allocated to the target reasoning task according to the remaining computing amount of the target reasoning task; the allocated computing resources are part of the available computing resources of the edge server corresponding to the edge server identifier in the output strategy;

[0130] The processing unit 303 is further used to calculate the payment fee of the terminal device according to the computing resources allocated to the target reasoning task and the preset computing resource usage pricing.

[0131] In one embodiment, the determining unit is further configured to determine, for an edge server corresponding to the edge server identifier in the output strategy, a sum of remaining computation amounts of all the inference tasks required for inference computation by the edge server;

[0132] The processing unit 303 is further configured to calculate the ratio between the total amount of remaining computation and the amount of remaining computation of the target reasoning task required for the reasoning calculation by the edge server, and determine the amount of computing resources allocated to the target reasoning task.

[0133] In one embodiment, the instant benefit of each terminal device is calculated by the following calculation formula:

[0134]

[0135] in, is a benefit function corresponding to the instant benefit of the terminal device; is the total inference latency of the target inference task of the terminal device; is the payment fee of the terminal device; k is the time.

[0136] In one embodiment, the model updating unit 302 is further used to obtain a sampling data set from multiple experience data in the experience pool at each moment, and use the sampling data set to update the weight parameters of the value network, the weight parameters of the average strategy network, and the weight parameters of the strategy network in the reinforcement learning model.

[0137] In actual application, the processing unit 303 can be implemented by a processor in an electronic device combined with a communication interface, and the determination unit, the acquisition unit 301, and the model update unit 302 can be implemented by the communication interface in an end-edge collaborative reasoning task offloading device for differentiated computing power pricing.

[0138] It should be noted that: the above-mentioned embodiment provides an end-to-end collaborative reasoning task unloading device for differentiated computing power pricing. When performing end-to-end collaborative reasoning task unloading for differentiated computing power pricing, only the division of the above-mentioned program modules is used as an example. In actual applications, the above-mentioned processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the end-to-end collaborative reasoning task unloading device for differentiated computing power pricing provided in the above-mentioned embodiment and the end-to-end collaborative reasoning task unloading method embodiment for differentiated computing power pricing belong to the same concept. The specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0139] Based on the hardware implementation of the above program modules, and in order to implement a method for offloading end-to-end collaborative reasoning tasks for differentiated computing power pricing provided in an embodiment of the present application, an embodiment of the present application also provides an electronic device, such as Figure 4 As shown, the electronic device 400 includes:

[0140] CPU 401, memory 402 and input / output interface 403;

[0141] The memory 402 is a temporary storage memory or a permanent storage memory;

[0142] The central processor 401 is configured to communicate with the memory 402 and execute instruction operations in the memory 402 to perform any of the above-mentioned end-edge collaborative reasoning task offloading methods for differentiated computing power pricing.

[0143] Of course, in actual application, the various components in the electronic device 400 are coupled together through the bus system 405. It can be understood that the bus system 405 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 405 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, Figure 3 Various buses are labeled as bus system 405 .

[0144] The memory 402 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device 400. Examples of such data include: any computer program used to operate on the electronic device 400.

[0145] It is understandable that when the processor in the electronic device described above executes the computer program, it can also implement the functions of the various units in the above-mentioned corresponding device embodiments, which will not be repeated here. Exemplarily, the computer program can be divided into one or more modules / units, and one or more modules / units are stored in the memory and executed by the processor to complete the various embodiments of the present application. One or more modules / units can be a series of computer program instruction segments that can perform specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device. For example, the computer program can be divided into the various units in the above-mentioned electronic device, and each unit can implement the specific functions described in the above-mentioned corresponding electronic device.

[0146] The electronic device may be a computing device such as a desktop computer, a notebook, a PDA, and a cloud server. The electronic device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the processor and the memory are merely examples of electronic devices and do not constitute a limitation on the electronic device. The electronic device may include more or fewer components, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0147] The processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of an electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device.

[0148] The memory can be used to store computer programs and / or modules. The processor realizes various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory can include a high-speed random access memory, and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (SMC, Smart Media Card), a secure digital (SD, Secure Digital) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0149] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed on a computer, the computer is enabled to execute any one of the above-mentioned methods for offloading end-edge collaborative reasoning tasks for differentiated computing power pricing.

[0150] An embodiment of the present application also provides a computer program product having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, it is used to implement the end-edge collaborative reasoning task offloading method for differentiated computing power pricing described in the first aspect of the embodiment of the present application or any specific implementation method of the first aspect.

[0151] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0152] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0153] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0154] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0155] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A method for offloading edge-to-end collaborative reasoning tasks for differentiated computing power pricing, characterized in that: include: Get the experience data in the experience pool; updating the reinforcement learning model according to the empirical data; For each terminal device, a state vector is constructed according to the channel gain of the link between each edge server and the terminal device at the current moment, the payment fee at the previous moment, the reasoning task split point output at the previous moment, and the edge server identifier output at the previous moment, and the state vector is input into the updated reinforcement learning model to obtain the task allocation strategy of the reasoning task of each terminal device, so that each terminal device offloads the reasoning task to the edge server for reasoning calculation according to the task allocation strategy; The empirical data includes the state vector, output strategy, and immediate benefit of each terminal device at each moment; The state vector includes at least a payment fee, an inference task split point and an edge server identifier; the payment fee represents the service fee corresponding to the amount of computing resources allocated by the edge server to complete the remaining inference tasks of each of the terminal devices; the output strategy includes the inference task split point and the edge server identifier; the immediate benefit is used to represent the immediate benefit of each of the terminal devices after unloading the inference task.

2. The method for offloading edge-to-end collaborative reasoning tasks for differentiated computing power pricing according to claim 1, characterized in that: The experience data in the experience pool is obtained in the following way: For each terminal device at each moment, a state vector is constructed according to the channel gain of the link between each edge server and the terminal device, the payment fee at the last moment, the reasoning task split point output at the last moment, and the edge server identifier output at the last moment; Input the state vector into a pre-built reinforcement learning model to obtain an output action space; the action space includes the selection probability of each reasoning task split point and the selection probability of each edge server; Determine an output strategy including the reasoning task split point and the edge server identifier according to the selection probability of each reasoning task split point and the selection probability of each edge server, so that the terminal device offloads the target reasoning task to the edge server corresponding to the edge server identifier for reasoning calculation according to the output strategy; Allocating computing resources to the target reasoning task according to the output strategy, and calculating payment fees according to the amount of computing resources allocated to the target reasoning task; Obtaining the inference time after the target inference task completes the inference calculation, and calculating the immediate benefit of each of the terminal devices according to the inference time and the payment fee; The state vector, the output strategy and the immediate benefit at each moment are taken as the experience data at each moment and stored in the experience pool.

3. The method for offloading edge-to-end collaborative reasoning tasks for differentiated computing power pricing according to claim 2, characterized in that: The step of determining the output strategy including the reasoning task split point and the edge server identifier according to the selection probability of each reasoning task split point and the selection probability of each edge server includes: According to the selection probability of each reasoning task split point and the selection probability of each edge server, each reasoning task split point and each edge server are randomly selected respectively to obtain an output strategy including the reasoning task split point and the edge server identifier.

4. The method for offloading edge-to-end collaborative reasoning tasks for differentiated computing power pricing according to claim 2, characterized in that: The allocating computing resources to the target reasoning task according to the output strategy, and calculating the payment fee according to the amount of computing resources allocated to the target reasoning task, includes: Obtaining the remaining computational amount of the target reasoning task according to the reasoning task splitting point in the output strategy; Determine the amount of computing resources allocated to the target reasoning task according to the remaining computing amount of the target reasoning task; the allocated computing resources are part of the available computing resources of the edge server corresponding to the edge server identifier in the output strategy; The payment fee of the terminal device is calculated according to the amount of computing resources allocated to the target reasoning task and the preset pricing for the usage of computing resources.

5. The method for offloading edge-to-end collaborative reasoning tasks for differentiated computing power pricing according to claim 4, characterized in that: The determining, according to the remaining computation amount of the target reasoning task, the amount of computing resources allocated to the target reasoning task comprises: For the edge server corresponding to the edge server identifier in the output strategy, determine the sum of remaining computation amounts of all the inference tasks required for inference computation by the edge server; The ratio between the total remaining computing amount and the remaining computing amount of the target reasoning task required for the reasoning calculation by the edge server is calculated, and the amount of computing resources allocated to the target reasoning task is determined.

6. The method for offloading edge-to-end collaborative reasoning tasks for differentiated computing power pricing according to claim 2, characterized in that: The calculating the immediate benefit of each terminal device according to the inference time and the payment fee includes: The instant benefit of each terminal device is calculated by the following formula: in, is a benefit function corresponding to the instant benefit of the terminal device; is the total inference latency of the target inference task of the terminal device; is the payment fee of the terminal device; k is the time.

7. The method for offloading edge-to-end collaborative reasoning tasks for differentiated computing power pricing according to claim 1, characterized in that: The updating of the reinforcement learning model according to the empirical data comprises: At each moment, a sampling data set is obtained by sampling multiple experience data in the experience pool, and the sampling data set is used to update the weight parameters of the value network, the weight parameters of the average strategy network, and the weight parameters of the strategy network in the reinforcement learning model.

8. A device for offloading edge collaborative reasoning tasks for differentiated computing power pricing, characterized in that: include: An acquisition unit, used for acquiring experience data in the experience pool; A model updating unit, used to update the reinforcement learning model according to the experience data; A processing unit is used to construct a state vector for each terminal device according to the channel gain of the link between each edge server and the terminal device at the current moment, the payment fee at the previous moment, the reasoning task split point output at the previous moment, and the edge server identifier output at the previous moment, input the state vector into the updated reinforcement learning model, and obtain a task allocation strategy for the reasoning task of each terminal device, so that each terminal device offloads the reasoning task to the edge server for reasoning calculation according to the task allocation strategy; The empirical data includes the state vector, output strategy, and immediate benefit of each terminal device at each moment; The state vector includes at least a payment fee, an inference task split point and an edge server identifier; the payment fee represents the service fee corresponding to the amount of computing resources allocated by the edge server to complete the remaining inference tasks of each of the terminal devices; the output strategy includes the inference task split point and the edge server identifier; the immediate benefit is used to represent the immediate benefit of each of the terminal devices after unloading the inference task.

9. An electronic device, characterized in that: include: CPU, memory and input / output interface; The memory is a short-term storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instruction operations in the memory to perform the end-edge collaborative reasoning task offloading method for differentiated computing power pricing as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program runs on a computer, the end-edge collaborative reasoning task offloading method for differentiated computing power pricing as described in any one of claims 1 to 7 is executed.

Images