Device-edge collaborative inference task offloading method for differentiated computing power pricing

Abstract

The embodiment of the present application discloses a method for offloading end-to-end collaborative reasoning tasks for differentiated computing power pricing. The method of the embodiment of the present application includes: obtaining experience data in the experience pool; updating the reinforcement learning model according to the experience data; constructing a state vector according to the channel gain at the current moment, the payment fee at the previous moment, the reasoning task split point and the edge server identifier, inputting the state vector into the updated reinforcement learning model, and obtaining the task allocation strategy for each reasoning task, so that each terminal device offloads the reasoning task to the edge server for reasoning calculation according to the task allocation strategy; wherein the experience data includes at least the payment fee of each terminal device, representing the service fee corresponding to the amount of computing resources allocated by the edge server to complete the remaining reasoning tasks of each terminal device. The embodiment of the present application can minimize the delay of reasoning and the cost of using edge computing power for each terminal device.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of edge computing technology, and in particular to a method for offloading end-to-end collaborative reasoning tasks for differentiated computing power pricing. Background Art

[0002] With the increasing application of deep learning in various intelligent devices, how to efficiently perform reasoning tasks in resource-constrained environments has become a key issue.

[0003] To solve this problem, the end-edge collaborative inference technology was proposed. This technology divides the deep learning model between the terminal device and the edge server, so that part of the task runs on the terminal device and the other part is completed on the edge server, thereby making full use of the computing resources of both parties and optimizing the entire inference process.

[0004] However, existing methods for optimizing device-edge collaborative inference mostly focus on scenarios involving a single terminal device and a single edge server, failing to adapt to the complex environments of multiple terminals and multiple edge servers. While existing methods for optimizing device-edge collaborative inference based on game theory exist, these methods are primarily suitable for static environments and struggle to cope with network fluctuations and device load changes in dynamic network conditions.

[0005] Furthermore, with the growing number of end-devices, deep learning applications, and edge server deployments, it's becoming increasingly common for multiple end-devices to simultaneously request computing resources. Due to the limited computing power, network bandwidth, and storage resources of edge servers, resource contention can easily occur, leading to degraded inference performance. Summary of the Invention

[0006] Based on the above problems, an embodiment of the present application provides an end-edge collaborative reasoning task offloading method for differentiated computing power pricing, with the aim of optimizing reasoning task allocation and resource utilization efficiency in an edge computing environment.

[0007] In a first aspect, embodiments of the present application provide a method for offloading device-edge collaborative inference tasks for differentiated computing power pricing, including:

[0008] Get experience data from the experience pool;

[0009] updating the reinforcement learning model based on the empirical data;

[0010] For each terminal device, a state vector is constructed based on the channel gain of the link between each edge server and the terminal device at the current moment, the payment fee at the previous moment, the reasoning task split point output at the previous moment, and the edge server identifier output at the previous moment. The state vector is input into the updated reinforcement learning model to obtain a task allocation strategy for the reasoning task of each terminal device, so that each terminal device offloads the reasoning task to the edge server for reasoning calculation according to the task allocation strategy;

[0011] Among them, the empirical data includes the state vector, output strategy, and immediate benefit of each terminal device at each moment; the state vector includes at least payment fee, reasoning task split point and edge server identifier; the payment fee represents the service fee corresponding to the amount of computing resources allocated by the edge server to complete the remaining reasoning tasks of each terminal device; the output strategy includes the reasoning task split point and edge server identifier; the immediate benefit is used to represent the immediate income of each terminal device after unloading the reasoning task.

[0012] In one embodiment, the experience data in the experience pool is obtained in the following manner:

[0013] For each terminal device at each moment, construct a state vector based on the channel gain of the link between each edge server and the terminal device, the payment fee at the previous moment, the inference task split point output at the previous moment, and the edge server identifier output at the previous moment;

[0014] Input the state vector into a pre-built reinforcement learning model to obtain an output action space; the action space includes the selection probability of each inference task split point and the selection probability of each edge server;

[0015] Determining, based on the selection probability of each inference task split point and the selection probability of each edge server, an output strategy including the inference task split point and the edge server identifier, so that the terminal device offloads the target inference task to the edge server corresponding to the edge server identifier for inference calculation according to the output strategy;

[0016] Allocating computing resources to the target reasoning task according to the output strategy, and calculating payment fees according to the amount of computing resources allocated to the target reasoning task;

[0017] Obtaining the inference time after the target inference task completes the inference calculation, and calculating the immediate benefit of each of the terminal devices based on the inference time and the payment fee;

[0018] The state vector, the output strategy and the immediate benefit at each moment are used as the experience data at each moment and stored in the experience pool.

[0019] In one embodiment, determining the output strategy including the inference task split point and the edge server identifier based on the selection probability of each inference task split point and the selection probability of each edge server includes:

[0020] According to the selection probability of each reasoning task split point and the selection probability of each edge server, each reasoning task split point and each edge server are randomly selected respectively to obtain an output strategy including the reasoning task split point and the edge server identifier.

[0021] In one embodiment, allocating computing resources to the target reasoning task according to the output strategy, and calculating payment fees according to the amount of computing resources allocated to the target reasoning task, includes:

[0022] Obtaining the remaining computational amount of the target reasoning task according to the reasoning task splitting point in the output strategy;

[0023] Determine the amount of computing resources allocated to the target reasoning task based on the remaining computing amount of the target reasoning task; the allocated computing resources are part of the available computing resources of the edge server corresponding to the edge server identifier in the output strategy;

[0024] The payment fee of the terminal device is calculated based on the amount of computing resources allocated to the target reasoning task and the preset pricing for the usage of computing resources.

[0025] In one embodiment, determining the amount of computing resources allocated to the target reasoning task based on the remaining computing amount of the target reasoning task includes:

[0026] For the edge server corresponding to the edge server identifier in the output strategy, determine the total remaining computational load of all the inference tasks required for inference computation by the edge server;

[0027] The ratio between the total amount of remaining computation and the amount of remaining computation of the target inference task required for inference computation by the edge server is calculated, and the amount of computing resources allocated to the target inference task is determined.

[0028] In one embodiment, calculating the immediate benefit of each terminal device based on the inference time and the payment fee includes:

[0029] The instant benefit of each terminal device is calculated using the following formula:

[0030]

[0031] in, is a benefit function corresponding to the instant benefit of the terminal device; is the total inference latency of the target inference task of the terminal device; is the payment fee of the terminal device; k is the time.

[0032] In one embodiment, updating the reinforcement learning model according to the empirical data includes:

[0033] At each moment, a sampling data set is obtained by sampling multiple experience data in the experience pool, and the sampling data set is used to update the weight parameters of the value network, the weight parameters of the average policy network, and the weight parameters of the policy network in the reinforcement learning model.

[0034] In a second aspect, an embodiment of the present application further provides a device-edge collaborative inference task offloading device for differentiated computing power pricing, including:

[0035] An acquisition unit, used for acquiring experience data from the experience pool;

[0036] A model updating unit, configured to update the reinforcement learning model according to the empirical data;

[0037] a processing unit configured to construct, for each terminal device, a state vector based on the channel gain of the link between each edge server and the terminal device at the current moment, the payment fee at the previous moment, the inference task split point output at the previous moment, and the edge server identifier output at the previous moment, input the state vector into the updated reinforcement learning model, and obtain a task allocation strategy for the inference task of each terminal device, so that each terminal device offloads the inference task to the edge server for inference calculation according to the task allocation strategy;

[0038] Among them, the empirical data includes the state vector, output strategy, and immediate benefit of each terminal device at each moment; the state vector includes at least payment fee, reasoning task split point and edge server identifier; the payment fee represents the service fee corresponding to the amount of computing resources allocated by the edge server to complete the remaining reasoning tasks of each terminal device; the output strategy includes the reasoning task split point and edge server identifier; the immediate benefit is used to represent the immediate income of each terminal device after unloading the reasoning task.

[0039] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0040] CPU, memory, input and output interfaces;

[0041] The memory is a transient storage memory or a persistent storage memory;

[0042] The central processing unit is configured to communicate with the memory and execute instruction operations in the memory to perform any one of the above-mentioned end-edge collaborative inference task offloading methods for differentiated computing power pricing.

[0043] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program runs on a computer, it executes any one of the above-mentioned methods for offloading end-to-end collaborative inference tasks for differentiated computing power pricing.

[0044] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0045] By incorporating payment fees into the state vector, the reinforcement learning model can effectively optimize the amount of computing resources on the edge server side of the inference task allocation and reduce the service fees that the terminal device needs to pay when completing the task.

[0046] Based on the link's channel gain, the previous payment fee, and the inference task split point, and by linking the edge server's computing resources with the terminal device's payment fee, the reinforcement learning model can make the optimal task offloading decision, minimizing the inference latency and the cost of using edge computing power. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0048] Figure 1 A flowchart of a method for offloading device-edge collaborative inference tasks for differentiated computing power pricing provided in an embodiment of the present application;

[0049] Figure 2 A schematic diagram of reasoning task segmentation and reasoning provided in an embodiment of the present application;

[0050] Figure 3 A schematic diagram of the structure of a device-edge collaborative inference task offloading device for differentiated computing power pricing provided in an embodiment of the present application;

[0051] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0053] With the increasing application of deep learning in various intelligent devices, how to efficiently perform reasoning tasks in resource-constrained environments has become a key issue.

[0054] To solve this problem, the end-edge collaborative inference technology was proposed. This technology divides the deep learning model between the terminal device and the edge server, so that part of the task runs on the terminal device and the other part is completed on the edge server, thereby making full use of the computing resources of both parties and optimizing the entire inference process.

[0055] However, existing methods for optimizing device-edge collaborative inference mostly focus on scenarios involving a single terminal device and a single edge server, failing to adapt to the complex environments of multiple terminals and multiple edge servers. While existing methods for optimizing device-edge collaborative inference based on game theory exist, these methods are primarily suitable for static environments and struggle to cope with network fluctuations and device load changes in dynamic network conditions.

[0056] Furthermore, with the growing number of end-devices, deep learning applications, and edge server deployments, it's becoming increasingly common for multiple end-devices to simultaneously request computing resources. Due to the limited computing power, network bandwidth, and storage resources of edge servers, resource contention can easily occur, leading to degraded inference performance.

[0057] Based on this, the various embodiments of this application propose an end-edge collaborative reasoning distributed optimization method for differentiated computing power pricing, so that each terminal device adopts a distributed reinforcement learning algorithm to decide the splitting points of the deep learning model and the collaborative edge server, thereby weighing the efficiency of each terminal device in performing reasoning tasks and the cost of using edge computing power in a dynamic communication and computing environment.

[0058] The following is a further detailed description of the various embodiments of the present application in conjunction with the accompanying drawings.

[0059] The embodiment of the present application provides a method for offloading edge-to-edge collaborative reasoning tasks for differentiated computing power pricing, such as Figure 1 As shown, the method includes steps S101-S103.

[0060] S101: Acquire experience data in the experience pool.

[0061] Among them, the empirical data includes the state vector, output strategy, and immediate benefit of each terminal device at each moment; the state vector includes at least the payment fee, the inference task split point, and the edge server identifier; the payment fee represents the service fee corresponding to the amount of computing resources allocated by the edge server to complete the remaining inference tasks of each terminal device; the output strategy includes the inference task split point and the edge server identifier; the immediate benefit is used to represent the immediate benefit of each terminal device after offloading the inference task.

[0062] In an embodiment of the present application, empirical data is a data set used by the reinforcement learning model for learning and optimization. The state, action, and benefits generated by the action of each terminal device at different times will be recorded for subsequent learning and decision-making of the reinforcement learning model.

[0063] The state vector is a collection of all possible variables used to describe the current state of the system. In this embodiment of the application, the state vector may include payment fees, reasoning task split points (hereinafter referred to as "split points"), edge server identifiers, and may also include the channel gain of the link between the current terminal device and each edge server.

[0064] The payment refers to the service fee paid by the end device to allow the edge server to complete its remaining inference tasks. The payment is related to the amount of computing resources allocated by the edge server to the end device's inference task. Including the payment in the state vector can help balance the cost when offloading inference tasks to the end device.

[0065] The reasoning task split point refers to the point where the terminal device's reasoning task is divided between the terminal device and the edge server (for details, please refer to Figure 2 The split point is used to execute the inference tasks before the split point on the end device, while the inference tasks after the split point are executed on the edge server. By considering the split point, inference tasks can be distributed between the end device and the edge server, achieving goals such as balancing computing load, reducing latency, and saving energy.

[0066] The edge server ID uniquely identifies the selected edge server, helping the system determine which edge server to assign the current inference task to. Choosing which edge server to complete the remaining inference computations is crucial for optimizing task allocation, as different edge servers may have varying amounts of available computing resources and processing power.

[0067] Instant benefit is used to measure the performance of the strategy / task allocation strategy selected at the current moment. It represents the immediate benefit obtained by each terminal device after offloading the inference task to the edge server. The immediate benefit is usually obtained by combining factors such as inference latency and resource usage cost.

[0068] By storing this data as experience data in the experience pool, we can record the process and results of historical reasoning task assignments, helping the reinforcement learning model make better decisions for future task assignments. For example, the reinforcement learning model will randomly sample data from the experience pool for training, gradually improving the efficiency and performance of reasoning task assignments.

[0069] S102: Update the reinforcement learning model based on the empirical data.

[0070] The reinforcement learning model extracts this empirical data from the experience pool, analyzes which output strategies have yielded high returns in the past and which strategies have resulted in poor returns, and trains based on this empirical data. This allows the reinforcement learning model to select task allocation strategies that deliver higher returns (such as shorter inference latency or lower computational cost) when encountering similar situations in the future. By continuously learning from this empirical data, the system's output task allocation strategies are gradually optimized, improving both the efficiency of inference tasks and the utilization of edge server computing resources.

[0071] S103: For each terminal device, a state vector is constructed based on the channel gain of the link between each edge server and the terminal device at the current moment, the payment fee at the previous moment, the inference task split point output at the previous moment, and the edge server identifier output at the previous moment. The state vector is input into the updated reinforcement learning model to obtain the task allocation strategy for the inference task of each terminal device, so that each terminal device can offload the inference task to the edge server for inference calculation according to the task allocation strategy.

[0072] Channel gain is a parameter that measures the transmission quality of the edge computing network, reflecting the quality of communication between the terminal device and the edge server. Higher channel gain means better communication conditions, which may lead to lower transmission delays. The embodiment of the present application constructs a state vector (including channel gain, payment fee, split point and other information) for each terminal device at each moment, and then inputs the state vector into the reinforcement learning model. The model will output the optimal task allocation strategy, including how to perform the split calculation at which layer of the original reasoning task model and which edge server to select for reasoning calculation. The terminal device will offload the task to the edge server according to the task allocation strategy to achieve efficient reasoning calculation.

[0073] In an embodiment of the present application, by incorporating payment fees into the state vector, the reinforcement learning model can effectively optimize the amount of computing resources on the edge server side of the reasoning task allocation, reducing the service fees that the terminal device needs to pay when completing the task.

[0074] Based on the link's channel gain, the previous payment fee, and the inference task split point, and by linking the edge server's computing resources with the terminal device's payment fee, the reinforcement learning model can make the optimal task offloading decision, minimizing the inference latency and the cost of using edge computing power.

[0075] In one embodiment, experience data in an experience pool is obtained in the following manner: for each terminal device at each moment, a state vector is constructed based on the channel gain of the link between each edge server and the terminal device, the payment fee at the previous moment, the inference task split point output at the previous moment, and the edge server identifier output at the previous moment; the state vector is input into a pre-constructed reinforcement learning model to obtain an output action space; the action space includes the selection probability of each inference task split point and the selection probability of each edge server; based on the selection probability of each inference task split point and the selection probability of each edge server, an output strategy including the inference task split point and the edge server identifier is determined, so that the terminal device offloads the target inference task to the edge server corresponding to the edge server identifier for inference calculation according to the output strategy; computing resources are allocated to the target inference task according to the output strategy, and the payment fee is calculated based on the amount of computing resources allocated to the target inference task; the inference time after the target inference task completes the inference calculation is obtained, and the immediate benefit of each terminal device is calculated based on the inference time and the payment fee; the state vector, output strategy, and immediate benefit at each moment are used as the experience data at each moment and stored in the experience pool.

[0076] For example, in practical applications, taking a single moment k as an example, each terminal device m (1≤m≤M) receives a signal from an edge server, and then uses a channel estimation algorithm (such as minimum mean square error estimation, maximum likelihood estimation, etc.) to estimate the channel gain of each edge server based on parameters such as received signal strength, time delay, and frequency change.

[0077] Record the payment amount at the last moment k-1 The split point selected at the last moment and the edge server selected at the last moment Build Status

[0078] And the status Input to the policy network and obtain the policy probability distribution with output dimension LN Each element in the strategy probability distribution represents the probability of selecting the lth (1≤l≤L) split point and the nth (1≤n≤N) edge server.

[0079] The output strategy is obtained by selecting from the strategy probability distribution. For example, random selection can be performed, that is, any split point and any edge server can be randomly selected as the output strategy, or a split point with a higher probability and an edge server with a higher probability can be selected as the output strategy. By introducing a probability-based selection mechanism, this method can dynamically allocate task split points and edge servers based on the current network environment (such as channel gain) and the characteristics of the inference task. This dynamic allocation strategy makes task offloading more adaptable and helps improve resource utilization efficiency.

[0080] After the terminal device m inputs the target reasoning task to be inferred from the input layer of the reinforcement learning model and obtains the output strategy, it The layer obtains the remaining reasoning tasks and transmits the remaining reasoning tasks to the edge server

[0081]

[0082] Allocate computing resources of the edge server to the target reasoning task according to the output strategy, and calculate the payment fee based on the amount of computing resources allocated to the target reasoning task The payment fee can be calculated based on the amount of computing resources allocated and the pricing amount of the computing resources.

[0083] Get the inference time after the target inference task completes the inference calculation The inference time includes the inference time on the edge server side. Then, the immediate benefit of each terminal device is calculated based on the inference time and payment fee. The state vector, output strategy, and immediate benefit obtained according to the above steps are used as the empirical data at time k

[0084] Perform the above steps at each moment in the same way, so that the state vector at each moment can be obtained, and the strategy and immediate benefit can be output as the experience data at each moment. And the experience data obtained at each moment Store it into the experience pool with capacity D in a first-in-first-out manner , where t is any moment.

[0085] In one implementation, an output strategy including the inference task splitting point and the edge server identifier is determined based on the selection probability of each inference task splitting point and the selection probability of each edge server, including: randomly selecting each inference task splitting point and each edge server based on the selection probability of each inference task splitting point and the selection probability of each edge server, to obtain an output strategy including the inference task splitting point and the edge server identifier.

[0086] After inputting the state vector into the reinforcement learning model, a probability distribution including the selection probability of each inference task split point and the selection probability of each edge server is obtained. Randomly sample the probability distribution to obtain the sampling result x and get the randomly selected split point and edge servers Then we get the output strategy

[0087] By randomly selecting task split points and edge server identifiers, the embodiments of the present application can flexibly respond to network environments with high uncertainty, avoiding system bottlenecks or excessive congestion caused by fixed allocations. This can effectively achieve load balancing between edge servers, preventing individual servers from being overloaded while other server resources remain idle. In high-concurrency scenarios, this helps reduce the overall computational latency of inference tasks, improving resource utilization and the response speed of inference tasks.

[0088] In one embodiment, computing resources are allocated to a target reasoning task according to an output strategy, and a payment fee is calculated based on the amount of computing resources allocated to the target reasoning task, including: obtaining the remaining computing amount of the target reasoning task according to the reasoning task split point in the output strategy; determining the amount of computing resources allocated to the target reasoning task based on the remaining computing amount of the target reasoning task; the allocated computing resources are part of the available computing resources of the edge server corresponding to the edge server identifier in the output strategy; and calculating the payment fee of the terminal device based on the computing resources allocated to the target reasoning task and the preset computing resource usage pricing.

[0089] According to the reasoning task split point in the output strategy Get the remaining computational capacity of the target reasoning task According to the remaining computation amount of the target reasoning task Determine the amount of computing resources allocated for the target reasoning task. For example, if terminal device A needs to perform a complex image recognition task, its remaining computing resources are Terminal device B needs to process sensor data, and its remaining computing power The edge server selected by both terminal device A and terminal device B is edge server S1. Device B is allocated 20% of S1's resources, and device A is allocated 70% of S1's computing resources. It should be understood that the amount of computing resources allocated to each inference task is determined based on the edge server selected in the output strategy. Specifically, the amount of computing resources allocated is a portion of the computing resources currently available on the selected edge server. The available computing resources of an edge server are dynamic and may depend on the number and type of tasks currently being processed by the edge server. The amount of computing resources allocated to the current inference task is a portion of the computing resources currently available on the edge server, not all of the available resources. This means that task allocation will take into account the remaining computing requirements of other inference tasks to ensure that the resource allocation of the edge server is rational and does not cause resource waste or server overload.

[0090] After the inference task is completed, the inference result and the inference time on the edge server are obtained. The edge service provider calculates the amount that the terminal device m needs to pay based on the differentiated pricing standard The function f is a monotonically increasing pricing function. For example, the pricing function can be The function can be expressed as This form, in which Represents the amount of computing resources allocated to edge server n for the inference task issued by terminal device m at time k. Both v1 and v2 are greater than 0. v1 can be used to represent the basic service fee or starting fee, representing the basic costs required by the edge server to provide computing resources (such as server maintenance and basic operating costs). v2 can be used to represent a variable cost coefficient that increases with the task volume or computational workload, used to calculate the proportional increase in the computing resource allocation for the allocated edge server. By setting the value of v2, edge server providers can adjust their charges based on the workload of different computing tasks, achieving more flexible pricing. For example, if v2 is set to a higher value, the payment amount will increase faster with the computational workload, which is suitable for situations with limited computing resources or high costs. If v2 is set to a lower value, the payment amount will increase more slowly, which is suitable for situations with abundant resources or low costs. The edge server provider sends the required payment amount to each terminal device and returns the inference results to the corresponding terminal device after receiving the payment.

[0091] The embodiment of the present application allocates the corresponding computing resources to the reasoning task by the remaining computing capacity and the available computing resources of the edge server, which can ensure that each task obtains the most appropriate amount of computing resources and avoid the problem of resource waste or shortage. This helps to improve the overall resource utilization. At the same time, the differentiated pricing mechanism can dynamically price according to the actual reasoning time of the reasoning task and the computing power of the edge server, and can flexibly respond to the remaining computing capacity requirements of different reasoning tasks. For tasks with higher computing requirements or longer time consumption, higher fees will be charged accordingly, thereby ensuring the fairness and rationality of pricing. Through the above embodiment, the reasoning results can be returned to the terminal device in a timely manner after the calculation is completed, and detailed cost information can be provided at the same time, so that the terminal device can not only quickly obtain the calculation results, but also clearly understand the resource consumption and cost expenditure of each task, thereby increasing the transparency of the edge computing system.

[0092] In one embodiment, the amount of computing resources allocated to the target reasoning task is determined based on the remaining computing amount of the target reasoning task, including: determining the total remaining computing amount of all reasoning tasks required for reasoning calculation by the edge server corresponding to the edge server identifier in the output strategy; calculating the ratio between the total remaining computing amount and the remaining computing amount of the target reasoning task required for reasoning calculation by the edge server, and determining the amount of computing resources allocated to the target reasoning task.

[0093] In actual applications, edge service providers calculate the computing power required by each terminal device on each edge server (i.e., the remaining computing power). The remaining computing capacity is the part of the inference task that the terminal device unloads to the server at a specific moment that still requires inference calculation). The computing resources per unit time are proportionally allocated to the corresponding terminal device to perform the inference calculation of the inference task.

[0094] For edge server n, the calculation formula for the allocation ratio of computing resources per unit time is as follows:

[0095]

[0096] in, The allocation ratio of computing resources per unit time for each inference task; represents the task allocation strategy of the mth (1≤m≤M) terminal device at time k, that is, whether edge server n is selected, is an indicator function, which is 1 if device m selects server n at time k, and 0 otherwise; is the remaining computational effort of the inference task.

[0097] It can be understood that the numerator of the formula represents the remaining computing amount of the inference task of a single terminal device that the edge server n needs to process at time k; the denominator represents the sum of the remaining computing amount of all inference tasks of all terminal devices that the edge server n needs to infer and calculate at time k.

[0098] In the embodiments of the present application, by allocating computing resources proportionally, it is possible to effectively handle situations where multiple terminal devices simultaneously offload their respective reasoning tasks to the same edge server. The computing resources obtained by each terminal device are proportional to its actual task requirements, ensuring balanced load distribution and avoiding situations where terminal devices monopolize resources. This allows the system to flexibly adapt to the task complexity of dynamic networks and improves the concurrent processing capabilities and processing efficiency of the edge computing system. For terminal devices with different computing requirements, it can also reduce waiting time and speed up the feedback speed of reasoning calculations.

[0099] In one embodiment, calculating the immediate benefit of each terminal device based on the inference time and the payment fee includes: calculating the immediate benefit of each terminal device using the following calculation formula:

[0100]

[0101] in, is the benefit function, corresponding to the immediate benefit of the terminal equipment; is the total inference latency of the target inference task of the terminal device; w is the weight parameter of the payment fee; is the payment fee of the terminal device; k is the time.

[0102] Immediate benefits The inference time and payment of fees The negative sign in the formula indicates that an increase in inference time and cost will reduce the immediate benefits of the terminal device. The weight parameter w controls the relative impact of cost on immediate benefits. For example, if w is large, the impact of payment costs on immediate benefits will be more significant. The weight parameter w can flexibly adjust the relative impact of inference time and payment costs on benefits and can be adjusted according to actual application needs. For example, in application scenarios that are very sensitive to latency, a higher weight can be given to inference time; in cost-sensitive scenarios, the weight of cost can be increased. The weight parameter can be set according to actual task requirements and application scenarios and is not limited here.

[0103] The embodiment of the present application provides a quantitative and immediate benefit evaluation method, which enables the system to evaluate the benefit of each terminal device based on a comprehensive consideration of inference delay and cost, and can more intuitively understand the overall benefit of the current task, and then adjust its behavior (such as determining the selection of split points and edge servers).

[0104] The embodiments of the present application enable the system to make adaptive adjustments based on real-time network conditions and task requirements. For example, when the network conditions are good and computing resources are abundant, the system can reduce inference latency, thereby improving efficiency; while in cases where resources are tight or costs are high, the system will prioritize cost factors to ensure maximum efficiency. Through real-time benefit evaluation, terminal devices can achieve lower inference latency or pay less, incentivizing reinforcement learning models to select optimized offloading strategies, and helping edge computing platforms allocate resources more reasonably and reduce unnecessary computing overhead.

[0105] In one embodiment, updating the reinforcement learning model based on the experience data includes: for each moment, sampling a plurality of experience data in the experience pool to obtain a sampling data set, and using the sampling data set to update the weight parameters of the value network, the weight parameters of the average policy network, and the weight parameters of the policy network in the reinforcement learning model.

[0106] At each moment, a batch B consisting of B samples sampled from the experience pool is used to update the value network, average policy network, and the weight parameters of the policy network. For details, please refer to the following formula:

[0107] The weight parameter update formula of the value network is:

[0108]

[0109] in, is the weight parameter of the value network at time k+1 after the update; is the weight parameter of the value network at time k after the update; α is the learning rate; B is the batch size, indicating that B samples are sampled from the experience pool; is the immediate benefit of the i-th sample terminal device m; γ is the discount factor; is the action space; is the state vector corresponding to the i+1th sample, is the state vector corresponding to the i-th sample; This is the predicted Q value at the current moment, indicating that in the current state Next, the terminal device selects an action The Q value that can be obtained; Indicates the status of the terminal device at the next moment Select the action x that brings the maximum reward and return the target Q value of the action, which reflects the cumulative reward in the future. This represents selecting the action x that results in the maximum Q value among all possible actions, a principle underlying reinforcement learning's maximization principle. This is the squared error between the target Q value and the predicted Q value. By minimizing this error, the reinforcement learning model can adjust weight parameters to bring the predicted Q value closer to the actual Q value, thereby optimizing the strategy.

[0110] The weight parameter update formula of the average policy network is:

[0111]

[0112] in, is the weight parameter of the average strategy network at time k+1 after update; is the weight parameter of the average policy network at time k after the update; α is the learning rate; B is the batch size, indicating that B samples are sampled from the experience pool; X is the action space; is the state vector corresponding to the i-th sample; yes The number of historical visits, The more times a state has been accessed in the past, the The greater the impact on the weight parameter update, the more it represents the state Contains more historical experience; Characterizes the current average policy network in state The probability of selecting action x is based on the current weight That is, the probability distribution output by the policy network is used to represent the selection probability of each action in a given state. Characterizing the historical policy network in state The probability of selecting action x, based on the parameters of the current historical policy network Represents the decision-making basis of past policy outputs under the same conditions. By calculating the difference between the probability output by the current policy network and the probability output by historical policies, the system can minimize the difference between the two. This allows the system to gradually optimize the current policy network, enabling it to make better choices in future decisions.

[0113] The weight parameter update formula of the policy network is:

[0114]

[0115] in, is the weight parameter of the policy network at time k+1 after the update; is the weight parameter of the policy network at time k after the update; α is the learning rate; B is the batch size, indicating that B samples are sampled from the experience pool; is the action space; is the state vector corresponding to the i+1th sample, is the state vector corresponding to the i-th sample; It is the predicted Q value at the current moment, indicating the Q value that the terminal device can obtain by selecting action x' in the current state; the exponential function exp() indicates the Q value in the future state. By calculating the expected rewards of possible future actions, the system can predict the rewards that can be obtained after performing a certain action in the next step. is the size of the action space, i.e. the total number of possible actions that can be taken in the system. Represents the number of actions after excluding an optimal action from the action space. By minimizing the square of the difference between the probability output by the current policy network and the probability of the optimal action, the system can continuously adjust the policy network to make better decisions in future task allocations.

[0116] In order to realize the end-edge collaborative reasoning task method for differentiated computing power pricing of the embodiment of the present application, the embodiment of the present application also provides an end-edge collaborative reasoning task offloading device for differentiated computing power pricing, such as Figure 3 As shown, the device includes:

[0117] An acquisition unit 301 is used to acquire experience data from an experience pool;

[0118] A model updating unit 302 is configured to update the reinforcement learning model according to the empirical data;

[0119] Processing unit 303 is configured to construct a state vector for each terminal device based on the channel gain of the link between each edge server and the terminal device at the current moment, the payment fee at the previous moment, the inference task split point output at the previous moment, and the edge server identifier output at the previous moment, input the state vector into the updated reinforcement learning model, and obtain a task allocation strategy for the inference task of each terminal device, so that each terminal device offloads the inference task to the edge server for inference calculation according to the task allocation strategy;

[0120] Among them, the empirical data includes the state vector, output strategy, and immediate benefit of each terminal device at each moment; the state vector includes at least payment fee, reasoning task split point and edge server identifier; the payment fee represents the service fee corresponding to the amount of computing resources allocated by the edge server to complete the remaining reasoning tasks of each terminal device; the output strategy includes the reasoning task split point and edge server identifier; the immediate benefit is used to represent the immediate income of each terminal device after unloading the reasoning task.

[0121] In one embodiment, the processing unit 303 is further configured to construct, for each terminal device at each moment, a state vector based on the channel gain of the link between each edge server and the terminal device, the payment fee at the previous moment, the inference task split point output at the previous moment, and the edge server identifier output at the previous moment;

[0122] The processing unit 303 is further configured to input the state vector into a pre-built reinforcement learning model to obtain an output action space; the action space includes a selection probability of each inference task split point and a selection probability of each edge server;

[0123] a determination unit, configured to determine, based on a selection probability of each inference task split point and a selection probability of each edge server, an output strategy including an inference task split point and an edge server identifier, so that the terminal device offloads the target inference task to an edge server corresponding to the edge server identifier for inference calculation according to the output strategy;

[0124] The processing unit 303 is further configured to allocate computing resources to the target reasoning task according to the output strategy, and calculate payment fees according to the amount of computing resources allocated to the target reasoning task;

[0125] The processing unit 303 is further configured to obtain the inference time after the target inference task completes the inference calculation, and calculate the immediate benefit of each terminal device based on the inference time and the payment fee;

[0126] The processing unit 303 is further configured to store the state vector, the output strategy and the immediate benefit at each moment as experience data at each moment into an experience pool.

[0127] In one embodiment, the processing unit 303 is further used to randomly select each reasoning task splitting point and each edge server according to the selection probability of each reasoning task splitting point and the selection probability of each edge server, and obtain an output strategy including the reasoning task splitting point and the edge server identifier.

[0128] In one embodiment, the processing unit 303 is further configured to obtain the remaining computational load of the target reasoning task according to the reasoning task splitting point in the output strategy;

[0129] The processing unit 303 is further configured to determine the amount of computing resources allocated to the target reasoning task based on the remaining computing amount of the target reasoning task; the allocated computing resources are part of the available computing resources of the edge server corresponding to the edge server identifier in the output strategy;

[0130] The processing unit 303 is further configured to calculate the payment fee of the terminal device based on the amount of computing resources allocated to the target reasoning task and the preset pricing for the amount of computing resources used.

[0131] In one embodiment, the determining unit is further configured to determine, for an edge server corresponding to the edge server identifier in the output strategy, a sum of remaining computational loads of all the inference tasks required for inference computation by the edge server;

[0132] The processing unit 303 is further configured to calculate a ratio between the total amount of remaining computation and the amount of remaining computation of the target inference task required for inference computation by the edge server, and determine an amount of computing resources allocated to the target inference task.

[0133] In one embodiment, the instantaneous benefit of each terminal device is calculated using the following formula:

[0134]

[0135] in, is a benefit function corresponding to the instant benefit of the terminal device; is the total inference latency of the target inference task of the terminal device; is the payment fee of the terminal device; k is the time.

[0136] In one embodiment, the model updating unit 302 is further used to obtain a sampling data set from multiple experience data in the experience pool at each moment, and use the sampling data set to update the weight parameters of the value network, the weight parameters of the average policy network, and the weight parameters of the policy network in the reinforcement learning model.

[0137] In actual application, the processing unit 303 can be implemented by a processor in an electronic device combined with a communication interface, and the determination unit, acquisition unit 301, and model update unit 302 can be implemented by the communication interface in an end-edge collaborative inference task offloading device for differentiated computing power pricing.

[0138] It should be noted that: the above embodiment provides an end-to-end collaborative reasoning task unloading device for differentiated computing power pricing. When performing end-to-end collaborative reasoning task unloading for differentiated computing power pricing, only the division of the above-mentioned program modules is used as an example. In actual applications, the above-mentioned processing can be assigned to different program modules as needed, that is, the internal structure of the device is divided into different program modules to complete all or part of the processing described above. In addition, the end-to-end collaborative reasoning task unloading device for differentiated computing power pricing provided by the above embodiment and the end-to-end collaborative reasoning task unloading method embodiment for differentiated computing power pricing belong to the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0139] Based on the hardware implementation of the above program modules, and in order to implement a method for offloading edge collaborative reasoning tasks for differentiated computing power pricing provided in the embodiment of the present application, the embodiment of the present application also provides an electronic device, such as Figure 4 As shown, the electronic device 400 includes:

[0140] CPU 401, memory 402 and input / output interface 403;

[0141] The memory 402 is a temporary storage memory or a permanent storage memory;

[0142] The central processing unit 401 is configured to communicate with the memory 402 and execute instruction operations in the memory 402 to perform any of the above-mentioned end-edge collaborative inference task offloading methods for differentiated computing power pricing.

[0143] Of course, in actual application, the various components in the electronic device 400 are coupled together through the bus system 405. It can be understood that the bus system 405 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 405 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, Figure 3 Various buses are labeled as bus system 405 .

[0144] The memory 402 in the embodiment of the present application is used to store various types of data to support the operation of the electronic device 400. Examples of such data include: any computer program used to operate on the electronic device 400.

[0145] It is understandable that when the processor in the electronic device described above executes the computer program, it can also implement the functions of the various units in the corresponding device embodiments described above, which will not be repeated here. For example, the computer program can be divided into one or more modules / units, and one or more modules / units are stored in the memory and executed by the processor to complete the various embodiments of the present application. One or more modules / units can be a series of computer program instruction segments that can perform specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device. For example, the computer program can be divided into the various units in the above-mentioned electronic device, and each unit can implement the specific functions described in the above-mentioned corresponding electronic device.

[0146] Electronic devices may be computing devices such as desktop computers, laptops, PDAs, and cloud servers. Electronic devices may include, but are not limited to, processors and memory. Those skilled in the art will appreciate that processors and memory are merely examples of electronic devices and do not limit the scope of electronic devices. Electronic devices may include more or fewer components, or combinations of certain components, or different components. For example, electronic devices may also include input / output devices, network access devices, buses, and the like.

[0147] The processor can be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the electronic device and connects all parts of the electronic device using various interfaces and lines.

[0148] The memory can be used to store computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory, and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area. The program storage area can store an operating system, at least one application required for a function, etc.; the data storage area can store data created based on the use of the terminal, etc. In addition, the memory can include high-speed random access memory and non-volatile memory, such as a hard disk, internal memory, a plug-in hard disk, a smart memory card (SMC, Smart Media Card), a secure digital (SD, Secure Digital) card, a flash card (FlashCard), at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0149] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program runs on a computer, the computer is enabled to execute any one of the above-mentioned methods for offloading end-edge collaborative inference tasks for differentiated computing power pricing.

[0150] An embodiment of the present application also provides a computer program product having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, it is used to implement the end-edge collaborative reasoning task offloading method for differentiated computing power pricing described in the first aspect of the embodiment of the present application or any specific implementation method of the first aspect.

[0151] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0152] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0153] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0154] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0155] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A method for offloading edge-to-end collaborative inference tasks for differentiated computing power pricing, characterized by: include: Get experience data from the experience pool; updating the reinforcement learning model based on the empirical data; For each terminal device, a state vector is constructed based on the channel gain of the link between each edge server and the terminal device at the current moment, the payment fee at the previous moment, the reasoning task split point output at the previous moment, and the edge server identifier output at the previous moment. The state vector is input into the updated reinforcement learning model to obtain a task allocation strategy for the reasoning task of each terminal device, so that each terminal device offloads the reasoning task to the edge server for reasoning calculation according to the task allocation strategy; The empirical data includes the state vector, output strategy, and immediate benefit of each terminal device at each moment; The state vector includes at least a payment fee, an inference task split point, and an edge server identifier; the payment fee represents the service fee corresponding to the amount of computing resources allocated by the edge server to complete the remaining inference tasks of each terminal device; the output strategy includes the inference task split point and the edge server identifier; the immediate benefit is used to represent the immediate benefit of each terminal device after unloading the inference task.

2. The method for offloading edge-to-end collaborative inference tasks for differentiated computing power pricing according to claim 1, characterized in that: The experience data in the experience pool is obtained in the following way: For each terminal device at each moment, construct a state vector based on the channel gain of the link between each edge server and the terminal device, the payment fee at the previous moment, the inference task split point output at the previous moment, and the edge server identifier output at the previous moment; Input the state vector into a pre-built reinforcement learning model to obtain an output action space; the action space includes the selection probability of each inference task split point and the selection probability of each edge server; Determining, based on the selection probability of each inference task split point and the selection probability of each edge server, an output strategy including the inference task split point and the edge server identifier, so that the terminal device offloads the target inference task to the edge server corresponding to the edge server identifier for inference calculation according to the output strategy; Allocating computing resources to the target reasoning task according to the output strategy, and calculating payment fees according to the amount of computing resources allocated to the target reasoning task; Obtaining the inference time after the target inference task completes the inference calculation, and calculating the immediate benefit of each of the terminal devices based on the inference time and the payment fee; The state vector, the output strategy and the immediate benefit at each moment are used as the experience data at each moment and stored in the experience pool.

3. The method for offloading edge-to-end collaborative inference tasks for differentiated computing power pricing according to claim 2 is characterized in that: The step of determining an output strategy including the reasoning task split point and the edge server identifier based on the selection probability of each reasoning task split point and the selection probability of each edge server includes: According to the selection probability of each reasoning task split point and the selection probability of each edge server, each reasoning task split point and each edge server are randomly selected respectively to obtain an output strategy including the reasoning task split point and the edge server identifier.

4. The method for offloading edge-to-edge collaborative inference tasks for differentiated computing power pricing according to claim 2 is characterized in that: Allocating computing resources to the target reasoning task according to the output strategy, and calculating payment fees according to the amount of computing resources allocated to the target reasoning task, includes: Obtaining the remaining computational amount of the target reasoning task according to the reasoning task splitting point in the output strategy; Determine the amount of computing resources allocated to the target reasoning task based on the remaining computing amount of the target reasoning task; the allocated computing resources are part of the available computing resources of the edge server corresponding to the edge server identifier in the output strategy; The payment fee of the terminal device is calculated based on the amount of computing resources allocated to the target reasoning task and the preset pricing for the usage of computing resources.

5. The method for offloading device-edge collaborative inference tasks for differentiated computing power pricing according to claim 4 is characterized in that: The determining, based on the remaining computational amount of the target reasoning task, the amount of computing resources allocated to the target reasoning task, includes: For the edge server corresponding to the edge server identifier in the output strategy, determine the total remaining computational load of all the inference tasks required for inference computation by the edge server; The ratio between the total amount of remaining computation and the amount of remaining computation of the target inference task required for inference computation by the edge server is calculated, and the amount of computing resources allocated to the target inference task is determined.

6. The method for offloading device-edge collaborative inference tasks for differentiated computing power pricing according to claim 2 is characterized in that: Calculating the immediate benefit of each terminal device according to the inference time and the payment fee includes: The instant benefit of each terminal device is calculated using the following formula: in, is a benefit function corresponding to the instant benefit of the terminal device; is the total inference latency of the target inference task of the terminal device; is the payment fee of the terminal device; k is the time.

7. The method for offloading edge-to-end collaborative inference tasks for differentiated computing power pricing according to claim 1, characterized in that: Updating the reinforcement learning model according to the empirical data includes: At each moment, a sampling data set is obtained by sampling multiple experience data in the experience pool, and the sampling data set is used to update the weight parameters of the value network, the weight parameters of the average policy network, and the weight parameters of the policy network in the reinforcement learning model.

8. A device-edge collaborative inference task offloading device for differentiated computing power pricing, characterized by: include: An acquisition unit, used for acquiring experience data from the experience pool; A model updating unit, configured to update the reinforcement learning model according to the empirical data; a processing unit configured to construct, for each terminal device, a state vector based on the channel gain of the link between each edge server and the terminal device at the current moment, the payment fee at the previous moment, the inference task split point output at the previous moment, and the edge server identifier output at the previous moment, input the state vector into the updated reinforcement learning model, and obtain a task allocation strategy for the inference task of each terminal device, so that each terminal device offloads the inference task to the edge server for inference calculation according to the task allocation strategy; The empirical data includes the state vector, output strategy, and immediate benefit of each terminal device at each moment; The state vector includes at least a payment fee, an inference task split point, and an edge server identifier; the payment fee represents the service fee corresponding to the amount of computing resources allocated by the edge server to complete the remaining inference tasks of each terminal device; the output strategy includes the inference task split point and the edge server identifier; the immediate benefit is used to represent the immediate benefit of each terminal device after unloading the inference task.

9. An electronic device, characterized in that: include: CPU, memory and input / output interfaces; The memory is a transient storage memory or a persistent storage memory; The central processing unit is configured to communicate with the memory and execute instruction operations in the memory to perform the end-edge collaborative inference task offloading method for differentiated computing power pricing as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program runs on a computer, the end-edge collaborative inference task offloading method for differentiated computing power pricing as described in any one of claims 1 to 7 is executed.

Images