Contribution-aware in-network pooled resource allocation optimization method in computing power network

Abstract

The present invention discloses a method for optimizing the allocation of in-network pooled resources based on contribution perception in a computing power network, comprising the following steps: constructing a computing power network resource allocation system including an infrastructure layer, a resource pool layer, and a CPN scheduler layer; constructing the computing power queue length of each resource pool in the resource pool layer based on the dynamics of tasks and mobile devices; using a deep reinforcement learning algorithm in the CPN scheduler layer to respectively establish a computing power model and a cache model, and constructing the models as Markov processes respectively; constructing a utility function of the computing power network resource allocation system based on the computing power model and the cache model; constructing a proximal policy optimization algorithm based on an attention mechanism using a proximal policy optimization algorithm and an attention mechanism; solving the long-term utility function using a proximal policy optimization algorithm based on an attention mechanism with the goal of maximizing the long-term utility function. The present invention achieves higher system utility and can significantly improve the performance of integrated computing tasks and caches.

Description

Technical Field

[0001] The present invention belongs to the technical field of communication networks, and specifically relates to a contribution-aware in-network pooled resource allocation optimization method in a computing power network. Background Art

[0002] The development of B5G / 6G technology will support a variety of large-scale data-intensive applications in the next decade, such as virtual and augmented reality, brain communication interfaces, remote surgery, etc. At the same time, it will generate zettabytes (ZB) of digital information and require more computing power to support it. The development of 6G not only benefits from the technology brought by 5G, but also requires more new technologies. Computing Power Network (CPN) provides an effective solution that enables deep integration of computing power and networks. CPN accelerates the diffusion of computing power from a group of data centers to multiple network edges and end users, and gives the infrastructure computing and caching capabilities.

[0003] Due to the current explosion of large amounts of data and resource-constrained applications, such as face recognition and intelligent manufacturing, CPN can use computing and storage resources on terminals, edges, and clouds to solve various diverse demand problems. In addition, in order to solve the tension of resource utilization in the system and improve resource utilization, CPN unifies the computing power in the end-edge cloud into one dimension to build a decentralized resource pool. However, how to build a resource pool in combination with scenarios and how to allocate resources between resource pools are still problems to be solved in CPN. In particular, with the mobility of users in B5G / 6G, the optimization problem has become more complicated. At the same time, the existing resource allocation scheme cannot meet the requirements of environmentally sensitive interactions and various needs between intelligent agents, that is, it has poor adaptability to complex environments. In addition, there are still a large number of idle computing and cache resources that are not fully utilized, and the contribution of the intelligent agents involved in learning is not evaluated, which reduces the system performance efficiency. Summary of the invention

[0004] In view of the problem that it is difficult to allocate resources between resource pools and the resource allocation efficiency is low in the prior art, the present invention proposes a contribution-aware in-network pooled resource allocation optimization method in a computing network, designs a dynamic resource pool in CPN, and puts limited resources on the resource pool that makes a greater contribution to the system based on the attention mechanism, thereby allocating resources and maximizing the long-term utility of the system. To solve the above technical problems, the technical solutions adopted by the present invention are as follows:

[0005] A contribution-aware in-network pooled resource allocation optimization method in a computing network comprises the following steps:

[0006] S1, constructing a computing network resource allocation system including an infrastructure layer, a resource pool layer, and a CPN scheduler layer, wherein the resource pool layer includes several resource pools, each of which is composed of resources provided by the infrastructure layer;

[0007] S2, based on the dynamic characteristics of tasks and mobile devices in the infrastructure layer, builds the computing queue length of each resource pool in the resource pool layer;

[0008] S3, in the CPN scheduler layer, a computing power model and a cache model are established using a deep reinforcement learning algorithm, and the computing power model and the cache model are constructed as Markov processes respectively;

[0009] S4, constructing a utility function of a computing power network resource allocation system based on the computing power model and cache model established in step S3;

[0010] S5, using the proximal policy optimization algorithm and the attention mechanism to construct a proximal policy optimization algorithm based on the attention mechanism;

[0011] S6, with the goal of maximizing the long-term utility function, the long-term utility function is solved using the proximal strategy optimization algorithm based on the attention mechanism established in step S5.

[0012] In step S2, the calculation formula of the computing power queue length of the resource pool is:

[0013] P m (t) = P m (t-1)-R m (t-1)+p m (t-1);

[0014] Where P m (t-1) represents the computing queue length of resource pool m at time slot t-1, R m (t-1) represents the computing power consumed by resource pool m at time slot t-1, p m (t-1) represents the computing power collected by resource pool m at time slot t-1, P m (t) represents the computing queue length of resource pool m at time slot t;

[0015] The computing power R consumed by resource pool m at the t-1 time slot m The calculation formula for (t-1) is:

[0016]

[0017] In the formula, It represents the computing power consumed by resource pool m to process tasks at time slot t-1. It represents the computing power lost due to the mobile device leaving resource pool m at time slot t-1.

[0018] The computing power consumed by resource pool m to process tasks during the t-1 time slot The calculation formula is:

[0019]

[0020] In the formula, represents the set of all tasks at time slot t-1, φ k (t-1) represents the CPU cycles required to process the kth task at the t-1 time slot, Indicates whether the kth task is processed in resource pool m at time slot t-1;

[0021] The computing power lost due to the mobile device leaving resource pool m during the t-1 time slot The calculation formula is:

[0022]

[0023] In the formula, p j→m (t-1) represents the probability of mobile device n moving from resource pool j to resource pool m at time slot t-1, h n It represents the idle computing capacity contributed by mobile device n to the resource pool.

[0024] The computing power p collected by resource pool m at time slot t-1 m The calculation formula for (t-1) is:

[0025] p m (t-1) = min(Λ-P m (t-1), η m (t-1));

[0026] Where P m (t-1) represents the computing power queue length of resource pool m at time slot t-1, that is, the available computing power of resource pool m, Λ represents the upper limit of the computing power collected by resource pool m at time slot t-1, η m (t-1) represents the computing power obtained when the tasks in resource pool m are completed and mobile devices enter at time slot t-1;

[0027] The computing capacity η obtained when the task execution in the resource pool m is completed and the mobile device enters the t-1 time slot m The calculation formula for (t-1) is:

[0028]

[0029] In the formula, It represents the computing power obtained by completing the task in resource pool m at time slot t-1. It represents the computing power obtained by the mobile device entering the resource pool m at the time slot t-1.

[0030] In step S3, the expression of the computing power ratio transfer probability matrix of the computing power model is:

[0031]

[0032] In the formula, represents the computing power ratio transfer probability matrix of resource pool m facing the kth task at time slot t, L represents the number of rows and columns of the matrix, Indicates the probability of computing power weight state transition;

[0033] The cache state probability matrix of the cache model is:

[0034]

[0035] In the formula, Γ c (t) represents the cache status probability matrix facing the cth request data at time slot t, Represents the cache state transition probability.

[0036] In step S4, the utility function of the computing power network resource allocation system is expressed as:

[0037] U(t)=U p (t)+U e (t);

[0038] Where U(t) represents the utility function at time slot t, U p (t) represents the utility brought by executing the task at time slot t, U e (t) represents the utility brought by caching request data at time slot t;

[0039] The utility U brought by executing the task at the time slot t p The calculation formula of (t) is:

[0040]

[0041] In the formula, represents the utility of the kth task being processed in resource pool m at time slot t, Indicates whether the kth task is processed in resource pool m at time slot t. P represents the effective computing power weight that resource pool m can provide when facing the kth task at time slot t. m (t) represents the computing queue length of resource pool m at time slot t, O k (t) represents the data size of the kth task at time slot t, φ k(t) represents the CPU cycles required to process the kth task at time slot t, and w k (t) represents the income of processing the kth task at time slot t, e m represents the energy consumption of resource pool m executing one CPU cycle, c m represents the price per joule of hardware energy consumption caused by executing a task, K represents the total number of tasks, and M represents the total number of mobile devices;

[0042] The utility U brought by caching the requested data during the time slot t e The calculation formula of (t) is:

[0043]

[0044] Where C represents the total number of requested data at time slot t, represents the utility of resource pool m caching the cth request data at time slot t, Indicates whether the cth request data is cached in resource pool m at time slot t. represents the return cost, represents the spectrum bandwidth orthogonally allocated from the base stations in resource pool m to the users covered by resource pool m, represents the spectrum efficiency of mobile devices connecting to base stations in resource pool m, Indicates whether the cth request data is in resource pool m at time slot t. Indicates the size of the request data to be cached. Indicates the cost of the space occupied by resource pool m to cache request data.

[0045] The step S5 comprises the following steps:

[0046] S5.1, using the proximal strategy optimization algorithm to establish a neural network model, the neural network model includes an Actor neural network and a Critic neural network, the Actor neural network includes a new-actor network and an old-actor network, the Critic neural network includes an attention layer built based on an attention mechanism, and the attention layer is used to perceive the contribution of each resource pool to the system utility;

[0047] S5.2, Strategy for initializing new-actor network and the old-actor network strategy Set the initial value of mobile devices n = 0, the initial value of time slots t = 0, set the total number of time slots T and the total number of mobile devices N;

[0048] S5.3, the initial state S of the receiving system t ;

[0049] S5.4, The new-actor network selects an action a according to the current policy and the state S t ; t ;

[0050] S5.5, According to the selected action a t obtain the observed reward r t and the next state S t+1 ;

[0051] S5.6, Store {s t , a t , r t} in the buffer;

[0052] S5.7, Calculate the importance weight r t (θ) of the new-actor network and the old-actor network;

[0053] S5.8, Judge whether t < T. If so, execute t = t + 1 and return to step S5.4. Otherwise, execute step S5.9;

[0054] S5.9, Evaluate the quality of the selected action a t in the state S t ;

[0055] S5.10, Assign the policy of the new-actor network to the policy of the old-actor network;

[0056] S5.11, Update the network parameters of the Critic neural network according to the mean square function;

[0057] S5.12, Update the network parameters of the new-actor network according to the loss function;

[0058] S5.13, Judge whether n < N. If so, execute n = n + 1 and return to step S5.3. Otherwise, the training ends.

[0059] The state space of the state S t is represented by χ(t), and its expression is:

[0060]

[0061]

[0062] where represents the effective computing power weight that the resource pool m can allocate to the Kth task at time slot t, represents whether the Cth request data is in the resource pool m at time slot t;

[0063] The action at The action space of is represented by a(t), and its expression is:

[0064]

[0065] In the formula, Indicates whether resource pool m executes the Kth task at time slot t, i.e., the task strategy. Indicates whether resource pool m executes the Cth request data, that is, the cache strategy, at time slot t.

[0066] In step S5.7, the importance weight r t The calculation formula of (θ) is:

[0067]

[0068] In the formula, π θ (a t ,s t ) represents the strategy of the new-actor network, Represents the strategy of the old-actor network.

[0069] In step S6, the expression for maximizing the long-term utility function is:

[0070]

[0071] Where U(t) represents the utility function at time slot t, T represents the total number of time slots, Indicates whether the cth request data is cached in resource pool m at time slot t. Indicates whether the kth task is processed in resource pool m at time slot t;

[0072] The constraints of the long-term utility function are:

[0073] P m (t)≤Λ, m∈M; (a)

[0074]

[0075]

[0076]

[0077] p m (t) = min(Λ-P m (t), η m (t)); (e)

[0078]

[0079] Where Pm (t) represents the computing queue length of resource pool m at time slot t, p m (t) represents the computing power collected by resource pool m at time slot t, Λ represents the upper limit of the computing power collected by resource pool m at time slot t, and η m (t) represents the computing power obtained by completing the task execution in resource pool m and user mobility at time slot t, represents the set of all task indices at time slot t, φ k (t) represents the CPU cycles required to process the task corresponding to task index k at time slot t, M represents the total number of mobile devices, Represents a collection of resource pools.

[0080] Beneficial effects of the present invention:

[0081] The present invention fully utilizes the idle resources in the cloud-edge-end hierarchical architecture by constructing a dynamic resource pool in the CPN scheduler layer, achieves higher resource utilization, uses deep reinforcement learning to perform computing tasks and cache decisions between different resource pools, and maximizes long-term system utility under complex and dynamic environmental conditions. The new proximal policy optimization algorithm based on the attention mechanism can process data with high spatial and computational complexity, give full play to the advantages of resource pools to perceive the contribution of different resource pools to the system, and significantly improve the performance of integrated computing tasks and caches. BRIEF DESCRIPTION OF THE DRAWINGS

[0082] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0083] Figure 1 It is a system architecture diagram of the present invention.

[0084] Figure 2 Flowchart for task execution and request data cache.

[0085] Figure 3 Schematic diagram of the actor-critic neural network based on the attention mechanism.

[0086] Figure 4 It is a schematic diagram of the algorithm flow of the present invention.

[0087] Figure 5 Schematic diagram of rewards obtained by the present invention and other algorithms under different batch size values.

[0088] Figure 6Schematic diagram of rewards obtained by the present invention and other algorithms at different learning rates.

[0089] Figure 7 This is a schematic diagram comparing the system effectiveness of the present invention and other algorithms.

[0090] Figure 8 The figure is a schematic diagram for comparing the computational utility of the present invention with other algorithms.

[0091] Fig. 9 The schematic diagram is a comparison of the cache effectiveness of the present invention and other algorithms.

[0092] Fig.10 Schematic diagram of the impact of the number of tasks on system utility under different algorithms.

[0093] Fig.11 Schematic diagram of the impact of task price on system utility under different algorithms.

[0094] Fig.12 Schematic diagram of the impact of the price of connecting to the cache server on system utility under different algorithms. DETAILED DESCRIPTION

[0095] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0096] A contribution-aware in-network pooled resource allocation optimization method in a computing network comprises the following steps:

[0097] S1, constructing a computing network resource allocation system including an infrastructure layer, a resource pool layer, and a CPN scheduler layer, wherein the resource pool layer includes several resource pools, each of which is composed of resources provided by the infrastructure layer;

[0098] like Figure 1 As shown, the computing network resource allocation system includes an infrastructure layer, a resource pool layer, a CPN scheduler layer, and an application service layer. The application service layer is used to display the AI ​​applications that the system can process. The infrastructure layer includes mobile devices, base stations, and cloud servers. The resource pool layer is an independent resource pool set formed by concentrating the available resources of the devices in the infrastructure layer, that is, a resource set established by the base station center with idle resources of mobile devices, base stations, and cloud servers. The CPN scheduler layer can obtain user needs and optimize resource allocation based on needs and built-in algorithms.

[0099] The resource pool set adopts express, Each resource pool includes a base station. The computing resources and cache resources provided by the base station, all mobile devices in the area covered by the base station, and the cloud services in the area where the base station is located constitute a resource pool in the area. Each resource pool can provide computing and cache resources according to its own situation. Considering the mobility of computing tasks and mobile devices, the computing power of each resource pool has dynamic characteristics. Specifically, as mobile devices enter and exit the resource pool, the computing power of the corresponding resource pool will increase or decrease. Similarly, the execution and completion of tasks will also lead to changes in computing power. The computing power queue length set of all resource pools adopts express, Among them, P m (t) represents the computing power queue length of resource pool m at time slot t, that is, the available computing power of resource pool m. The computing power queue length P m (t) is constructed based on the computation and completion of tasks and the mobility of the mobile devices held by the users. The mobile device collection is constructed using express, Mobile devices communicate with the base station via wireless links. The system operates on unit-sized time slots, which are set using express,

[0100] like Figure 2 As shown in the figure, the system can be applied in the following case, where users put meaningful datasets in the Multi-Object Tracking Benchmark (MOT) dataset on the cloud server in resource pool 4, so that all users can do video processing services in the future. The MOT dataset focuses on pedestrian tracking. When users want to do video tasks, they will retrieve part of the data from the cloud server for analysis. Specifically, the user task request in resource pool 1 is first sent to the CPN scheduler. According to the user's needs and its own algorithm, the scheduler obtains the final resource allocation strategy, which includes which resource pool provides computing power to calculate the task and which resource pool caches the data. If the scheduler decides to calculate the task in resource pool 2, it will send a message to the base station in resource pool 2. After receiving the message, the base station prepares the corresponding computing resources for task calculation. If the scheduler decides that resource pool 3 caches the data, resource pool 3 will first check whether the neighbor resource pool has the data. If so, it will directly cache the corresponding data of the neighbor resource pool. If not, it will download it from the cloud server in resource pool 4.

[0101] S2, based on the dynamic characteristics of tasks and mobile devices in the infrastructure layer, builds the computing queue length of each resource pool in the resource pool layer;

[0102] The expression of the computing power queue length of the resource pool is:

[0103] P m (t) = P m (t-1)-R m (t-1)+p m (t-1);

[0104] Where P m (t-1) represents the computing queue length of resource pool m at time slot t-1, R m (t-1) represents the computing power consumed by resource pool m at time slot t-1, p m (t-1) represents the computing power collected by resource pool m at time slot t-1, P m (t) represents the computing queue length of resource pool m at time slot t.

[0105] The computing power R consumed by resource pool m at the t-1 time slot m The calculation formula for (t-1) is:

[0106]

[0107] In the formula, It represents the computing power consumed by resource pool m to process tasks at time slot t-1. It represents the computing power lost due to the mobile device leaving resource pool m at time slot t-1.

[0108] The computing power consumed by resource pool m to process tasks during the t-1 time slot The calculation formula is:

[0109]

[0110] In the formula, represents the set of all tasks at time slot t-1, φ k (t-1) represents the CPU cycles required to process the kth task at the t-1 time slot, Indicates whether the kth task is processed in resource pool m at time slot t-1.

[0111] The computing power lost due to the mobile device leaving resource pool m during the t-1 time slot The calculation formula is:

[0112]

[0113] In the formula, p j→m (t-1) represents the probability of mobile device n moving from resource pool j to resource pool m at time slot t-1, h n It represents the idle computing capacity contributed by mobile device n to the resource pool.

[0114] The computing power p collected by resource pool m at time slot t-1 m The calculation formula for (t-1) is:

[0115] p m (t-1) = min(Λ-P m (t-1), η m (t-1));

[0116] Where P m (t-1) represents the computing power queue length of resource pool m at time slot t-1, that is, the available computing power of resource pool m, Λ represents the upper limit of the computing power collected by resource pool m at time slot t-1, η m (t-1) represents the computing power obtained when the tasks in resource pool m are completed and the mobile device enters at time slot t-1.

[0117] The computing capacity η obtained when the tasks in the resource pool m are completed and the mobile device enters the t-1 time slot m The calculation formula for (t-1) is:

[0118]

[0119] In the formula, It represents the computing power obtained by completing the task in resource pool m at time slot t-1. Indicates the computing power obtained by the mobile device entering resource pool m at time slot t-1.

[0120] When the task is executed at t-2, this part of the computing power will be released at t-1, and the resource pool will reuse this part of the computing power. The computing power obtained by completing the task execution in the resource pool m at the t-1 time slot The calculation formula is:

[0121]

[0122] In the formula, Indicates whether the kth task is executed in resource pool m at time slot t-2. Represents the set of all tasks at time slot t-2.

[0123] The computing power obtained by the mobile device entering the resource pool m at the t-1 time slot The calculation formula is:

[0124]

[0125] In the formula, h n represents the idle computing capacity contributed by mobile device n to the resource pool, p m→i(t) represents the probability that a mobile device switches from resource pool m to resource pool i at time slot t.

[0126] S3, in the CPN scheduler layer, the deep reinforcement learning (DRL) algorithm is used to establish the computing power model and the cache model respectively, and the computing power model and the cache model are constructed as Markov processes respectively;

[0127] use Indicates the computing power weight allocated by resource pool m to the kth task at time slot t, and its value is 0-1. can be simulated as a random variable and divided into discrete levels δ = {δ 0， δ1,…,δ G-1}, G is the number of discrete levels, The discrete level is It indicates that its transfer over time can be represented by a Markov chain, that is, x k ,y k ∈δ, therefore, the computing power ratio transfer probability matrix The expression is:

[0128]

[0129] In the formula, L represents the number of rows and columns of the matrix, represents the probability of computing power transfer, where represents the effective computing power weight allocated to the kth task by resource pool m at time slot t+1,

[0130] Assume that the average request arrival rate of C request data follows the Poisson distribution λ E , the probability of requesting data follows a Zipf-like distribution, β is the Zipf slope, and the average request rate of the cth request data at time slot t for:

[0131]

[0132] In the formula, c β Indicates the probability that the cth request data is selected.

[0133] Whether the cth request data is cached in the resource pool can be regarded as a random variable ξ c , construct a two-state Markov chain, whose state space is Then the transition probability is in, ξ c(t+1) indicates whether the c-th request data is cached in the resource pool at time slot t+1, and the cache status probability matrix Γ of the c-th request data at time slot t c (t) is:

[0134]

[0135] Assuming that the mobile devices in this application have different computing capabilities, in order to simplify the spectrum utilization between the mobile devices and the base station, a virtual node u0(t) is introduced to represent the user status in the resource pool at time slot t. Specifically, the mobile devices have location coordinates and different computing capability levels, that is, each node in the resource pool has a different weight. Using the centroid node method, the node, that is, each mobile device, is used as a vertex, and all nodes are connected to form a polygon. The polygon is divided into multiple triangles. The centroid of each triangle is obtained to obtain the centroid of the polygon, that is, the virtual node u0(t).

[0136] S4, constructing a utility function of a computing power network resource allocation system based on the computing power model and cache model established in step S3;

[0137] The calculation formula of the utility function is:

[0138] R(t)=U(t)=U p (t)+U e (t);

[0139] In the formula, R(t) represents the reward function at time slot t, U(t) represents the utility function at time slot t, and U p (t) represents the utility brought by executing the task at time slot t, U e (t) represents the utility of caching request data at time slot t.

[0140] The utility U brought by executing the task at the time slot t p The calculation formula of (t) is:

[0141]

[0142] In the formula, represents the utility of the kth task being processed in resource pool m at time slot t, Indicates whether the kth task is processed in resource pool m at time slot t. P represents the effective computing power weight that resource pool m can provide when facing the kth task at time slot t. m (t) represents the computing power queue length of resource pool m at time slot t, that is, the available computing power of resource pool m, O k (t) represents the data size of the kth task at time slot t, φ k(t) represents the CPU cycles required to process the kth task at time slot t, and w k (t) represents the income of processing the kth task at time slot t, e m represents the energy consumption of resource pool m when executing one CPU cycle, in W / GHz, c m represents the price per joule of hardware energy consumed by executing a task, and K represents the total number of all tasks.

[0143] The utility U brought by caching the requested data during the time slot t e The calculation formula of (t) is:

[0144]

[0145] Where C represents the total number of requested data at time slot t, represents the utility of resource pool m caching the cth request data at time slot t, Indicates whether the cth request data is cached in resource pool m at time slot t. The return cost is the return cost saved when the requested data is cached. represents the spectrum bandwidth orthogonally allocated from the base stations in resource pool m to the users covered by resource pool m, represents the spectrum efficiency of mobile devices connecting to base stations in resource pool m, Indicates whether the cth request data is in resource pool m at time slot t. Indicates the size of the request data to be cached. Indicates the cost of the space occupied by resource pool m to cache request data.

[0146] The utility function constructed in this application takes into account the mobility of tasks and mobile devices. At time slot t, the computing power obtained by completing the task execution in resource pool m and user mobility is The computing power consumed by executing tasks and caching request data is The task arrival rate is modeled as a Poisson process with a rate of Represents the expected number of tasks arriving at the task queue at each time slot. The set of tasks at time slot t is Indicates that the total number of tasks under this time slot is In this embodiment, it is assumed that the computing time of each task cannot exceed the length of a time slot, and a 3-tuple parameter is used. <O k (t),φ k (t),ω k (t)>The characteristics of the marking task, where O k (t) represents the data size of the kth task, φ k (t) represents the CPU cycles required to process the kth task, ωk (t) represents the income of processing the kth task. This model can make the computing tasks more flexible. K(t)} indicates whether all tasks are executed in the resource pool. It means that the kth task is executed in resource pool m at time slot t, otherwise it is not executed. In this embodiment, each task is executed by only one resource pool, which is expressed as

[0147] This application uses a Hidden Markov Model (HMM) to predict the probability of a mobile device entering each resource pool. Specifically, the HMM uses a forward-backward algorithm to predict the location of a mobile device based on the recently observed mobility trajectory. The location observation sequence of mobile device n is recorded as Among them l t is the position of mobile device n at time slot t. According to the position l of mobile device t t , we can know the order in which mobile devices enter the resource pool is Hidden Markov Model λ H It can be represented by a triple parameter <π, A, B>, where π = {π i} represents the set of initial hidden state probabilities, that is, the probability set of the mobile device in the resource pool, and the hidden state refers to which resource pool the mobile device is in; π i Indicates that the mobile device remains hidden at time slot t i The probability of A = {a ij} is from the hidden state s i To the hidden state s j The transition probability set, B = {b i (τ)} represents the hidden state s at time slot τ i The observed state o τ The probability set, π i 、a ij and b i (τ) can be calculated as follows:

[0148]

[0149] In the formula, Represents the hidden state s i To the hidden state s j The probability of Represents the hidden state s maintained at time slot t i The probability of Indicates that the hidden state at time slot τ is s j When the observation state is o τThe probability of a new round of model updates is Indicates that the hidden state at time slot t is s j Given the HMM model and the observation sequence, When the time slot is t, the hidden state is s i The probability of t (i, j) = P(s i ,s j |o1o2…o T ,λ H ) is the time slot t from the hidden state s i To the hidden state s j The probability of H Represents the HMM model.

[0150] Combine the above formula with the observed state o τ Combined, the observed state is the position l τ , thereby predicting the probability of the user entering each resource pool at the τ+1 time slot. Then select the resource pool with the highest probability and set p i →j(t) indicates that the mobile device moves from resource pool i to resource pool j at time slot t. Therefore, based on the above, we can obtain p j→m (t-1) and p m→i (t-1).

[0151] S5, such as Figure 4 As shown in the figure, the proximal policy optimization algorithm (PPO) and the attention mechanism are used to construct a proximal policy optimization algorithm based on the attention mechanism, including the following steps:

[0152] S5.1, using the proximal strategy optimization algorithm to establish a neural network model, the neural network model includes an Actor neural network and a Critic neural network, the Actor neural network includes a new-actor network and an old-actor network, the Critic neural network includes an attention layer built based on an attention mechanism, and the attention layer is used to perceive the contribution of each resource pool to the system utility;

[0153] In order to improve the training speed and make the sampled data reusable, the Actor neural network includes a new-actor network and an old-actor network. The similarity of the action probabilities between the two actor networks is used to limit the update range of the new strategy. The network architecture of the new-actor network and the old-actor network is the same. The actor network uses the observed state of the local resource pool as the input layer, contains a hidden layer, and uses the nonlinear activation function Relu; a softmax layer outputs the probabilities of all actions.

[0154] The critic neural network is similar to the Actor neural network, but different from the Actor neural network, it adds an attention layer after the input layer. Since the mobility of users changes the computing power and storage resource size in the resources, and users move between adjacent base stations, the computing power size and cache status of the resource pool are used as factors affecting the system task offloading decisions and data caching decisions. An attention mechanism is introduced in the neural network to obtain information about the resource pool. The attention mechanism is to identify the potential impact of the characteristics of the resource pool on the system. That is, the weight of the resource pool in the system is recorded, and coordinated and balanced from various angles. The input of the critic neural network is the state of each resource pool. For example Figure 3 As shown in the figure, the contribution of each resource pool to the system can be obtained based on the attention value of the attention mechanism. The darker color indicates that the resource pool has a greater contribution and a higher weight, so the neural network pays more attention to this computing power pool, and the lighter color pool indicates that its contribution is small.

[0155] The attention mechanism adopts the query-key-value concept to find the weight matrix between each resource pool. Define the state of each resource pool x i That is, the state of each resource pool in the state space χ(t) is the input h of the attention mechanism i , through the matrix W q , W k , W v For input h i The query value, key value, and value of each resource pool are obtained by linear change as follows:

[0156]

[0157] In the formula, h i represents the state of the i-th resource pool, W q , W k , W v Both represent learnable parameter matrices.

[0158] In order to obtain the similarity between the query element of resource pool i and the key element of resource pool j, this application applies an alignment function (such as a point function) and uses the distribution function softmax to map the alignment function score to the attention weight, that is, to obtain the influence weight of the resource pool on the system utility. The attention weight α of the resource pool on the system ij The calculation formula is:

[0159]

[0160] In the formula, scr ij Indicates the correlation between states, scrij =a(q i , k j ).

[0161] Finally, in order to retain the input features, the model introduces the value value and obtains the attention value att i As the input of the next hidden layer, the Critic neural network can then use limited resources to focus on the resource pool that has a greater impact on the system utility and continue learning. i as follows:

[0162]

[0163] S5.2, Strategy for initializing new-actor network and the old-actor network strategy Set the initial value of mobile devices n = 0, the initial value of time slots t = 0, set the total number of time slots T and the total number of mobile devices N;

[0164] S5.3, the initial state S of the receiving system t ;

[0165] Status S t The state space of is represented by χ(t), which is expressed as:

[0166]

[0167]

[0168] In the formula, represents the effective computing power weight that can be allocated to the Kth task by resource pool m at time slot t, Indicates whether the Cth request data is in resource pool m at time slot t. Among them, ξ c (t)∈{0,1}.

[0169] S5.4, the new-actor network follows the current strategy and state S t Select action a t ;

[0170] Action a t The action space of is represented by a(t), and its expression is:

[0171]

[0172] In the formula, Indicates whether resource pool m executes the Kth task, i.e., the task strategy, at time slot t. Its value is 1 for execution and 0 for non-execution. Indicates whether the resource pool m executes the Cth request data at time slot t, that is, the caching policy. A value of 1 represents caching, and a value of 0 represents non-caching.

[0173] S5.5, According to the selected action a t Obtain the observed reward r t And the next state S t+1 ;

[0174] The said reward r t Is represented by the reward function R(t), and its expression is:

[0175] R(t) = U(t) = U p (t) + U e (t).

[0176] S5.6, Store {s t , a t , r t} in the buffer;

[0177] S5.7, Calculate the importance weight r t (θ) of the new-actor network and the old-actor network;

[0178] The said importance weight r t (θ) is calculated by the formula:

[0179]

[0180] In the formula, π θ (a t , s t ) represents the policy of the new-actor network, Represents the policy of the old-actor network.

[0181] S5.8, Judge whether t < T. If so, execute t = t + 1 and return to step S5.4. Otherwise, execute step S5.9;

[0182] S5.9, Use the advantage function A π (s t , a t ) to evaluate the quality of the selected action a t in the state S t ;

[0183] The said advantage function A π (s t , a t ) is calculated by the formula:

[0184] A π (s t, a t )=Q π (s t , a t )-V π (s t );

[0185] In the formula, Q π (s t , a t ) represents the sum of discounted rewards obtained based on TD-error, V π (s t ) indicates the status value.

[0186] The sum of discounted rewards Q obtained according to TD-error π (s t , a t ) can be expressed as:

[0187] Q π (s t , a t )=r t +ΥV π (s t+1 );

[0188] In the formula, r t represents the reward obtained at time slot t, Υ represents the discount factor, V π (s t+1 ) represents the next state value, V π (s t+1 ) is the state value obtained by the critic neural network.

[0189] S5.10, assign the strategy of the new-actor network to the strategy of the old-actor network to calculate the importance weight for updating the network parameters;

[0190] S5.11, according to the mean square function L MSE Update the network parameters of the Critic neural network;

[0191] The mean square function L MSE The formula is:

[0192]

[0193] Where V(t) represents the state value at time slot t, Indicates the expected value of the state.

[0194] S5.12, according to the loss function L CLIPUpdate the network parameters of the new-actor network, and use the clipping method and TD-error method to limit the changes in each network update.

[0195] The loss function L CLIP has the following formula:

[0196]

[0197] In the formula, represents the advantage function A π (s t , a t ); has the following formula:

[0198]

[0199] In the formula, ∈ represents a hyperparameter.

[0200] S5.13, Judge whether n < N. If so, execute n = n + 1 and return to step S5.3. Otherwise, the training ends.

[0201] S6, Aiming to maximize the long-term utility function, use the proximal policy optimization algorithm of the attention mechanism established in step S5 to solve the long-term utility function.

[0202] The expression for maximizing the long-term utility function is:

[0203]

[0204] The constraint conditions of the long-term utility function are:

[0205] P m (t) ≤ Λ, m ∈ M; (a)

[0206]

[0207]

[0208]

[0209] p m (t) = min(Λ - P m (t), η m (t)); (e)

[0210]

[0211] In the formula, P m (t) represents the computing power queue length of resource pool m at time slot t, that is, the available computing power of resource pool m, p m(t) represents the computing power collected by resource pool m at time slot t, Λ represents the upper limit of the computing power collected by resource pool m at time slot t, and η m (t) represents the computing power obtained by completing the task execution in resource pool m and user mobility at time slot t, represents the set of all task indices at time slot t, φ k (t) represents the CPU cycles required to process the task corresponding to task index k at time slot t. Constraint (a) indicates that the computing power of the resource pool does not exceed the set upper limit, constraint (b) indicates whether the kth task is executed in the resource pool, constraint (c) indicates whether the cth request data is cached in the resource pool, constraint (d) indicates that there is only one resource pool for computing the kth task at time slot t, constraint (e) indicates the value of the computing power of resource pool m, and constraint (f) indicates that the computing power consumed by resource pool m to execute all tasks cannot exceed its available computing power.

[0212] In CPN, especially in joint allocation optimization, resources are abstracted into computable modules, which requires the system to quantify the contribution of each participant. Construct an Actor neural network consisting of an input layer, a hidden layer, and an output layer, where the hidden layer is a fully connected layer with 200 hidden nodes. In order to find out the influence of the service resource pool on its own decision and make a judgment that is beneficial to the entire system, the Critic neural network consists of an input layer, two hidden layers, and an output layer. Both hidden layers consist of an attention mechanism layer and a fully connected layer containing 100 hidden nodes. The discount factor of the APPO algorithm is 0.9, and the other main parameter settings are shown in the following table:

[0213]

[0214] In order to show the superiority of the attention-based proximal policy optimization algorithm (APPO) of the present invention, we used the actor-critic mechanism (AC), the actor-critic with attention mechanism (AAC), the DQN algorithm (Deep Q Network), the static allocation algorithm and the random allocation algorithm in resource allocation. The AC algorithm is a deep learning algorithm based on value and policy gradient; AAC is an AC algorithm with attention mechanism; DQN is an algorithm combining Q-learning and convolutional neural network (CNN); the static allocation algorithm allocates computing power and storage resources to each resource pool at a time; the random allocation algorithm is that computing tasks and data are randomly assigned to a certain resource pool of the system for computing and caching.

[0215] APPO algorithm parameter comparison: Figure 5 and Figure 6 It can be seen that when the minimum batch size = 32 and the learning rate is 0.001, the maximum reward is obtained. As the number of rounds gradually increases, the reward gradually increases and begins to converge. The reason why other parameters did not achieve good results may be that the convergence is too fast, the number of training times is small, the parameter correction is slow, and the network easily converges to some bad local optimal points.

[0216] Comparison of APPO algorithm with other algorithms: Figure 7 , Figure 8 and Fig. 9 It can be seen that the APPO algorithm can obtain the highest system utility, computing power utility and cache utility. Because the APPO algorithm not only pays more attention to the status of the resource pool, but also limits the network update range by using the old and new networks in the training process, and reuses the sampled data using offline strategies, it can improve the algorithm training speed and take correct actions to accelerate the learning process, so as to better obtain higher system utility.

[0217] Comparison of different system parameters: Fig.10 , Fig.11 and Fig.12 It can be seen that APPO has a great advantage in both the number of tasks and the system price. This is because the APPO algorithm pays more attention to the resource pool that contributes more to the system utility, thereby reducing system overhead and improving model efficiency.

[0218] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A contribution-aware in-network pooled resource allocation optimization method in a computing network, characterized in that: It includes the following steps: S1. Construct a computing power network resource allocation system including an infrastructure layer, a resource pool layer, and a CPN scheduler layer. The resource pool layer includes several resource pools, and each resource pool is composed of resources provided by the infrastructure layer; S2. Based on the tasks in the infrastructure layer and the dynamic characteristics of mobile devices, construct the computing power queue length of each resource pool in the resource pool layer; S3. In the CPN scheduler layer, use the deep reinforcement learning algorithm to establish a computing power model and a cache model respectively, and construct the computing power model and the cache model as Markov processes respectively; S4. Based on the computing power model and the cache model established in step S3, construct the utility function of the computing power network resource allocation system; S5. Use the proximal policy optimization algorithm and the attention mechanism to construct the proximal policy optimization algorithm based on the attention mechanism; S6. With the goal of maximizing the long-term utility function, use the proximal policy optimization algorithm based on the attention mechanism established in step S5 to solve the long-term utility function.

2. The method for optimizing in-network pooled resource allocation based on contribution perception in a computing power network according to claim 1, characterized in that: In step S2, the calculation formula for the computing power queue length of the resource pool is: P m (t)=P m (t-1)-R m (t-1)+p m (t-1); Where P m (t-1) represents the computing queue length of resource pool m at time slot t-1, R m (t-1) represents the computing power consumed by resource pool m at time slot t-1, p m (t-1) represents the computing power collected by resource pool m at time slot t-1, P m (t) represents the computing queue length of resource pool m at time slot t; The computing power R consumed by resource pool m at the t-1 time slot m The calculation formula for (t-1) is: In the formula, It represents the computing power consumed by resource pool m to process tasks at time slot t-1. It represents the computing power lost due to the mobile device leaving resource pool m at time slot t-1.

3. The method for optimizing in-network pooled resource allocation based on contribution perception in a computing power network according to claim 2, characterized in that: The computing power consumed by resource pool m to process tasks during the t-1 time slot The calculation formula is: In the formula, represents the set of all tasks at time slot t-1, φ k (t-1) represents the CPU cycles required to process the kth task at the t-1 time slot, Indicates whether the kth task is processed in resource pool m at time slot t-1; The computing power lost due to the mobile device leaving resource pool m during the t-1 time slot The calculation formula is: In the formula, p j→m (t-1) represents the probability of mobile device n moving from resource pool j to resource pool m at time slot t-1, h n It represents the idle computing capacity contributed by mobile device n to the resource pool.

4. The method for optimizing in-network pooled resource allocation based on contribution perception in a computing power network according to claim 2, characterized in that: The computing power p collected by resource pool m at time slot t-1 m The calculation formula for (t-1) is: p m (t-1)=min(Λ-P m (t-1), the m (t-1)); Where P m (t-1) represents the computing power queue length of resource pool m at time slot t-1, that is, the available computing power of resource pool m, Λ represents the upper limit of the computing power collected by resource pool m at time slot t-1, η m (t-1) represents the computing power obtained when the tasks in resource pool m are completed and mobile devices enter at time slot t-1; The computing capacity η obtained when the tasks in the resource pool m are completed and the mobile device enters the t-1 time slot m The calculation formula for (t-1) is: In the formula, It represents the computing power obtained by completing the task in resource pool m at time slot t-1. Indicates the computing power obtained by the mobile device entering resource pool m at time slot t-1.

5. The method for optimizing in-network pooled resource allocation based on contribution perception in a computing power network according to claim 1, characterized in that: In step S3, the expression of the computing power ratio transition probability matrix of the computing power model is: In the formula, represents the computing power ratio transfer probability matrix of resource pool m facing the kth task at time slot t, L represents the number of rows and columns of the matrix, Indicates the probability of computing power weight state transition; The cache state probability matrix of the cache model is: In the formula, Γ c (t) represents the cache status probability matrix facing the cth request data at time slot t, Represents the cache state transition probability.

6. The method for optimizing in-network pooled resource allocation based on contribution perception in a computing power network according to claim 1, characterized in that: In step S4, the expression of the utility function of the computing power network resource allocation system is: U(t)=U p (t)+U e (t); Where U(t) represents the utility function at time slot t, U p (t) represents the utility brought by executing the task at time slot t, U e (t) represents the utility brought by caching request data at time slot t; The utility U brought by executing the task at the time slot t p The calculation formula of (t) is: In the formula, represents the utility of the kth task being processed in resource pool m at time slot t, Indicates whether the kth task is processed in resource pool m at time slot t. P represents the effective computing power weight that resource pool m can provide when facing the kth task at time slot t. m (t) represents the computing queue length of resource pool m at time slot t, O k (t) represents the data size of the kth task at time slot t, φ k (t) represents the CPU cycles required to process the kth task at time slot t, and w k (t) represents the income of processing the kth task at time slot t, e m represents the energy consumption of resource pool m executing one CPU cycle, c m represents the price per joule of hardware energy consumption caused by executing a task, K represents the total number of tasks, and M represents the total number of mobile devices; The utility U brought by caching the requested data during the time slot t e The calculation formula of (t) is: Where C represents the total number of requested data at time slot t, represents the utility of resource pool m caching the cth request data at time slot t, Indicates whether the cth request data is cached in resource pool m at time slot t. represents the return cost, represents the spectrum bandwidth orthogonally allocated from the base stations in resource pool m to the users covered by resource pool m, represents the spectrum efficiency of mobile devices connecting to base stations in resource pool m, Indicates whether the cth request data is in resource pool m at time slot t. Indicates the size of the request data to be cached. Indicates the cost of the space occupied by resource pool m to cache request data.

7. The method for optimizing in-network pooled resource allocation based on contribution perception in a computing power network according to claim 1, characterized in that: Step S5 includes the following steps: S5.

1. Use the proximal policy optimization algorithm to establish a neural network model. The neural network model includes an Actor neural network and a Critic neural network. The Actor neural network includes a new-actor network and an old-actor network. The Critic neural network includes an attention layer constructed based on the attention mechanism. The attention layer is used to perceive the contribution of each resource pool to the system utility; S5.2, Strategy for initializing new-actor network and the old-actor network strategy Set the initial value of mobile devices n = 0, the initial value of time slots t = 0, set the total number of time slots T and the total number of mobile devices N; S5.3, the initial state S of the receiving system t ; S5.4, the new-actor network follows the current strategy and state S t Select action a t ; S5.5, according to the selected action a t Get the observed reward r t and the next state S t+1 ; S5.6, {s t , a t , r t }store in buffer; S5.7, calculate the importance weights r of the new-actor network and the old-actor network t (θ); S5.

8. Judge whether t < T. If so, execute t = t + 1 and return to step S5.

4. Otherwise, execute step S5.9; S5.9, using the advantage function to evaluate the state S t Next Select Action a t The good or bad; S5.

10. Assign the policy of the new-actor network to the policy of the old-actor network; S5.

11. Update the network parameters of the Critic neural network according to the mean square function; S5.

12. Update the network parameters of the new-actor network according to the loss function; S5.

13. Judge whether n < N. If so, execute n = n + 1 and return to step S5.

3. Otherwise, the training ends.

8. The method for optimizing in-network pooled resource allocation based on contribution perception in a computing power network according to claim 7, characterized in that: The state S t The state space of is represented by χ(t), which is expressed as: In the formula, represents the effective computing power weight that can be allocated to the Kth task by resource pool m at time slot t, Indicates whether the Cth request data is in resource pool m at time slot t; The action a t The action space of is represented by a(t), and its expression is: In the formula, Indicates whether resource pool m executes the Kth task at time slot t, i.e., the task strategy. Indicates whether resource pool m executes the Cth request data, that is, the cache strategy, at time slot t.

9. The method for optimizing in-network pooled resource allocation based on contribution perception in a computing power network according to claim 7, characterized in that: In step S5.7, the importance weight r t The calculation formula of (θ) is: In the formula, π θ (a t ,s t ) represents the strategy of the new-actor network, Represents the strategy of the old-actor network.

10. The method for optimizing in-network pooled resource allocation based on contribution perception in a computing power network according to claim 1, characterized in that: In step S6, the expression of maximizing the long-term utility function is: Where U(t) represents the utility function at time slot t, T represents the total number of time slots, Indicates whether the cth request data is cached in resource pool m at time slot t. Indicates whether the kth task is processed in resource pool m at time slot t; The constraint condition of the long-term utility function is: P m (t)≤Λ,m∈M; (a) p m (t)=min(Λ-P m (t), the m (t); (e) Where P m (t) represents the computing queue length of resource pool m at time slot t, p m (t) represents the computing power collected by resource pool m at time slot t, Λ represents the upper limit of the computing power collected by resource pool m at time slot t, and η m (t) represents the computing power obtained by completing the task execution in resource pool m and user mobility at time slot t, represents the set of all task indices at time slot t, φ k (t) represents the CPU cycles required to process the task corresponding to task index k at time slot t, M represents the total number of mobile devices, Represents a collection of resource pools.

Images