Multi-dimensional resource joint optimization method in MEC auxiliary system

By building a multi-dimensional resource model in the MEC auxiliary system and using deep reinforcement learning algorithms, the problem of single resource optimization goals in the existing technology is solved, and the joint optimization of computing, communication and cache resources is realized, improving system performance and user experience .

CN118945688APending Publication Date: 2024-11-12SHANGHAI UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410946105.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The resource optimization solution for user computing and offloading in the existing MEC auxiliary system only considers two resource dimensions, and fails to effectively schedule more resources in the system, resulting in low resource utilization and single optimization goals, which limits the overall improvement of system performance. .

Method used

A multi-dimensional resource joint optimization method based on MEC auxiliary system is proposed. By constructing a network model, communication model and cache model, combined with deep reinforcement learning algorithm, the joint optimization of computing, communication and cache resources is realized, and Markov decision-making process and Near-end strategy optimization algorithm solves the problem of minimizing system weighted overhead.

Benefits of technology

Multi-dimensional joint optimization of computing, communication and cache resources is realized, which significantly reduces the average delay and energy consumption of computing tasks of user equipment, and improves user experience quality and system performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118945688A_ABST
    Figure CN118945688A_ABST
Patent Text Reader

Abstract

A multi-dimensional resource joint optimization method based on user-oriented computing unloading in an MEC auxiliary system comprises the following steps: constructing a system model for user-oriented computing unloading in an MEC auxiliary network, wherein the system model comprises a network model, a communication model, a cache model and a computing unloading model; a joint optimization problem with the aim of minimizing system weighted overhead is constructed based on the model, an optimal strategy is solved through a deep reinforcement learning algorithm, and multi-dimensional resource joint optimization is achieved. According to the method, in an MEC-assisted network scene, different calculation task types of users are considered, joint optimization calculation, communication and cache (3C) resources are combined, the resources of all dimensions are fully utilized, the potential advantages of MEC are exerted, in the joint optimization problem modeling process, the time delay and energy consumption of user equipment calculation task processing are reduced at the same time, and the efficiency of joint optimization is improved. The overall performance and the user experience of the system are remarkably improved, and the problem that the resource optimization target is single in the prior art is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a technology in the field of mobile edge computing, specifically a multi-dimensional resource joint optimization method for user computing offloading based on MEC-assisted system. Background Art

[0002] To alleviate the problem of insufficient resources in terminal equipment (UE), wireless communication networks have proposed a new solution combined with mobile edge computing (MEC). This solution uses the rich resources of MEC servers to sink computing power and caching capabilities to the edge of the network, close to the source of data requests. This not only expands UE resources, but also reduces data transmission delays and network congestion, avoiding the need to process through remote servers or the cloud every time.

[0003] The resource optimization for user computing offloading in the existing MEC-assisted system mainly has the following defects and deficiencies: the current resource optimization scheme usually only considers two resource dimensions, fails to effectively schedule more resources in the system, and results in low resource utilization. With the surge in communication network traffic and the intensification of resource competition, communication networks have put forward higher requirements for the joint optimization of higher-dimensional resources to give full play to the potential advantages of MEC. In addition, when modeling the optimization problem of resource management, the existing schemes usually only focus on a single optimization objective, such as latency or energy consumption. This single-objective optimization lacks a more complete perspective to meet user needs, which limits the overall improvement of system performance. Therefore, the joint optimization of multiple objectives is particularly important. Summary of the invention

[0004] In view of the deficiency of the prior art that the simultaneous optimization of multi-dimensional resources is not fully considered, the present invention proposes a multi-dimensional resource joint optimization method based on the MEC-assisted system for user computing offloading. In the MEC-assisted network scenario, the method considers the different computing task types of users, jointly optimizes computing, communication, and cache (3C) resources, fully utilizes resources of various dimensions, and gives play to the potential advantages of MEC. Compared with the method that only considers single-objective optimization, the present invention considers reducing the latency and energy consumption of computing task processing of user devices in the process of joint optimization problem modeling, significantly improves the overall performance of the system and user experience, and makes up for the problem of single resource optimization target in the prior art. The present invention can effectively solve the proposed weighted overhead, i.e., the problem of minimizing latency and energy consumption, while optimizing multi-dimensional resources at the same time.

[0005] The present invention is achieved through the following technical solutions:

[0006] The present invention relates to a multi-dimensional resource joint optimization method based on user computing offloading in an MEC-assisted system. A system model for user computing offloading in an MEC-assisted network including a network model, a communication model, a cache model and a computing offloading model is constructed, and a joint optimization problem with the goal of minimizing the system weighted overhead is constructed based on the model. The optimal strategy is then solved by a deep reinforcement learning algorithm to achieve multi-dimensional resource joint optimization.

[0007] The network model describes MEC nodes, user equipment (UE) nodes, base station nodes and network topology, providing a network scenario for multi-dimensional resource optimization.

[0008] The communication model ensures that the offloading process of UE computing tasks complies with wireless communication standards and protocols, including the conditions of wireless channel characteristics and transmission rate, and improves data transmission efficiency as much as possible, provides a basis for the delay and energy consumption in the computing offloading process, and improves the utilization of communication resources.

[0009] The cache model is designed to dynamically adjust the allocation of cache resources, monitor the usage of cache resources, and improve the utilization rate of cache resources.

[0010] The computing offloading model is designed to model the computing task offloading decision of the UE. The offloading decision fully considers the delay and energy consumption of the UE computing task processing and improves the computing resource utilization.

[0011] The joint optimization problem adopts a reasonable mathematical method to comprehensively and jointly optimize the offloading decision, cache decision and channel allocation in the process of UE computing task processing, aiming to minimize the weighted average overhead of the system to achieve joint optimization of 3C resources.

[0012] The joint optimization problem is preferably converted into a Markov decision process (MDP) model, and then solved using a proximal policy optimization (PPO) reinforcement learning model to obtain the optimal strategy. Technical Effects

[0013] Compared with the existing two-dimensional resource optimization (computing resources and communication resources, cache resources and communication resources) scheme of MEC-assisted system, the present invention realizes the multi-dimensional joint optimization of computing, communication and cache (3C) resources, which provides resource information from multiple angles for the processing of user computing tasks, fully mobilizes various resources in the system to serve the processing of computing tasks, and adopts the more stable PPO algorithm in the intelligent reinforcement learning algorithm to solve multi-objective problems, thereby effectively reducing the average delay and energy consumption of UE computing task processing and improving the quality of user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is a flow chart of the present invention;

[0015] Figure 2 Schematic diagram of the system model for MEC-assisted joint 3C resource optimization;

[0016] Figure 3 Schematic diagram of deep reinforcement learning framework;

[0017] Figure 4 Provide a description of the problem-solving framework based on the PPO method;

[0018] Figure 5 Schematic diagram of average delay and energy consumption of different mechanisms in the embodiment as the number of UEs changes;

[0019] In the figure: (a) The average delay of different mechanisms varies with the number of users (b) The average energy consumption of different mechanisms varies with the number of users;

[0020] Figure 6 Schematic diagram of the change of system energy consumption of different mechanisms of the embodiment with the MEC computing power. DETAILED DESCRIPTION

[0021] like Figure 1 As shown, this embodiment relates to a multi-dimensional resource joint optimization method for user computing offloading based on MEC-assisted system, specifically including:

[0022] Step 1: Build Figure 2 The system model of MEC-assisted joint 3C resource optimization shown;

[0023] The system model includes: a network model, a communication model, a cache model and a computing offloading model, wherein: the network model describes the MEC nodes, UE nodes, base station nodes and their topological structure, providing an analysis scenario for subsequent steps; the communication model describes the communication conditions when the UE computing task is offloaded, including the channel model and the transmission rate, so as to improve the utilization of communication resources; the cache model describes the cache decision of the application required to process different computing task types, and adjusts the cache resource allocation to make full use of the cache resources of the MEC; the computing offloading model models the computing task offloading decision of the UE to make full use of the computing resources of the MEC.

[0024] The network model includes: a number of UEs as a set and a base station connected to the MEC server, which is used to process the UE's computing tasks and cache the applications that process the computing tasks, as well as a set of subchannels with the same bandwidth based on OFDM technology Among them: UEs in A series of calculation tasks will be generated at all times Since the types of computing tasks are different, each type of computing task is represented by a corresponding application program. Perform calculation processing, indicating that the calculation task is the hth type of calculation task; through the binary variable m j,h ∈{0,1} describes the jth computing task of UE Whether it is handled by application h, where: The number of CPU cycles required to process the computational task; is the data volume of the computing task, is the data size of the task, The size of the data used to process the computational task; The amount of data after the computing task is processed.

[0025] The application program is a software or program that performs a specific computing task.

[0026] To simplify the calculation, assume that the computing tasks generated by the UE are countable. is regarded as the number of computing tasks generated by the UE at time t, and This means that the same application can handle multiple computing tasks. Since the UE itself has certain computing and cache resources, computing tasks can be processed locally or offloaded to the MEC server for processing.

[0027] The binary offloading index of the computing task This means that the jth computing task of the uth UE will be processed locally on the UE. This indicates that the j-th computing task of the u-th UE will be offloaded to the MEC server for processing.

[0028] The communication model is used for reliable transmission during UE offloading computing tasks, including: an uplink transmission submodel for computing task offloading and a downlink transmission submodel for result return, wherein: during the uplink transmission process, the uplink transmission rate achieved by the u-th UE offloading the h-th type of computing task through subchannel k is The total transmission rate of the u-th UE offloading computing tasks represents the channel allocation index at time t, which is described as when subchannel k is allocated to user equipment u, On the contrary, if the subchannel If it is not assigned to user equipment u, B is the channel bandwidth, and the bandwidth of all uplink subchannels is the same; is the transmission power when u uses subchannel k to offload the jth computing task; is the channel gain when u uses subchannel k to offload the jth computing task; σ 2 is the noise power of the channel model. The downlink transmission process uses a separate channel to transmit the results of the computing task. When the bandwidth of the channel is B 0 , the transmission power is p m When the result is transmitted back, the downlink transmission rate

[0029] In this embodiment, at the same time, a subchannel is only allocated to one user equipment, that is,

[0030] The cache model includes: the cache decision needs to meet: Updates to cache decisions need to satisfy The update of cache status needs to satisfy After the cache status is updated, the cached data content still cannot exceed the maximum capacity of the MEC cache resource, that is, it needs to meet Where: Cache decision variables Indicates whether the hth application has been cached in the MEC server, described as Indicates that the hth application is not cached in MEC, otherwise The cache has the hth application; M c The maximum cache resource capacity of MEC; is the update decision variable for application h, described as Indicates that the cache status of application h remains unchanged. Indicates that the application h will be cached. Indicates that application h will be deleted.

[0031] The computation offloading model includes: the local processing delay of the jth computation task of the uth UE Energy consumption Uplink transmission delay Uplink transmission energy consumption Downlink transmission delay Downlink transmission energy consumption The total delay of the transmission process Total energy consumption Processing delay of the jth computing task of the uth UE on the MEC server Processing energy consumption Where: f u is the CPU frequency of user equipment u; when the application processing the computing task has been cached in MEC, the transmission delay When the application processing the computing task is not cached in the MEC, the transmission delay The data after the calculation task is completed is The fixed bandwidth of the downlink transmission process is B 0 ; is the ratio of computing resources allocated to user equipment u by MEC, satisfying f MEC It is the maximum computing resource of MEC.

[0032] Step 2: Modeling the joint optimization problem for minimizing weighted overhead, including:

[0033] 2.1 A series of computing tasks generated by computing u Total delay Total energy consumption

[0034] 2.2 Construct the weighted average overhead minimization problem of all UEs passing through T time slots in the system model in step 1: The constraints include: C1: C2: C3: C4: C5: C6: C7: Among them: Optimization variables Offloading decision variables for UE’s computational tasks; Assigning decision variables to subchannels; Represents the update decision of the application; Represents the caching decision of the application; and are the latency and energy consumption weight coefficients of the computing task respectively, and

[0035] Step 3: Optimization problem conversion based on Markov decision process (MDP), that is, learning the underlying model through interaction with the environment, converting the joint optimization problem model established in step 2 into an MDP model, including:

[0036] State space: The system state at any time slot t is defined as s t = {g t ,Q t ,|M t |,φ t}∈S, state variable is the channel gain of u transmitting the jth computing task to MEC through subchannel k, is the information of the j-th computing task of u, is the number of computing tasks generated by u, is the cache state of application h at the initial state of MEC at time slot t.

[0037] Action space: Action a obtained at any time t t ={α t ,b t ,ψ t}∈A, and at the same time, the jth action in the system is and in: is u’s offloading decision for the jth computing task, The set of subchannels selected for u to transmit the jth computing task ψ t The cache update decision of MEC at the end of the time slot These decisions form a set of actionable actions that enable the UE and MEC server to interact. At each time step t, the UE makes decisions including whether to offload computing tasks and application data and select multiple data transmission subchannels. The decision made by the MEC server determines whether to update the cached application. The UE and MEC server form a system and are considered as an intelligent agent in the proposed algorithm. Through the actions taken by the system at each time step, the state of the system is continuously updated, thereby accumulating experience and continuously evaluating.

[0038] Reward function: r t = r(s t ,a t )=-C ave , where: s t and a t are the initial state and action at the beginning of time slot t, respectively, and the weighted average cost C of completing all UE computing tasks in the entire system ave Implement the reward function r t .

[0039] Step 4: Based on Figure 4 The proximal policy optimization (PPO) reinforcement learning model shown solves the MDP model in step 3 to obtain the optimal policy.

[0040] The PPO reinforcement learning model includes: an actor network and a critic network, where: the actor network uses π θ For the proximal strategy, the critic network adopts V σ Evaluate the value function associated with the state, θ and σ are the network parameters of the actor network and the critic network respectively.

[0041] In this embodiment, the PPO reinforcement learning model introduces proximal constraints To prevent the strategy from updating too much, thus improving the stability and efficiency of training, specifically: the actor network state at the previous moment is regarded as an old actor network state, and the proximal strategy is adopted Generate action a based on the current observation t The probability density distribution of the critic network V σ Assessment and Status t Associated Rewards t After the replay buffer D stores the "state, action, reward value" tuple of each time step, the tuple is used to train the new network π θ and V σ of the sample.

[0042] The loss function used in the training of the actor network is the clipping substitution loss function in: is the expectation of a set of training sample batches, is the ratio of the new strategy to the old strategy, which is used to ensure the stability of the strategy update process, ∈ is the clipping fraction, and 1-≤clip(g t (θ),1-,1+)≤1+, for A t =Q(s t ,a t )-V σ (s t ) is evaluated as

[0043] The loss function used in the training of the critic network is Where: G t is an estimate of the discounted return, calculated as is the target state value function, It is regularly updated with the latest σ to ensure the stability of the learning process.

[0044] Through specific actual experiments, the effectiveness of the method of the present invention under different conditions was verified under the experimental environment settings described in Table 1. During the training of the reinforcement learning framework, the network converged after 100 episodes. The test results show that when the CPU frequency of the MEC server is 3GHz, the system energy consumption is reduced by up to 175J.

[0045] Table 1

[0046] This example uses the following methods to perform reasoning deployment and performance verification:

[0047] During the experimental comparison, several control mechanisms were designed by increasing the number of UEs to verify the effectiveness of the proposed optimization method. The control mechanisms are: complete offloading, that is, all computing tasks generated by UEs are transmitted to the MEC server for processing, not processed locally; local execution, that is, all computing tasks generated by UEs are processed locally, not offloaded to the MEC server for processing; random offloading, that is, all UEs make random offloading decisions, without optimizing offloading decisions.

[0048] In terms of algorithm framework, the present invention uses an open source framework based on Python and PyTorch 2.0 to implement the PPO algorithm and uses the Adam optimizer for training. The PPO algorithm uses a three-layer fully connected neural network with 64 neurons in each hidden layer. The learning rate used to train the actor and critic networks is set to 0.0003, and the limit policy update parameter ε is fixed to 0.2.

[0049] like Figure 5 As shown, when the CPU frequencies of the MEC server and user equipment are fixed, as the number of user equipment increases, the intelligent 3C resource optimization solution proposed in the present invention is significantly better than the other three mechanisms in performance. It is manifested in lower latency and energy consumption when processing UE computing tasks, thereby improving the processing sensitivity of computing tasks. The three comparison mechanisms and this method use the same system structure and parameters to highlight the necessity of joint optimization of various resources in the present invention. Through intelligent optimization and resource allocation, the present invention significantly reduces the average latency and energy consumption of the system, thereby improving the overall system performance.

[0050] like Figure 6 As shown in the figure, when the number of user devices in the entire system is fixed, as the computing power of the MEC server increases, the multi-dimensional resource intelligent optimization strategy adopted by the present invention performs better than other mechanisms in reducing system energy consumption. In particular, since the local execution mechanism does not take advantage of the computing and caching advantages of the MEC server, all computing tasks are processed locally, resulting in the inability to optimize system energy consumption. The research results show that as the computing power of MEC increases, the present invention can make full use of system resources and significantly reduce system energy consumption.

[0051] Compared with the prior art, this method more comprehensively considers the optimization of the three resource dimensions of computing, caching, and communication. Specifically, by adopting a stable intelligent reinforcement learning algorithm, dynamic and intelligent decisions are made on the offloading of UE computing tasks, the caching of applications, and the allocation of channel resources. This enables joint consideration of multiple perspectives when processing UE computing tasks in the MEC-assisted system, making full use of system resources. At the same time, the present invention comprehensively considers the latency and energy consumption factors of UE computing task processing, making full use of MEC resources while improving the quality of user experience.

[0052] The above-mentioned specific implementation can be partially adjusted in different ways by those skilled in the art without departing from the principle and purpose of the present invention. The protection scope of the present invention shall be based on the claims and shall not be limited by the above-mentioned specific implementation. Each implementation scheme within its scope shall be subject to the constraints of the present invention.

Claims

1. A multi-dimensional resource joint optimization method for user computing offloading in MEC-assisted system, It is characterized in that By building a system model for user computing offloading in the MEC-assisted network that includes a network model, a communication model, a cache model, and a computing offloading model, and based on this model, a joint optimization problem with the goal of minimizing the system weighted overhead is constructed. The optimal strategy is then solved using a deep reinforcement learning algorithm to achieve multi-dimensional resource joint optimization. The joint optimization problem is converted into a Markov decision process (MDP) model, and then the optimal strategy is solved by using a proximal policy optimization (PPO) reinforcement learning model.

2. According to claim 1, the multi-dimensional resource joint optimization method based on user computing offloading in the MEC-assisted system, Its characteristics are: The system model includes: a network model, a communication model, a cache model and a computing offloading model, wherein: the network model describes the MEC nodes, UE nodes, base station nodes and their topological structure, providing an analysis scenario for subsequent steps; the communication model describes the communication conditions when the UE computing task is offloaded, including the channel model and the transmission rate, so as to improve the utilization of communication resources; the cache model describes the cache decision of the application required to process different computing task types, and adjusts the cache resource allocation to make full use of the cache resources of the MEC; the computing offloading model models the computing task offloading decision of the UE to make full use of the computing resources of the MEC.

3. According to claim 2, the multi-dimensional resource joint optimization method based on user computing offloading in the MEC-assisted system, Its characteristics are: The network model includes: a number of UEs as a set and a base station connected to the MEC server, which is used to process the UE's computing tasks and cache the applications that process the computing tasks, as well as a set of subchannels with the same bandwidth based on OFDM technology Among them: UEs in A series of calculation tasks will be generated at all times Since the types of computing tasks are different, each type of computing task is represented by a corresponding application program. Perform calculation processing, indicating that the calculation task is the hth type of calculation task; through the binary variable m j,h ∈{0,1} describes the jth computing task of UE Whether it is handled by application h, where: The number of CPU cycles required to process the computational task; is the data volume of the computing task, is the data size of the task, The size of the data used to process the computational task; The amount of data after the computing task is processed.

4. According to claim 2, the multi-dimensional resource joint optimization method based on user computing offloading in the MEC-assisted system, Its characteristics are: The communication model includes: an uplink transmission sub-model for unloading computing tasks and a downlink transmission sub-model for returning results, wherein: during the uplink transmission process, the uplink transmission rate achieved by the u-th UE unloading the h-th type of computing tasks through the sub-channel k is The total transmission rate of the u-th UE offloading computing tasks represents the channel allocation index at time t, which is described as when subchannel k is allocated to user equipment u, On the contrary, if the subchannel If it is not assigned to user equipment u, B is the channel bandwidth, and the bandwidth of all uplink subchannels is the same; is the transmission power when u uses subchannel k to offload the jth computing task; is the channel gain when u uses subchannel k to offload the jth computing task; σ 2 is the noise power of the channel model. The downlink transmission process uses a separate channel to transmit the calculation task processing results. When the channel bandwidth is B 0 , the transmission power is p m When the result is transmitted back, the downlink transmission rate 5. According to claim 2, the multi-dimensional resource joint optimization method based on user computing offloading in the MEC-assisted system, Its characteristics are: The cache model includes: the cache decision needs to meet: Updates to cache decisions need to satisfy The update of cache status needs to satisfy After the cache status is updated, the cached data content still cannot exceed the maximum capacity of the MEC cache resource, that is, it needs to meet Where: Cache decision variables Indicates whether the hth application has been cached in the MEC server, described as Indicates that the hth application is not cached in MEC, otherwise The cache has the hth application; M c The maximum cache resource capacity of MEC; is the update decision variable for application h, described as Indicates that the cache status of application h remains unchanged. Indicates that the application h will be cached. Indicates that application h will be deleted.

6. According to claim 2, the multi-dimensional resource joint optimization method based on user computing offloading in the MEC-assisted system, Its characteristics are: The computation offloading model includes: the local processing delay of the jth computation task of the uth UE Energy consumption Uplink transmission delay Uplink transmission energy consumption Downlink transmission delay Downlink transmission energy consumption The total delay of the transmission process Total energy consumption Processing delay of the jth computing task of the uth UE on the MEC server Processing energy consumption Where: f u is the CPU frequency of user equipment u; when the application processing the computing task has been cached in MEC, the transmission delay When the application processing the computing task is not cached in the MEC, the transmission delay The data after the calculation task is processed is The fixed bandwidth of the downlink transmission process is B 0 ; is the ratio of computing resources allocated to user equipment u by MEC, satisfying f MEC It is the maximum computing resource of MEC.

7. According to claim 1, the multi-dimensional resource joint optimization method based on user computing offloading in the MEC-assisted system, Its characteristics are: The joint optimization problem is constructed in the following way: 2.1 A series of computing tasks generated by computing u Total delay Total energy consumption 2.2 Construct the weighted average overhead minimization problem of all UEs passing through T time slots in the system model: The constraints include: C1: C2: C3: C4: C5: C6: C7: Among them: Optimization variables Offloading decision variables for UE’s computational tasks; Assigning decision variables to subchannels; Represents the update decision of the application; Represents the caching decision of the application; and are the latency and energy consumption weight coefficients of the computing task respectively, and 8. According to claim 1, the multi-dimensional resource joint optimization method based on user computing offloading in the MEC-assisted system, Its characteristics are: The solution is obtained through a deep reinforcement learning algorithm, i.e., the optimization problem is converted based on a Markov decision process (MDP), i.e., the underlying model is learned through interaction with the environment, and after the joint optimization problem is converted into an MDP model, the MDP model is solved based on a proximal policy optimization (PPO) reinforcement learning model to obtain the optimal strategy.

9. The multi-dimensional resource joint optimization method for user computing offloading based on MEC-assisted system according to claim 8, Its characteristics are: The deep reinforcement learning algorithm includes: State space: The system state at any time slot t is defined as s t = {g t ,Q t ,|M t |,φ t }∈S, state variable is the channel gain of u transmitting the jth computing task to MEC through subchannel k, is the information of the j-th computing task of u, is the number of computing tasks generated by u, is the cache state of application h at the initial state of MEC at time slot t; Action space: Action a obtained at any time t t ={α t ,b t ,ψ t }∈A, and at the same time, the jth action in the system is and in: is u’s offloading decision for the jth computing task, The set of subchannels selected for u to transmit the jth computing task ψ t The cache update decision of MEC at the end of the time slot These decisions form a set of actionable actions that enable the UE and MEC server to interact. At each time step t, the UE makes decisions on whether to offload computing tasks and application data and select multiple data transmission subchannels. The MEC server makes decisions on whether to update cached applications. The UE and MEC server form a system and are considered as an intelligent agent in the proposed algorithm. Through the actions taken by the system at each time step, the state of the system is continuously updated, thereby accumulating experience and continuously evaluating; Reward function: r t = r(s t ,a t )=-C ave , where: S t and A t are the initial state and action at the beginning of time slot t, respectively, and the weighted average cost C of completing all UE computing tasks in the entire system ave Implement the reward function R t .

10. The multi-dimensional resource joint optimization method for user computing offloading based on MEC-assisted system according to claim 8, Its characteristics are: The PPO reinforcement learning model includes: an actor network and a critic network, where: the actor network uses π θ For the proximal strategy, the critic network adopts V σ Evaluate the value function associated with the state, where θ and σ are the network parameters of the actor network and the critic network respectively; The PPO reinforcement learning model introduces proximal constraints To prevent the strategy from updating too much, thus improving the stability and efficiency of training, specifically: the actor network state at the previous moment is regarded as an old actor network state, and the proximal strategy is adopted Generate action a based on the current observation t The probability density distribution of the critic network V σ Assessment and Status t Associated Rewards t After the playback buffer D stores the "state, action, reward value" tuple of each time step, the tuple is used to train the new network π θ and V σ Samples of The loss function used in the training of the actor network is the clipping substitution loss function in: is the expectation of a set of training sample batches, is the ratio of the new strategy to the old strategy, which is used to ensure the stability of the strategy update process, ∈ is the clipping fraction, and 1-≤clip(g t (θ),1-,1+)≤1+, for A t =Q(s t ,a t )-V σ (s t ) is evaluated as The loss function used in the training of the critic network is Where: G t is an estimate of the discounted return, calculated as is the target state value function, It is regularly updated with the latest σ to ensure the stability of the learning process.