Optimization method and system in IPBFT consensus multiple access edge computing system

Abstract

The invention belongs to the technical field of wireless communication, and discloses an optimization method and system in an IPBFT consensus multiple access edge computing system, which comprises the steps of safety task unloading and computing resource allocation, and comprises the step of jointly optimizing a binary task unloading decision and computing resource block allocation. Meanwhile, an improved practical IPBFT consensus algorithm is disclosed in a block chain system to enhance a consensus program, and then consensus committee selection is carried out, so that consensus delay and failure rate are reduced to the greatest extent. The system is subjected to joint optimization under the influence of factors such as edge node computing power, IPBFT node number limitation, task processing and block chain consensus delay. The small base station not only serves as an edge node in the MEC system, but also serves as a consensus node of block consensus in the block chain system. Due to the uncertainty of the task and the dynamic nature of the network, the task is converted into a Markov decision process, and the optimal joint solution is dynamically learned by using near-end strategy optimization.

Description

Technical Field

[0001] The present invention belongs to the field of wireless communication technology, and in particular, relates to an optimization method and system in a multi-access edge computing system with an IPBFT consensus, and specifically to security task offloading and computing resource allocation. Background Art

[0002] Currently, the popularity of the Internet of Things (IoT) has led to a surge in user equipment (UE) that affects our daily lives. However, traditional centralized architectures, especially in cloud computing, have difficulty meeting the mobility and rapid response requirements of these terminals. This challenge has stimulated the emergence of multi-access edge computing (MEC). MEC is particularly suitable for real-time applications, enabling rapid response and localized decision-making. The scalability of cloud services combined with the low latency of edge computing creates a powerful and efficient computing ecosystem, providing a comprehensive solution that meets the diverse needs of different industries. In edge computing systems, tasks are typically processed on edge servers or cloud servers, which are collectively referred to as Compute Power Providers (CPP).

[0003] In the above scenarios, consortium blockchain systems are expected to significantly improve the reliability of collaboration. Task processing, resource utilization, and payment information are recorded as transactions, each of which is digitally signed by the service provider. Each blockchain node retains a copy of all transactions as immutable evidence of service provision and payment to facilitate the resolution of potential disputes, thereby establishing a robust and trusted environment for decentralized data processing in edge computing systems. The operator alliance deploys smart contracts to specify responsibilities, reward and punishment criteria, and triggers, etc., to ensure automatic execution to prevent disputes over contract performance. Integrating blockchain and MEC networks in a unified system can help create a task processing service environment with trustworthy, transparent, secure, immutable, and automated characteristics. However, there are still some challenges and problems when applying deep reinforcement learning (DRL) to MEC systems with IPBFT consensus:

[0004] 1) Network architecture integration issues

[0005] First, it is necessary to design a multi-system architecture, including edge computing systems and blockchain systems, to ensure that data and tasks between systems can be efficiently transmitted and processed. Secondly, the integration of consensus mechanisms and computing resources is also a key issue. The Practical Byzantine Fault Tolerance (PBFT) consensus algorithm requires frequent communication and information exchange between nodes, which consumes a lot of network bandwidth and computing resources. Therefore, it is necessary to optimize the consensus algorithm to reduce resource consumption and improve efficiency.

[0006] In particular, PBFT faces the following challenges: (1) The number of consensus nodes is fixed, and all blockchain nodes participate in the consensus process, resulting in low consensus efficiency; (2) Due to the single-point full node and two full-node broadcasts, PBFT brings high bandwidth overhead, limiting its application in large-scale networks; (3) There is a risk of selecting malicious or faulty nodes when arbitrarily selecting the master node, causing consensus failure. Although changing the protocol solves this problem, its high communication complexity will lead to resource waste, extended latency, and reduced system stability; (4) PBFT encounters difficulties in dynamic scenarios and lacks an effective mechanism for seamless transition during node changes. Restarting the entire network is costly, and frequent replacements will seriously affect the availability of the system.

[0007] 2) Real-time and latency requirements

[0008] In the MEC system based on IPBFT consensus, real-time and latency requirements are crucial. MEC systems are designed to provide low-latency, high-real-time services, so they have extremely high requirements for the speed of data processing and task response. However, the IPBFT consensus mechanism involves complex inter-node communication and multiple message delivery, which increases network latency. This poses severe challenges to application scenarios that require instant feedback and processing, such as autonomous driving, real-time monitoring, and augmented reality. Therefore, optimizing the consensus algorithm to reduce communication overhead, improve the efficiency of consensus, and design efficient data transmission and processing mechanisms are the key to meeting real-time and latency requirements. Only under the premise of ensuring real-time response and minimal latency of the system can the advantages of MEC systems in various applications be fully utilized.

[0009] 3) Trust and security issues

[0010] Due to the distributed and heterogeneous nature of the MEC system, a reliable trust relationship needs to be established between nodes to ensure the security of data transmission and processing. The IPBFT consensus mechanism improves the reliability of the system through fault-tolerant design, but it also faces the risk of selecting malicious nodes or faulty nodes, which may lead to consensus failure or system attacks. To this end, it is necessary to combine encryption technology, access control, and node reputation evaluation mechanisms to ensure that only trusted nodes can participate in the consensus process. At the same time, efficient security protocols are designed to prevent data tampering, unauthorized access, and network attacks to ensure the overall security and stability of the system. Only when trust and security issues are effectively resolved can the MEC system provide reliable computing and services to meet the high security requirements of users and applications.

[0011] 4) Environmental dynamics

[0012] In the MEC system based on IPBFT consensus, due to the randomness of task status and network status, and the tight coupling of various variables under different constraints, the system must have a high degree of flexibility and adaptability. The reputation value of the node is also constantly changing, which requires the system to dynamically adjust the task allocation and resource scheduling strategy to ensure the continuity and stability of the service. Therefore, designing algorithms and mechanisms that can cope with this high degree of dynamism, such as real-time load balancing, dynamic resource management, and flexible consensus protocols, is the key to ensure the efficient operation of the MEC system in a dynamic environment. These mechanisms need to be able to respond quickly to environmental changes, maintain the stability and efficiency of the system, and maximize the use of available resources to meet the needs of various applications.

[0013] Through the above analysis, the problems and defects of the prior art are as follows:

[0014] (1) Insufficient network architecture integration: Existing technologies still have deficiencies in designing multi-system architectures, making it difficult to efficiently integrate edge computing systems and blockchain systems, which limits the transmission and processing efficiency of data and tasks.

[0015] (2) Timeliness and latency requirements are difficult to meet: The existing IPBFT consensus mechanism involves complex inter-node communication and multiple message transmissions, which causes latency problems in application scenarios that require low latency and high real-time performance (such as autonomous driving and real-time monitoring), affecting the performance and application effect of the MEC system.

[0016] (3) Trust and security issues: The IPBFT consensus mechanism faces the risk of malicious nodes or faulty nodes being selected, which may lead to consensus failure or threaten system security. Existing technologies have not yet formed an effective solution in terms of node reputation evaluation and trust establishment mechanism.

[0017] (4) Insufficient response to environmental dynamics: Existing technologies are unable to effectively cope with the high dynamics of MEC systems. The systems lack sufficient flexibility and adaptability when dealing with random changes in task status and network status. Summary of the invention

[0018] In view of the problems existing in the prior art, the present invention provides an optimization method and system for a multi-access edge computing system of IPBFT consensus.

[0019] The present invention is implemented as follows: an optimization method in a multi-access edge computing system of IPBFT consensus, the method applies the strategy actor and critic network of the intelligent agent, and adopts the deep reinforcement learning DRL method to make decisions, including:

[0020] Initialize the parameters of the policy actor network and the critic network, and set the training-related hyperparameters; the agent interacts with the environment based on the current strategy, performs actions and performs state transitions; samples the entire segment in the environment using the parameters of the sampling policy actor network, and stores the trajectory in memory; calculates the discounted reward, advantage function, and objective function; updates the parameters of the policy actor network and the critic network as well as the sampling policy actor and critic network parameters; repeats the training until the strategy converges, and uses the trained strategy for computational offloading and resource allocation.

[0021] Further, the following steps are included:

[0022] S101, initialize the parameters θ and ω of the agent's strategy actor network and critic network, sample the parameters θ′ and ω′ of the strategy actor network and critic network, the number of episodes, initialize the learning rates μ and σ corresponding to the strategy actor network and critic network, the discount factor γ, initialize the experience pool; initialize the network layout parameters, such as the data size D of the input task i (t), task workload C i (t) and other parameters;

[0023] S102, initializing the state of the agent, the agent interacts with the environment, and the policy network generates actions based on the current policy;

[0024] S103, the agent executes the generated action, obtains an immediate reward according to the executed action, and transfers the environment state to the next state;

[0025] S104, sampling the entire segment in the environment according to the parameters of the sampling strategy actor network, and storing the trajectory in the memory;

[0026] S105, calculating the discount reward;

[0027] S106, calculating the advantage function, adding the clipping factor to constrain the update rate, and calculating the objective function at the same time;

[0028] S107, update the parameters of the strategy actor and critic network;

[0029] S108, updating the sampling strategy actor and critic network parameters according to the updated strategy actor and critic network parameters;

[0030] S109, repeat iterative training, select the optimal action according to each state, obtain the maximum benefit, and finally obtain the optimal computing resource allocation and computing offloading strategy.

[0031] Further, in S102, the agent interacts with the environment; at the beginning of each round, the system state s(t) is initialized; wherein the state includes three parts, namely, the state s of the task task (t), the state of the network s net (t) and the state information s in the consensus con (t), as follows:

[0032] s(t)={s task (t),s net (t),s con (t)}.

[0033] The status of the task task (t) is given by the following formula:

[0034] s task (t)={D(t),C(t)},

[0035] Among them, the input data volume of the task The workload of the task

[0036] Network Status net (t) is given by:

[0037] s net (t) = {A(t), R(t), U(t), r m (t),f m (t)},

[0038] Among them, the link availability between UE and SBSs is The link connection rate between UE and SBSs is Number of computing resource blocks of SBSs The link connection rate between UE and MBS is r m (t); The computing resources allocated by MBS to each task are fm (t+1);

[0039] Status information during consensus process con (t) is given by the following formula:

[0040]

[0041] Where W(t-1)={W k (t)} is the credit value of the node at time slot t-1, Malicious Node Indicators The number of malicious nodes is When the master node is a malicious node, whether the master node k reaches a consensus indicator When the non-primary node is a malicious node, the indicator of whether the non-primary node k has reached consensus When the master node is a normal node, the malicious delay of master node k in the Pre-prepare stage is When the non-primary node is a normal node, the malicious delay of the non-primary node k in the Commit phase is The number of failures at the previous moment is N fail (t-1); the average consensus delay is

[0042] At each time slot t, the agent decides its action a(t) based on the state s(t) given by

[0043]

[0044] in, It is the task offloading decision of UE; Make CRB allocation decisions during task processing; Select decision indicators for consensus nodes, where Select the indicator for the master node, y k (t) Select decision indicators for non-primary nodes.

[0045] Furthermore, the calculation consensus of the reward in S103: the agent executes the generated action, obtains an immediate reward according to the executed action, and transfers the environment state to the next state is as follows:

[0046]

[0047] In the above formula, Object(t) represents our objective function.

[0048] Further, the S106: calculates the advantage function:

[0049]

[0050] Among them, at this time there are:

[0051] δ t =r(t)+γV(s t+1 ; w)-V(s t ;w).

[0052] γ is the discount factor. PPO introduces a θ′ A further improvement of the actor objective function of (θ) is to constrain the update rate by adding a clipping factor and update the PPO actor by maximizing the objective function, as follows:

[0053]

[0054] Where ∈ is a hyperparameter, the clip function converts (π(a t ∣s t ;θ)) / π(a t ∣s t ; The value of θ′) is restricted to the range of [1-,1+]; ensuring that the two distributions remain relatively close after minimizing the clip function.

[0055] Further, the S107: update the parameters of the strategy actor and the critic network; the parameters of the actor are updated by the following formula:

[0056]

[0057] Considering the mean square error function of the value estimation, the loss function of the critic network is given: L critic (w)=[V(s t+1 ; w)-V(s t ;w)] 2 ,

[0058] Updated by this formula:

[0059]

[0060] Among them, δ t Indicates TD error.

[0061] Another object of the present invention is to provide a multi-access edge computing system based on IPBFT consensus, the system comprising:

[0062] System initialization module, which initializes all necessary parameters, including initializing experience memory, policy actor network parameters θ and critic network parameters w, sampling policy actor parameters θ' and critic network parameters w';

[0063] Configuration module, used to set application-specific parameters, including task parameters such as the data size of the input task;

[0064] The agent module is used to generate actions based on the current network state at the beginning of each cycle; it is used to collect data samples, calculate the advantage function, and update the policy and value network during the policy evaluation process;

[0065] Action execution module, which is used to perform task offloading, computing resource allocation in task processing, selection of master nodes and non-master nodes in the consensus process;

[0066] The reward acquisition module is used to execute actions and calculate instant rewards. The reward acquisition module is designed based on whether the constraints of the system are met. If all constraints are met, rewards are obtained, otherwise penalties are obtained;

[0067] A state transfer module is used to transfer the system state from the current state to the next state;

[0068] The experience replay module is used to store the experience tuples of each system state, action, reward, and next state;

[0069] The data sampling module is used to extract certain fragments from the stored experience for learning;

[0070] The network update module is used to update the actor network and the critic network based on the data of the experience playback module;

[0071] Parameter update module, used for parameter update of policy actor network and critic network, as well as parameter update of sampling policy actor and critic network.

[0072] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of an optimization method in a multi-access edge computing system based on IPBFT consensus.

[0073] Another object of the present invention is to provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of an optimization method in a multi-access edge computing system based on IPBFT consensus.

[0074] In combination with the above technical solutions and the technical problems solved, please analyze the advantages and positive effects of the technical solutions to be protected by the present invention from the following aspects:

[0075] First, in view of the technical problems existing in the above-mentioned prior art and the difficulty of solving the problems, the technical solutions to be protected by the present invention and the results and data during the research and development process are closely combined to analyze in detail and deeply how the technical solutions of the present invention solve the technical problems, and some creative technical effects brought about after solving the problems. The specific description is as follows:

[0076] The first is efficiency and security. The present invention considers a hybrid cloud and edge computing network supporting blockchain, which includes two systems: an MEC system and a blockchain system. The MEC system is designed for task execution, while the blockchain system plays a vital role in establishing a secure and trusted transaction platform for the MEC system. In the MEC system, the Macro Base Station (MBS) is positioned at a considerable distance, providing a wide coverage area to ensure that all UEs can obtain uninterrupted wireless communication services. Each Small Base Station (SBSs) demonstrates proficiency in covering specific UEs, thereby establishing a localized coverage area. SBSs achieve seamless connectivity through high-speed wired connections. Integrated with the cloud server, the MBS enhances its processing capabilities, ensuring powerful processing capabilities for the terminal.

[0077] Secondly, considering the dynamics of the task, the volatility of network conditions and the random behavior of nodes in consensus, we use PPO to learn the environment state and obtain the optimal joint decision-making strategy for offloading, computing resource allocation and consensus committee selection.

[0078] Second, considering the technical solution as a whole or from the perspective of the product, the technical effects and advantages of the technical solution to be protected by the present invention are described in detail as follows:

[0079] The present invention implements a method for optimizing secure task offloading and computing resource allocation in a multi-access edge computing system supported by a consortium blockchain. The method has made significant progress in the following key aspects:

[0080] 1) Accurate task offloading and computing resource allocation

[0081] This method significantly improves the resource utilization and system performance of edge computing by introducing precise task offloading and resource allocation strategies. The MEC system provides computing power for user devices to process tasks, while the consortium blockchain system provides trust and security for user devices during the task offloading process.

[0082] 2) Efficient consensus performance

[0083] This method uses Delegated Proof of Stake (DPoS) and integrates it into PBFT for block consensus. The improved method is the Practical Byzantine Fault Tolerance algorithm. This integrated IPBFT method takes into account both scalability and complexity, solves the limitations of the PBFT algorithm, and significantly improves consensus performance.

[0084] 3) Minimum UE task processing cost

[0085] Task processing will bring economic costs, so this method carefully designs the reward mechanism to minimize the cost of UE processing tasks.

[0086] 4) Minimize consensus delays and failure rates

[0087] Consensus delay refers to the time required to complete consensus, and consensus failure may be caused by many reasons, such as unsatisfied constraints, malicious master nodes, etc. This method minimizes consensus delay and failure rate by optimizing unloading strategies, etc., to improve the stability and efficiency of the system.

[0088] 5) Integration of Deep Reinforcement Learning

[0089] Combining deep reinforcement learning algorithms with edge computing systems enables the system to self-learn and adjust based on real-time data. By adopting the PPO algorithm, the system can still make the best decision in the absence of clear instructions.

[0090] 6) Improvement of system stability and reliability

[0091] By accurately calculating and optimizing the immediate rewards and state transitions after action execution, this method significantly enhances the stability and reliability of the system, especially when processing large amounts of data and high concurrent requests.

[0092] 7) The network’s autonomous learning and optimization capabilities

[0093] This method enables the system to continuously optimize its decision-making process through continuous iterative training and experience-based network updates, thereby improving overall performance.

[0094] 8) Joint Optimization Problem

[0095] This method jointly optimizes task offloading decisions, computing resource allocation, and master-slave node selection to improve the overall performance and efficiency of the system. This comprehensive optimization strategy not only improves resource utilization, but also enhances the system's adaptability and stability in complex network environments.

[0096] 9) Complexity and dynamism

[0097] The complexity of the network and the dynamic nature of the environment make it difficult for traditional optimization methods to effectively deal with these challenges. To address this problem, this method uses the PPO algorithm to improve the adaptability and performance of the system when facing complex and dynamic environments.

[0098] 10) Real-time and efficiency

[0099] This method can improve the real-time response speed and resource utilization efficiency of the system when processing tasks.

[0100] Third, the optimization method in the MEC system based on IPBFT consensus provided by the present invention is based on the use of mathematical models to guide the behavior and learning process of the system. The technical effects brought by these mathematical models can be discussed based on their characteristics:

[0101] 1) Even if the reward is calculated

[0102] Even the computational consensus of rewards focuses on the overall network cost, consensus delays and long failure rates.

[0103] Cost savings: By directly linking rewards to costs, this approach encourages reducing costs across the network and effectively improves resource utilization efficiency.

[0104] 2) Update of the policy master actor and critic network

[0105] Update the current policy network by randomly sampling small batches of experience data and using the gradient ascent method.

[0106] Policy optimization: By continuously adjusting the policy network parameters, the system can learn and adopt more effective decision-making strategies.

[0107] Improved responsiveness: Using small batches of data enables the network to quickly adapt to environmental changes, enhancing the system's dynamic adjustment capabilities.

[0108] 3) Parameter update formula

[0109] The method of updating the target network parameters using the policy actor and critic network parameters is described, involving the current network and the target network parameters.

[0110] Policy gradually approaches: By introducing the clipping function, the system can smoothly transition to the new policy and prevent performance fluctuations due to drastic changes.

[0111] Continuous learning and adaptation: This continuous parameter update mechanism ensures that the system can adapt to long-term environmental changes.

[0112] Fourth, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the following important aspects:

[0113] (1) The expected benefits and commercial value of the technical solution of the present invention after transformation are:

[0114] The present invention realizes the optimization method in the MEC system based on IPBFT consensus by introducing the PPO algorithm, which significantly improves the network performance and service quality. This can not only meet the growing demand for high-performance computing, but also optimize resource utilization and reduce network costs. In terms of commercial applications, the present invention can be widely used in application scenarios that require efficient processing of large amounts of data and ensure data security and trustworthiness, bringing significant economic and social benefits to related industries. Therefore, the present invention has broad market prospects and huge commercial value. BRIEF DESCRIPTION OF THE DRAWINGS

[0115] Figure 1 It is a flow chart of an optimization method in a MEC system based on IPBFT consensus provided by an embodiment of the present invention;

[0116] Figure 2 is an applicable scene graph provided by an embodiment of the present invention;

[0117] Figure 3 It is a flowchart for implementing the optimization method in the MEC system based on IPBFT consensus provided by an embodiment of the present invention;

[0118] Figure 4 It is a comprehensive convergence performance comparison diagram provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0119] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0120] 1. Explanatory Examples In order to enable those skilled in the art to fully understand how to implement the present invention, this section provides an illustrative example that expands and describes the technical solution of the claims.

[0121] Example 1

[0122] In view of the problems existing in the existing technology, this implementation provides an optimization method in the MEC system based on IPBFT consensus. Figure 2 It is a scenario diagram to which the method of the present invention can be applied. First, it includes two systems: a multi-access edge computing system and a blockchain system. The MEC system is designed for task execution, while the blockchain system plays a vital role in establishing a secure and reliable transaction platform for the MEC system.

[0123] In the MEC system, there are I user equipments, K small base stations and 1 macro base station, where the sets of UEs and SBSs are denoted as and MBS is positioned at a considerable distance, providing a wide coverage area, ensuring that all UEs can obtain uninterrupted wireless communication services. Each SBS demonstrates proficiency in covering a specific UE, thereby establishing a localized coverage area. SBSs achieve seamless connectivity through high-speed wired connections. Integrated with cloud servers, MBS enhances its processing capabilities, ensuring powerful processing capabilities for terminals.

[0124] The system of the present invention works in time slots, where the time range is divided into T time slots, the set of which is represented by The length of each time slot is △t. At the beginning of each time slot, each UE generates a computationally intensive task to be processed within the time slot. The computing power of each UE is limited and may not be enough to process the task while meeting the quality of service constraints. However, the task can be transferred to the edge server of the SBS or the cloud server of the MBS through wireless communication, thus resorting to MEC. In the following, MEC servers and cloud servers are referred to as computing capacity providers, and all CPPs are managed by the same operator. After the CPP processes a task, the operator will charge the UE a certain fee for the use of communication and computing resources in task processing.

[0125] After the task processing is completed, CPPs package the information of the computing task, the processing results, and the corresponding fees as transactions as evidence of the CPPs processing the task and as evidence of the UE paying for the corresponding task processing service. Then, each CPP uploads the transactions it generates to the blockchain system, where the transactions will first be verified and then packaged into a new block. Once the whole network reaches a consensus on the new block, it will be attached to the distributed ledger, that is, the blockchain of each node. Only when the block is successfully added to the blockchain will the task processing results be returned to the user and the CPPs will receive the corresponding service fees. Otherwise, the task processing will fail and the consensus node will be punished. When a block is successfully added to the blockchain, the transactions it contains will be irreversible, so that the privacy and security of the user can be effectively guaranteed. A controller is deployed in the MBS to collect and maintain lists such as terminal information, edge server details, service details, and application information to optimize the network configuration on demand. The controller receives and trains the data through the DRL algorithm, enabling it to make intelligent joint optimization decisions for the MEC and blockchain systems.

[0126] Example 2

[0127] Considering the dynamic characteristics of the network and the uncertainty of information acquisition, this makes the problem quite complicated. In order to effectively solve this challenge, we remodel the problem as a Markov decision process (MDP). Since the variables involved are non-continuous, we choose to use the PPO algorithm, which is a DRL algorithm designed to solve reinforcement learning problems in continuous action spaces and can support real-time online decision making.

[0128] like Figure 1 As shown, the embodiment of the present invention provides an optimization method in a MEC system based on IPBFT consensus, which specifically includes the following steps:

[0129] S101, initialize the parameters θ and ω of the agent's policy actor network and critic network, sample the parameters θ′ and ω′ of the policy actor network and critic network, the number of episodes, initialize the learning rates μ and σ corresponding to the critic network and policy network, the discount factor γ, initialize the experience pool; initialize the network layout parameters, such as the data size D of the input task i (t), task workload C i (t) and other parameters;

[0130] S102, initializing the state of the agent, the agent interacts with the environment, and the main strategy network generates actions based on the current strategy;

[0131] S103, the agent executes the generated action, obtains an immediate reward according to the executed action, and transfers the environment state to the next state;

[0132] S104, sampling the entire segment in the environment according to the parameters of the sampling strategy actor network, and storing the trajectory in the memory;

[0133] S105, calculating the discount reward;

[0134] S106, calculating the advantage function, adding the clipping factor to constrain the update rate, and calculating the objective function at the same time;

[0135] S107, update the parameters of the strategy actor and critic network;

[0136] S108, updating the sampling strategy actor and critic network parameters according to the updated strategy actor and critic network parameters;

[0137] S109, repeat iterative training, select the optimal action according to each state, obtain the maximum benefit, and finally obtain the optimal computing resource allocation and computing offloading strategy.

[0138] Furthermore, in S102, the agent interacts with the environment; at the beginning of each round, the system state s(t) is initialized. The state includes three parts, namely, the state s of the task task (t), the state of the network s net (t) and the state information s in the consensus con (t), as follows:

[0139] s(t)={s task (t),s net (t),s con (t)}.

[0140] The status of the task task (t) is given by the following formula:

[0141] s task (t)={D(t),C(t)},

[0142] The input data volume of the task D(t) = {D i The workload of the task C(t) = {C i (t)},i∈I.

[0143] Network Status net (t) is given by:

[0144] s net (t) = {A(t), R(t), U(t), r m (t),f m (t)},

[0145] Among them, the link availability between UE and SBSs is The link connection rate between UE and SBSs is Number of computing resource blocks of SBSs The link connection rate between UE and MBS is r m (t); The computing resources allocated by MBS to each task are f m (t+1).

[0146] Status information during consensus process con (t) is given by the following formula:

[0147]

[0148] Where W(t-1)={W k (t)} is the credit value of the node at time slot t-1, Malicious Node Indicators The number of malicious nodes is When the master node is a malicious node, whether the master node k reaches a consensus indicator When the non-primary node is a malicious node, the indicator of whether the non-primary node k has reached consensus When the master node is a normal node, the malicious delay of master node k in the Pre-prepare stage is When the non-primary node is a normal node, the malicious delay of the non-primary node k in the Commit phase is The number of failures at the previous moment is N fail (t-1); the average consensus delay is

[0149] At each time slot t, the agent decides its action a(t) based on the state s(t) given by

[0150]

[0151] in, It is the task offloading decision of UE; Make CRB allocation decisions during task processing; Select decision indicators for consensus nodes, where For master node selection, y k (t) Select decision indicators for non-primary nodes.

[0152] Furthermore, the consensus for calculating the reward in S103: the agent executes the generated action, obtains an immediate reward according to the executed action, and transfers the environment state to the next state is as follows:

[0153]

[0154] In the above formula, Object(t) represents our objective function.

[0155] Furthermore, the S106: calculates the advantage function:

[0156]

[0157] At this time there are:

[0158] δ t =r(t)+γV(s t+1 ; w)-V(s t ;w).

[0159] In order to improve the performance, PPO introduces a J-based θ′ Further improvement of the actor objective function of (θ). By adding a clipping factor to constrain the update rate, the PPO actor can be updated by maximizing the objective function, as follows:

[0160]

[0161] Where ∈ is a hyperparameter, the clip function converts (π(a t ∣s t ;θ)) / π(a t ∣s t ; θ′) is restricted to the range [1-,1+]. This approach ensures that after minimizing the clip function, the two distributions remain relatively close and avoid significant differences.

[0162] Furthermore, the S107: updating the parameters of the strategy actor and the critic network; the parameters of the actor are updated by the following formula:

[0163]

[0164] This algorithm considers the mean square error function of the value estimation and gives the loss function of the critic network:

[0165] L critic (w)=[V(s t+1 ; w)-V(s t ;w)] 2 ,

[0166] At the same time, it can be updated by this formula:

[0167]

[0168] Among them, δ t Indicates TD error.

[0169] In order to elaborate on the optimization method in the MEC system based on IPBFT consensus, the present invention provides two specific application embodiments, including key details of the implementation scheme.

[0170] Application Example 1: Smart City IoT Application

[0171] 1) System initialization

[0172] Deploy the policy actor and critic network in the smart city monitoring center. Initialize the network parameters, including the number of rounds, training steps, learning rate, etc. Deploy multiple IoT devices (such as edge computing nodes, sensors) and base stations to collect and transmit data.

[0173] 2) Agent-environment interaction

[0174] Each IoT device acts as an intelligent agent and generates actions based on the state of its monitored environment (such as adjusting the sensor's acquisition frequency, the edge node's data processing priority, or the frequency of uploading data, etc.). The generation of actions is based on the current policy.

[0175] 3) Action execution and reward acquisition

[0176] After performing an action, the agent receives an immediate reward based on the effect of the action (e.g., data transfer rate, data accuracy). The system then moves to the next state, which may be due to changes in the environment or changes in the data stream at different time periods.

[0177] 4) Optimization and update

[0178] Based on the collected data and rewards, the policy actor and critic network are updated to optimize the control strategy of the sensor and the data processing priority of the edge node. The resource allocation, such as data transmission rate and allocation of computing resources, is gradually improved using reinforcement learning algorithms.

[0179] 5) Energy consumption and performance optimization

[0180] The performance of the PPO algorithm is evaluated through simulation experiments or field tests. Key indicators such as the coverage, signal quality, and data transmission rate of the communication network are analyzed and compared with traditional methods. According to the evaluation results, the algorithm parameters are adjusted and optimized to further improve the performance and stability of the communication network. Through the above steps, the MEC network joint optimization method based on IPBF consensus proposed in the present invention can monitor a large number of IoT devices in real time in smart cities, such as smart lighting control, environmental monitoring, etc.

[0181] Application Example 2: Financial and Payment Systems

[0182] 1) System initialization

[0183] In the financial and payment systems, the center deploys the PPO network and initializes the network parameters, and also deploys the necessary infrastructure such as servers and databases. These facilities not only support the real-time processing and data storage of payment transactions, but also ensure that the system can operate safely and efficiently, meeting the security, stability and performance requirements of financial services.

[0184] 2) Agent-environment interaction

[0185] Each server acts as an intelligent agent, continuously interacting with the environment and generating corresponding actions based on the current state of the environment. These actions may include resource allocation, data processing optimization, and security control to ensure the stability and performance optimization of financial and payment systems.

[0186] 3) Action execution and reward acquisition

[0187] After executing the current action, immediate rewards are obtained based on key indicators such as rationality of resource allocation, data transmission rate, and security. The system then moves to the next state based on changes in the environment or time periods.

[0188] 4) Optimization and update

[0189] The collected data and rewards are used to update the parameters of the PPO network. Through continuous iteration and training, the algorithm gradually improves the stability and efficiency of the communication system, thereby ensuring the security and reliability of the payment system.

[0190] 5) Payment system reliability

[0191] Through the PPO algorithm of the present invention, intelligent management and optimization of financial and payment systems are achieved. The algorithm focuses on improving the reliability and stability of communication links and ensuring the security and integrity of data during transmission. In addition, the PPO algorithm also optimizes the allocation and utilization efficiency of system resources, significantly improves the response speed and processing capacity of the overall system, thereby enhancing user experience and system reliability. Through the above steps, the present invention provides an efficient and intelligent solution for financial and payment system security.

[0192] In these two embodiments, the optimization method in the MEC system based on IPBFT consensus provides an efficient, reliable and energy-saving solution, which is suitable for different application scenarios, from smart cities to financial and payment systems, demonstrating its wide application potential and technical advantages.

[0193] In order to comprehensively evaluate the performance of the present invention, we compared it with several representative benchmark algorithms. These benchmark algorithms have their own characteristics and advantages and can effectively handle similar problems. By comparing the performance of these algorithms under the same conditions, we can more clearly understand the advantages of the present invention and the potential room for improvement.

[0194] 1) Random offloading: The offloading decision of each UE is randomly determined. Each UE's task can be processed independently on itself, SBS or MBS. However, a task can only be processed at one location. The allocation of computing resources and the selection of blockchain committees are optimized.

[0195] 2) Random Committee: The blockchain committee nodes are randomly selected from all SBSs, including master nodes and non-master nodes. Task offloading decisions and computing resource allocation decisions are optimized.

[0196] 3) The present invention: It represents the proposed PPO-based joint task offloading, computational resource allocation, and blockchain committee selection algorithm.

[0197] exist Figure 4 In this paper, we show the convergence performance of all the above algorithms under the default parameter settings. The convergence performance is evaluated based on three performance indicators: convergence speed, convergence stability and reward value obtained. The results show that the rewards of the algorithm proposed in this paper tend to stabilize around the 100th round, and the reward return after that is about 5.2×10 4 The random unloading algorithm converges around the 170th round, with a reward of 2×10 4 The random committee algorithm converges around the 150th round, and its reward is 1.2×10 4 By carefully observing the three curves, it can be inferred that the algorithm proposed by the present invention has the fastest convergence speed and is superior to the two benchmark algorithms in terms of performance.

[0198] 3. Evidence of the effects of the embodiments. The embodiments of the present invention have achieved some positive effects during the development or use process, and indeed have great advantages over the prior art. The following content is described in conjunction with the data, charts, etc. of the test process.

[0199] Figure 4 In this paper, we provide a comprehensive performance evaluation of the proposed algorithm by comparing it with three baseline algorithms. The graph clearly shows that the return of the proposed algorithm stabilizes at about the 100th episode, after which its return fluctuates around 5.2×10^4. The PPO-based-committee-selection algorithm converges at about the 270th episode, and its return fluctuates around 4.2×10^4. Random-offloading converges at about the 170th episode, and its return fluctuates around 2×10^4. A careful look at the three curves shows that the proposed algorithm shows the fastest convergence speed and outperforms the two baseline algorithms in terms of performance.

[0200] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0201] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. An optimization method in a multi-access edge computing system with IPBFT consensus, characterized in that: The method applies the agent's policy actor and critic network and adopts the deep reinforcement learning (DRL) method to make decisions, including: Initialize the parameters of the policy actor network and the critic network, and set the training-related hyperparameters; the agent interacts with the environment based on the current strategy, performs actions and performs state transitions; samples the entire segment in the environment using the parameters of the sampling policy actor network, and stores the trajectory in memory; calculates the discounted reward, advantage function, and objective function; updates the parameters of the policy actor network and the critic network as well as the sampling policy actor and critic network parameters; repeats the training until the strategy converges, and uses the trained strategy for computational offloading and resource allocation.

2. The optimization method in the multi-access edge computing system based on IPBFT consensus as claimed in claim 1, characterized in that: The steps include: S101, initialize the parameters θ and ω of the agent's strategy actor network and critic network, sample the parameters θ′ and ω′ of the strategy actor network and critic network, the number of episodes, initialize the learning rates μ and σ corresponding to the strategy actor network and critic network, the discount factor γ, initialize the experience pool; initialize the network layout parameters, such as the data size D of the input task i (t), task workload C i (t) and other parameters; S102, initializing the state of the agent, the agent interacts with the environment, and the policy network generates actions based on the current policy; S103, the agent executes the generated action, obtains an immediate reward according to the executed action, and transfers the environment state to the next state; S104, sampling the entire segment in the environment according to the parameters of the sampling strategy actor network, and storing the trajectory in the memory; S105, calculating the discount reward; S106, calculating the advantage function, adding the clipping factor to constrain the update rate, and calculating the objective function at the same time; S107, update the parameters of the strategy actor and critic network; S108, updating the sampling strategy actor and critic network parameters according to the updated strategy actor and critic network parameters; S109, repeat iterative training, select the optimal action according to each state, obtain the maximum benefit, and finally obtain the optimal computing resource allocation and computing offloading strategy.

3. The optimization method in the multi-access edge computing system based on IPBFT consensus as claimed in claim 1, characterized in that: In S102, the agent interacts with the environment; at the beginning of each round, the system state s(t) is initialized; wherein the state includes three parts, namely, the state s of the task task (t), the state of the network s net (t) and the state information s in the consensus con (t), as follows: s(t)={s task (t),s net (t),s con (t)}. The status of the task task (t) is given by the following formula: s task (t)={D(t),C(t)}, Among them, the input data volume of the task The workload of the task Network Status net (t) is given by: s net (t)={A(t),R(t),U(t),r m (t),f m (t)}, Among them, the link availability between the user equipment UE and the small base stations (SBSs) is The link connection rate between UE and SBSs is Number of computing resource blocks of SBSs The link connection rate between UE and macro base station (MBS) is r m (t); The computing resources allocated by MBS to each task are f m (t+1); Status information during consensus process con (t) is given by the following formula: Where W(t-1)={W k (t)} is the credit value of the node at time slot t-1, Malicious Node Indicators The number of malicious nodes is When the master node is a malicious node, whether the master node k reaches a consensus indicator When the non-primary node is a malicious node, the indicator of whether the non-primary node k has reached consensus When the master node is a normal node, the malicious delay of master node k in the Pre-prepare stage is When the non-primary node is a normal node, the malicious delay of the non-primary node k in the Commit phase is The number of failures at the previous moment is N fail (t-1); the average consensus delay is At each time slot t, the agent decides its action a(t) based on the state s(t) given by in, It is the task offloading decision of UE; Make CRB allocation decisions during task processing; Select decision indicators for consensus nodes, where Select the indicator for the master node, y k (t) Select decision indicators for non-primary nodes.

4. The optimization method in the multi-access edge computing system based on IPBFT consensus as claimed in claim 1, characterized in that: S103: The agent executes the generated action, obtains an immediate reward according to the executed action, and transfers the environment state to the next state. The calculation consensus of the reward is as follows: In the above formula, Object(t) represents our objective function.

5. The optimization method in the multi-access edge computing system based on IPBFT consensus as claimed in claim 1, characterized in that: S106: Calculate the advantage function: At this time there are: δ t =r(t)+γV(s t+1 ;w)-V(s t ;w). PPO (proximal policy optimization) introduces a θ′ A further improvement of the actor objective function of (θ) is to constrain the update rate by adding a clipping factor and update the PPO actor by maximizing the objective function, as follows: Where ∈ is a hyperparameter, the clip function converts (π(a t ∣s t ;θ)) / π(a t ∣s t ; The value of θ′) is restricted to the range of [1-,1+]; ensuring that the two distributions remain relatively close after minimizing the clip function.

6. The optimization method in the multi-access edge computing system based on IPBFT consensus as claimed in claim 1, characterized in that: S107: Update the parameters of the strategy actor and critic network; the parameters of the actor are updated by the following formula: Considering the mean square error function of the value estimation, the loss function of the critic network is given: L critic (w)=[V(s t+1 ; w)-V(s t ;w)] 2 , Updated by this formula: Among them, δ t Indicates TD error.

7. A multi-access edge computing system based on IPBFT consensus according to the method of claim 1, characterized in that: The system comprises: System initialization module, which initializes all necessary parameters, including initializing experience memory, policy actor network parameters θ and critic network parameters w, sampling policy actor parameters θ' and critic network parameters w'; Configuration module, used to set application-specific parameters, including task parameters such as the data size of the input task; The agent module is used to generate actions based on the current network state at the beginning of each cycle; it is used to collect data samples, calculate the advantage function, and update the policy and value network during the policy evaluation process; Action execution module, which is used to perform task offloading, computing resource allocation in task processing, selection of master nodes and non-master nodes in the consensus process; The reward acquisition module is used to execute actions and calculate instant rewards. The reward acquisition module is designed based on whether the constraints of the system are met. If all constraints are met, rewards are obtained, otherwise penalties are obtained; A state transfer module is used to transfer the system state from the current state to the next state; The experience replay module is used to store the experience tuples of each system state, action, reward, and next state; The data sampling module is used to extract certain fragments from the stored experience for learning; The network update module is used to update the actor network and the critic network based on the data of the experience playback module; Parameter update module, used for parameter update of policy actor network and critic network, as well as parameter update of sampling policy actor and critic network.

8. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the optimization method in the multi-access edge computing system based on IPBFT consensus as claimed in claim 1.

9. A computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to perform the steps of the optimization method in a multi-access edge computing system based on IPBFT consensus as claimed in claim 1.

Images