Block chain and digital twin supported edge cache optimization method and system in D2D network

Abstract

The invention relates to the field of wireless communication, and discloses a method and a system for optimizing edge caching in a device-to-device network supported by a block chain and digital twinning. An objective function is maximized by jointly optimizing content placement and content acquisition policies, the objective function comprehensively considering content acquisition gain, network life cycle, and user switching cost. According to the method, the problems of short network life, long edge cache delay, high cost and the like in the wireless network are solved. In the invention, each user can obtain the content from other users or from the content server through the D2D link from the own cache buffer area. The digital twin monitors the network operation state by storing key data such as the position of each user, the estimated processing capacity and the residual energy. In order to ensure the security and credibility of the transaction between the users, the block chain technology is used to supervise the transaction, and the reputation value of the user is continuously updated. In order to solve the complex problem under the dynamic network, a near-end strategy optimization algorithm is adopted to solve the complex problem.

Description

Technical Field

[0001] The present invention belongs to the field of wireless communication technology, and in particular to an edge cache optimization method and system in a D2D network supported by blockchain and digital twins. Background Art

[0002] Currently, with the rapid development of wireless communications, existing mobile networks are facing a rapid increase in mobile data traffic. This unprecedented traffic growth has brought great challenges to traditional cellular networks due to their dependence on computing, storage, and communication resources. If user equipment (UE) directly accesses or obtains content from remote data centers such as cloud servers, long delays may occur. Mobile edge caching has become a promising technology that can not only reduce content access latency but also reduce network traffic load. In an edge caching system, content can be stored in base stations (BSs) and edge servers.

[0003] Device-to-Device (D2D) communication technology enables communication between UEs and has been used to reduce data transmission latency and edge caching technology. Digital twin is an emerging technology that connects the physical and digital domains and can be used for real-time prediction, optimization, monitoring, control and enhanced decision-making, and can be used to achieve real-time communication and sustainable computing. Blockchain is a distributed ledger technology that records and verifies transaction data through decentralization and immutability. The integration and application of digital twin and blockchain technology in D2D networks requires solving a series of challenges:

[0004] 1) Issues with data real-timeness and accuracy

[0005] Ensure that user data and network status information obtained from the digital twin system can be updated in real time and accurately reflect the actual situation. This includes real-time data collection and transmission, timely data processing and analysis, updating the digital twin model to reflect the latest status changes, and rapid response and adaptation to dynamic changes in network environment and user behavior. This ensures that the system can provide accurate and up-to-date information at any time, providing a reliable foundation for optimizing network performance and resource allocation.

[0006] 2) High-dimensional and dynamically changing data processing issues

[0007] Effectively process and analyze complex high-dimensional user data, and adapt to dynamic changes such as user location and device status. And the goal is usually defined as a long-term goal. In this case, the present invention needs to design a higher-performance algorithm to achieve target optimization with low complexity and good performance. At the same time, it is also necessary to design a flexible system architecture and real-time data processing pipeline that can quickly respond to dynamic changes such as user location changes and device status updates to ensure that network performance and user experience are always in the best state. Through these means, the system can maintain efficient and stable operation in a complex and changing environment.

[0008] 3) Optimal allocation of network resources

[0009] Optimize edge cache and network resource allocation strategies to improve network performance and user experience. This includes jointly optimizing content placement strategies and content acquisition strategies, and dynamically adjusting cache content based on changes in user demand, network load, and device status. Specifically, by predicting users' content access patterns, high-frequency access content can be cached in edge nodes in advance, thereby reducing data transmission delays and network congestion. In addition, it is necessary to consider the resource capabilities and energy consumption characteristics of different devices, and reasonably schedule computing tasks and data transmission paths to maximize resource utilization efficiency and user service quality. Through these optimization measures, the network system can provide a smoother and more satisfactory user experience while ensuring high performance and reliability.

[0010] 4) Network lifespan issues

[0011] Users constantly obtain content from edge nodes, whether from their own cache or from other users through D2D links, which consumes their energy. In order to extend the network life, it is necessary to design an efficient energy management strategy to balance energy consumption and performance. Specific measures include optimizing content acquisition strategies, reducing unnecessary transmission times, and giving priority to using nodes with sufficient energy for data exchange. Summary of the invention

[0012] In view of the problems existing in the prior art, the present invention provides an edge cache optimization method and system in a D2D network supported by blockchain and digital twins. The network system includes two subsystems: an edge cache subsystem and a blockchain subsystem. In the edge cache subsystem, the terminal can act as a cache node to store content locally. Therefore, each UE can obtain content from its own cache buffer, through a D2D link, or from other UEs from a content server. The blockchain subsystem manages the delay in the block consensus process and continuously updates the UE's reputation value, thereby ensuring the security and reliability of D2D data sharing.

[0013] The present invention is implemented as follows: an edge cache optimization method in a D2D network supported by blockchain and digital twins, which applies the actor and critic networks of an intelligent body and adopts a deep reinforcement learning method to make decisions; initializes the parameters of the policy actor network and the critic network, and sets the training-related hyperparameters; the intelligent body interacts with the environment based on the current strategy, executes actions and performs state transfer; performs truncation, calculates estimated returns and advantage functions; updates the actor network and the critic network; repeats training until a predefined life stop growth condition is reached, and uses the trained strategy for content placement and content acquisition.

[0014] Further, the following steps are included:

[0015] S101, initialize the agent's strategy actor network π(a|s; θ) and its parameters θ, and initialize the critic network V ω (s) and its parameters ω, experience pool M, maximum number of rounds T max ; Initialize the learning rates μ and σ of the actor network and critic network respectively; Initialize the network layout parameters, such as the number of users I, the number of contents F, etc.;

[0016] S102, the agent interacts with the environment and collects information about the current state s t information, such as user location, estimated processing capacity and remaining energy;

[0017] S103, based on the current state s t , using the old strategy network π(a t |s t θ old ) Get action a t , get content placement and acquisition strategies;

[0018] S104, the agent performs action a t , observe the reward r t , and obtain the state s at the next moment t+1 ;

[0019] S105, the interactive data <s t ,a t ,r t ,s t+1 >Store in the experience pool to form a trajectory sequence;

[0020] S106, when the trajectory reaches the truncation length T, use the critic network V ω (s) Evaluate the terminal state and get the estimated reward R t Then, the advantage function A(s) is calculated based on the trajectory sequence. t ,at );

[0021] S107. Randomly select a set of experience tuples from the experience pool. Each tuple includes state s t , action a t and the corresponding reward R t ;

[0022] S108. The selected experience tuple is used to train the critic network. During the training process, the parameters ω of the critic network are updated so that the loss function L critic (ω) minimum;

[0023] S109, randomly select a set of tuples from the experience pool for training the actor network;

[0024] S110, constructing the strategy objective function L clip (ω), use the gradient ascent method to update the parameters θ of the actor network so that the policy objective function L clip (ω) maximized;

[0025] S111, continue to iterate the above steps until the predefined life cycle growth stop condition is reached. In each iteration, the agent contacts the environment, collects trajectory data, and uses this data to adjust the parameters of the actor network, thereby continuously improving the strategy and improving the performance of the agent.

[0026] Furthermore, in S102, the agent interacts with the environment and initializes the system state s at the beginning of each cycle. t , specifically expressed as:

[0027]

[0028] in,

[0029] u(t-1)={u i (t-1)}, where u i (t-1) = {x i (t-1),y i (t-1)} is the coordinate of UE i at time slot t-1;

[0030] in Indicates whether UE i requests content f. Note that a UE can only request one content in time slot t;

[0031] in is the preference of UE i for content f at time slot t;

[0032] represents the estimated processing capability of UE i in time slot t;

[0033] represents the remaining energy of each UE at the beginning of time slot t. Specifically,

[0034] represents the fast fading coefficient of the D2D communication link between UE i and UE j at time slot t;

[0035] represents the distance between UE i and UE j at time slot t;

[0036] represents the Rayleigh fast fading coefficient between UE i and the base station at time slot t; represents the distance between UE i and the base station at time slot t;

[0037] Indicates whether there is a direct physical link between UE i and UE j at time slot t;

[0038] Indicates whether there is a direct physical link and social connection between UE i and UE j at time slot t;

[0039] represents the set of neighbor users of UE i at time slot t;

[0040] represents the number of neighbor users of UE i at time slot t;

[0041] represents the tanimoto coefficient between UE i and UE j at time slot t; represents the D2D sharing probability between UE i and UE j at time slot t;

[0042] ΔT vali (t) represents the block verification delay deviation;

[0043] represents the transaction size of UE i at time slot t;

[0044] represents the number of signature verification failures received from UE i at time slot t; represents the reputation value of UE i in time slot t-1;

[0045] represents the growth index of UE i’s lifetime at time slot t;

[0046] represents the temporary lifetime of each UE in time slot t, and its initial lifetime is 0;

[0047] In each time slot t, the agent determines its action, denoted as in: represents the content placement strategy; A(t) = {a i,j (t)}, Indicates the content acquisition strategy.

[0048] Furthermore, the consensus for calculating the reward in S103: the agent executes the generated action, obtains an immediate reward according to the executed action, and transfers the environment state to the next state is as follows:

[0049]

[0050] Furthermore, the S106: calculates the advantage function:

[0051] A(s t ,a t )=δ t +γδ t+1 +...+γ T-t+1 δ t+1 ,

[0052] Represents the advantage of a truncated trajectory of length T. The advantage function refers to the advantage or relative benefit obtained by an agent taking a certain action relative to other possible actions, where

[0053] δ t =r t +γV(s t+1 )-V(s t ),

[0054] Among them, V(s t ) is the output of the critic network. Based on this, A(s t ,a t ) can be rewritten as

[0055] A(s t ,a t )=R(t)-V(s t ),

[0056] Where R(t) is used to estimate the expected return at time step t.

[0057] Furthermore, in S108: updating the parameters of the critic network, the updating formula is as follows:

[0058]

[0059] in,

[0060]

[0061] Where N represents the number of mini-batches. Indicates the status of the nth sample in the mini-batch.

[0062] Furthermore, in S110: updating the parameters of the actor network, the updating formula is as follows:

[0063]

[0064] Among them, η is a hyperparameter, L clip (θ) is expressed as

[0065]

[0066] Another object of the present invention is to provide a system for edge cache optimization in a D2D network supported by blockchain and digital twins, comprising:

[0067] An agent module that configures the parameters used to initialize the policy actor network and the critic network and sets the training-related hyperparameters;

[0068] an interaction module configured to enable the agent to interact with the environment, perform actions and make state transitions;

[0069] an evaluation module, configured to truncate trajectories and compute estimated rewards and advantage functions;

[0070] An update module configured to update the actor network and the critic network until a predefined life stop growth condition is reached and use the trained policies for content placement and content acquisition.

[0071] Furthermore, the agent module further includes:

[0072] An initialization unit, which is configured to initialize the agent's policy actor network and its parameters, the critic network and its parameters, the experience pool, and the maximum number of rounds;

[0073] A network layout unit is configured to initialize network layout parameters, including the number of users and the number of contents.

[0074] Furthermore, the interaction module also includes:

[0075] a state collection unit configured to initialize the system state at the beginning of each cycle and collect information about the current state, including user location, estimated processing capacity, remaining energy, etc.;

[0076] a policy generation unit configured to generate content placement and acquisition policies based on a current state by using an old policy network to obtain actions;

[0077] A reward observation unit is configured to allow the agent to perform actions and observe rewards, obtain the state at the next moment, and store the interaction data into the experience pool to form a trajectory sequence.

[0078] Furthermore, the evaluation module further includes:

[0079] an evaluation unit configured to evaluate the terminal state using a critic network and calculate an advantage function when the trajectory reaches a cutoff length;

[0080] A training unit is configured to randomly select experience tuples from the experience pool, train the parameters of the critic network and the actor network, and update the parameters of the actor network using the gradient ascent method according to the policy objective function until a predefined life cycle stop growth condition is reached.

[0081] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0082] First, the present invention has important benefits in the integration of blockchain and D2D networks supported by digital twins, as follows:

[0083] The first is efficiency and security. The present invention considers a D2D-assisted cellular network that supports blockchain and edge caching, which includes two subsystems, namely an edge caching subsystem and a blockchain subsystem. The edge caching subsystem is used for content file caching and sharing, and it consists of a remote content server, a BS, and UEs randomly distributed within the coverage of the BS. The blockchain subsystem is used to build a secure and reliable data information sharing transaction platform, and the UE also serves as a blockchain node. The content acquisition transactions from the edge caching subsystem are mainly processed securely, thereby ensuring the traceability of content acquisition. Transactions related to cache and UE reputation updates are packaged into blocks, verified through a consensus mechanism process, and then recorded in the blockchain. The digital twin layer is integrated in the edge caching subsystem to store key data such as terminal location, terminal content requests, estimated processing capacity, remaining energy, and real-time monitoring of network operation status.

[0084] Secondly, considering the dynamic nature of tasks, the volatility of network conditions and the random behavior of content requests, the present invention adopts Proximal Policy Optimization (PPO) to learn the environment state and obtain the joint optimization decision of content placement and content acquisition.

[0085] Second, the significant technical progress achieved by the present invention is to realize the edge cache optimization method in the D2D network supported by blockchain and digital twins, which has made significant progress in the following key aspects:

[0086] 1) Accurate content placement and content acquisition strategy

[0087] This method significantly improves the hit rate and resource utilization of edge cache by introducing content placement and content acquisition strategies. Specifically, this method can select content with high content popularity and place it on the appropriate UE, thereby reducing transmission delay. At the same time, by optimizing the content acquisition strategy, it ensures that the UE can efficiently obtain the required content from the nearest and communicable cache node, further improving network performance and user experience.

[0088] 2) Intelligent access control strategy

[0089] By introducing advanced learning mechanisms, the system can intelligently adjust the UE's content control strategy to adapt to dynamically changing network conditions. This not only enhances network stability and user experience, but also effectively balances the load and avoids overloading of a single node.

[0090] 3) Integration of reinforcement learning

[0091] Combining the DRL algorithm with the edge computing system enables the system to self-learn and adapt based on real-time data, making optimal decisions without explicit instructions. This adaptive capability is critical to handling complex and dynamically changing edge computing environments, ensuring that the system can respond quickly and adjust strategies.

[0092] 4) Maximize gain

[0093] The reward mechanism of this method focuses on the content acquisition gain of UE, and maximizes the gain of users when acquiring content through optimization strategies. This not only improves user satisfaction, but also optimizes the utilization efficiency of network resources.

[0094] 5) Energy efficiency optimization

[0095] This method takes into account the energy consumption of the device and optimizes the content transmission path and cache location by designing an efficient energy management strategy, reducing unnecessary energy consumption, extending the network life and improving the overall performance.

[0096] 6) Enhanced multi-user collaboration:

[0097] By introducing a multi-user collaboration mechanism, the system allows users to share cache resources and content, further improving resource utilization and content acquisition efficiency. This collaboration mechanism can not only reduce network transmission delays, but also reduce overall energy consumption.

[0098] 7) Efficient data analysis and prediction

[0099] The system uses historical data and user behavior patterns to make predictions, thereby optimizing content acquisition and caching strategies. This helps identify popular content in advance, adjust cache locations in a timely manner, and avoid potential network congestion.

[0100] 8) Improvement of system stability and reliability

[0101] By accurately calculating and optimizing the immediate rewards and state transfer after the action is executed, this method significantly enhances the stability and reliability of the system. Especially when processing large amounts of data and high concurrent requests, the system can maintain efficient operation and ensure continuous improvement of service quality and user experience.

[0102] 9) The network’s autonomous learning and optimization capabilities

[0103] This method enables the system to continuously optimize its decision-making process through continuous iterative training and experience-based network updates. This autonomous learning and optimization capability not only improves overall performance, but also enables the system to dynamically adapt to environmental changes, ensuring the best service quality in various usage scenarios.

[0104] These technological advances work together to enable blockchain and digital twin-enabled D2D networks to not only improve performance, but also achieve significant improvements in energy efficiency, stability, and adaptability, which are critical to modern complex computing needs.

[0105] Third, the edge cache optimization method in the D2D network supported by blockchain and digital twins provided by the present invention is centered on using mathematical models to guide the behavior and learning process of the system. The technical effects brought by these mathematical models can be discussed based on their characteristics:

[0106] 1) Calculation of instant rewards

[0107] The calculation formula of the immediate reward focuses on the system's content acquisition gain, network lifetime, and switching cost.

[0108] Content acquisition gain: UE can obtain content from local cache, from other users through D2D links, or directly from content servers. Different content acquisition methods will bring different content acquisition gains, and the specific gain depends on the selected method.

[0109] Network life: Each UE is initially fully charged, and gradually consumes power during content caching and acquisition. Once the power of a UE is lower than the preset power threshold, its life will stop increasing, that is, the life of the UE is the network life. Therefore, even if the goal of the reward mechanism is to maximize the network life to ensure the overall performance and stability of the network.

[0110] Switching cost: In a dynamic mobile user environment, since the UE's location is not fixed, attention needs to be paid to the UE's reassociation. When reassociation occurs, the UE may experience service interruption and reduced communication quality. Mobile network operators need to update the UE's service profile to adapt to mobility, which incurs switching costs.

[0111] 2) Update of Actor and Critic Network

[0112] Update the current policy network by randomly sampling small batches of experience data and using the gradient ascent method.

[0113] Policy optimization: By continuously adjusting the parameters of the policy network, the system can learn and adopt more effective decision-making strategies. This process ensures that the network can choose the best action when faced with different situations, improving overall performance and efficiency.

[0114] Improved responsiveness: Using small batches of data for updates enables the network to quickly adapt to environmental changes and enhances the system’s ability to dynamically adjust. This approach not only speeds up learning, but also makes the system more flexible and reliable in responding to emergencies and environmental changes.

[0115] Improved stability: Through small batch updates and gradient ascent, fluctuations in the training process are reduced, ensuring the stability and convergence of the network. This can effectively prevent the instability caused by large batch data updates and ensure a smoother and more efficient training process.

[0116] 3) Parameter update formula

[0117] The target network method for updating the policy actor and critic network parameters is described, involving the parameters of the current network and the target network.

[0118] Smooth policy transition: By introducing the pruning function, the system can smoothly transition to the new policy and avoid performance fluctuations caused by drastic changes. This method maintains stability during the update process and gradually improves the decision quality.

[0119] Continuous learning and adaptation: This continuous parameter update mechanism ensures that the system can cope with long-term environmental changes. By constantly adjusting parameters, the system can continue to operate efficiently and optimize its own strategies in different situations and dynamic environments.

[0120] Fourth, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the expected benefits and commercial value after the technical solution is transformed into:

[0121] The present invention uses the PPO algorithm to optimize the edge cache in the D2D network supported by blockchain and digital twins, significantly improving network performance and service quality. This can not only effectively cope with the growing demand for user devices, but also optimize the cache and acquisition strategy of content, thereby reducing network operating costs. In terms of commercial applications, the present invention is suitable for a wide range of application scenarios that require efficient processing of big data and ensure data security and credibility, bringing significant economic and social benefits to related industries. Therefore, the present invention has broad prospects and huge commercial value in the market.

[0122] Fifth, the present invention solves multiple technical problems existing in the prior art in edge cache optimization in industrial applications. Traditional D2D (Device-to-Device) network edge caching methods usually face problems such as low resource management efficiency, slow response speed, and insufficient reliability during data transmission. These problems are particularly prominent in complex industrial environments supported by blockchain and digital twins, making it difficult to ensure the real-time and accuracy of data, thereby affecting the performance and stability of the overall system. In addition, traditional methods usually lack intelligent decision-making mechanisms and cannot effectively respond to dynamically changing network environments and user needs, limiting their widespread application in industrial applications.

[0123] By introducing the deep reinforcement learning (DRL) method, the present invention has achieved significant technical progress in industrial applications. First, the actor and critic network of intelligent agents can dynamically adapt to complex network environments and make intelligent decisions to optimize content placement and acquisition strategies. This intelligent optimization process not only improves the resource utilization of the system, but also greatly reduces the delay in content transmission, and significantly enhances the response speed and overall performance of edge cache. Especially in industrial scenarios, the ability to respond in real time and dynamically adjust cache strategies is of great significance to ensuring the continuity of production lines and the immediate processing of industrial data.

[0124] In addition, the present invention realizes efficient network resource management in industrial applications. By introducing blockchain and digital twin technology, the system can perform cache optimization under a decentralized framework to ensure the security and integrity of data transmission. At the same time, through the continuous interaction and learning between the intelligent agent and the environment, the system can continuously optimize its strategy to adapt to the changing needs in the industrial environment. Compared with the traditional static optimization method, the dynamic optimization mechanism of the present invention can better meet the real-time requirements in industrial scenarios and reduce the maintenance cost of the system.

[0125] Finally, the edge cache optimization method of the present invention has achieved significant technical progress in industrial applications, which not only improves the efficiency and stability of D2D networks, but also achieves high reliability and low latency in data transmission. This provides a better solution for data management and communication in industrial production processes, significantly improves the intelligence level and automation capabilities of industrial systems, and lays a solid foundation for the development of future smart manufacturing and industrial Internet of Things. BRIEF DESCRIPTION OF THE DRAWINGS

[0126] The following briefly introduces the drawings required for the technical solutions of the embodiments of the present invention for a clearer explanation. These drawings show some embodiments of the present invention and provide a method for ordinary technicians in the field to obtain other implementation schemes based on these examples without the need for creative work.

[0127] Figure 1 It is a flow chart of an edge cache optimization method in a D2D network supported by blockchain and digital twins provided by an embodiment of the present invention;

[0128] Figure 2 This is a scenario diagram for the application of the edge cache optimization method in a D2D network supported by blockchain and digital twins provided by an embodiment of the present invention.

[0129] Figure 3 This is a flowchart for implementing the edge cache optimization method in a D2D network supported by blockchain and digital twins provided in an embodiment of the present invention.

[0130] Figure 4 This is a comprehensive convergence performance comparison diagram of the edge cache optimization method in the D2D network supported by the blockchain and digital twin provided by the embodiments of the present invention. DETAILED DESCRIPTION

[0131] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined purpose, the specific implementation methods, structural features and corresponding effects of the present invention are described in detail below with reference to the accompanying drawings and embodiments.

[0132] The main parameters, algorithms and mathematical models of the present invention focus on solving the deficiencies of the prior art in D2D network edge cache optimization. The key parameters include the initial parameter settings of the agent's strategy actor network and critic network, learning rate, experience pool capacity, maximum number of rounds, etc. The precise setting of these parameters directly affects the learning efficiency and decision accuracy of the system. By optimizing these parameters, the system can better adapt to complex network environments and ensure stability and efficiency in different industrial application scenarios.

[0133] In terms of algorithms, the present invention adopts a deep reinforcement learning (DRL) algorithm to achieve dynamic optimization of content placement and acquisition strategies through continuous interaction and learning between the agent and the environment. The algorithm builds an actor-critic architecture to enable the agent to make optimal decisions under changing network conditions, especially in resource-limited industrial environments, and can effectively balance data transmission delays and cache efficiency. Compared with traditional rule-based or static optimization methods, the DRL algorithm enables the system to have adaptive and self-learning capabilities, significantly improving the robustness of the algorithm and decision-making quality.

[0134] The mathematical model introduced in this invention, especially the model for calculating the advantage function and reward mechanism, solves the problem of the difficulty in accurately evaluating and predicting system performance in the prior art. By accurately calculating indicators such as state transitions, rewards, and estimated returns, the system can better adjust policy parameters and improve the performance of the agent in practical applications. Such mathematical models not only improve the system's predictive ability, but also enhance the system's adaptability in complex industrial environments, enabling it to continuously optimize edge caching strategies under changing conditions.

[0135] Ultimately, the present invention has achieved significant technical progress in industrial applications, mainly reflected in the improvement of optimization accuracy, response speed and system reliability. By optimizing parameters, improving algorithms and applying mathematical models, the present invention effectively solves the problems of insufficient resource utilization, delayed decision-making and poor system stability in traditional methods. Its intelligent and adaptive edge cache optimization method provides more efficient and reliable solutions for the fields of industrial Internet of Things, smart manufacturing, etc., and significantly promotes the process of intelligence and automation of industrial systems. In response to the problems existing in the prior art, this implementation provides an edge cache optimization method in D2D networks supported by blockchain and digital twins, Figure 2 It is a scenario diagram to which the method of the present invention can be applied. It includes two subsystems, namely the edge cache subsystem and the blockchain subsystem. The edge cache subsystem is used for content file caching and sharing, and it consists of a remote content server, a BS, and UEs randomly distributed within the coverage of the BS. The blockchain subsystem is used to build a secure and reliable data information sharing and trading platform, and the UE also serves as a blockchain node.

[0136] The BS is connected to the content server via a wired optical fiber link, and all UEs can communicate directly with the BS via a wireless 4G or 5G link. The set of UEs is represented as Its index is denoted as i. In addition to radio stations near stadiums, similar situations also occur around concert venues. In these occasions, a large number of audiences gather together and use mobile devices to access music or video streams at the same time, which puts huge traffic pressure on the surrounding radio stations. In order to prevent network congestion caused by repeated downloading of content from the BS, the UE can act as a cache node, store the content locally, and share it with other UEs through D2D links. There are F contents in the content server. The content file is represented as The size of each file is s f In this framework, the content server stores all layers of all F files, and UEs that support caching can cache copies of popular content files. Assuming that the system of the present invention is executed in slots indexed by t, and the length of each slot is Δt, the set of slots is represented as

[0137] The blockchain subsystem mainly processes content sharing transactions from the edge cache subsystem securely, thereby ensuring the traceability of content sharing. Transactions related to cache and UE reputation updates are packaged into blocks, verified through a consensus mechanism process, and then recorded in the blockchain.

[0138] Due to the dynamic characteristics of the network and the uncertainty of information acquisition, the problem becomes extremely complicated. In order to effectively meet this challenge, the present invention uses Markov Decision Process (MDP) to remodel the problem. Considering that the variables involved are non-continuous, the present invention selects the PPO algorithm as a solution. PPO is a DRL algorithm designed to solve reinforcement learning problems in continuous action spaces, which can support real-time online decision-making.

[0139] like Figure 1 As shown, an embodiment of the present invention provides an edge cache optimization method in a D2D network supported by blockchain and digital twins, which specifically includes the following steps:

[0140] S101, initialize the agent's strategy actor network π(a|s; θ) and its parameters θ, and initialize the critic network V ω (s) and its parameters ω, experience pool M, maximum number of rounds T max ; Initialize the learning rates μ and σ of the actor network and critic network respectively; Initialize the network layout parameters, such as the number of users I, the number of contents F, etc.;

[0141] S102, the agent interacts with the environment and collects information about the current state s t information, such as user location, estimated processing capacity and remaining energy;

[0142] S103, based on the current state s t , using the old strategy network π(a t |s t θ old ) Get action a t , get content placement and acquisition strategies;

[0143] S104, the agent performs action a t , observe the reward r t , and obtain the state s at the next moment t+1 ;

[0144] S105, the interactive data <s t ,a t ,r t ,s t+1 >Store in the experience pool to form a trajectory sequence;

[0145] S106, when the trajectory reaches the truncation length T, use the critic network V ω (s) Evaluate the terminal state and get the estimated reward R t Then, the advantage function A(s) is calculated based on the trajectory sequence. t ,a t );

[0146] S107. Randomly select a set of experience tuples from the experience pool. Each tuple includes state s t , action a t and the corresponding reward R t ;

[0147] S108. The selected experience tuple is used to train the critic network. During the training process, the parameters ω of the critic network are updated so that the loss function L critic (ω) minimum;

[0148] S109, randomly select a set of tuples from the experience pool for training the actor network;

[0149] S110, constructing the strategy objective function L clip (ω), use the gradient ascent method to update the parameters θ of the actor network so that the policy objective function L clip (ω) maximized;

[0150] S111, continue to iterate the above steps until the predefined life cycle growth stop condition is reached. In each iteration, the agent contacts the environment, collects trajectory data, and uses this data to adjust the parameters of the actor network, thereby continuously improving the strategy and improving the performance of the agent.

[0151] Furthermore, in S102, the agent interacts with the environment and initializes the system state s at the beginning of each cycle. t , specifically expressed as:

[0152]

[0153] in,

[0154] u(t-1)={u i (t-1)}, where u i (t-1) = {x i (t-1),y i (t-1)} is the coordinate of UE i at time slot t-1;

[0155] in Indicates whether UE i requests content f. Note that a UE can only request one content in time slot t;

[0156] in is the preference of UE i for content f at time slot t;

[0157] represents the estimated processing capability of UE i in time slot t;

[0158] represents the remaining energy of each UE at the beginning of time slot t. Specifically,

[0159] represents the fast fading coefficient of the D2D communication link between UE i and UE j at time slot t;

[0160] represents the distance between UE i and UE j at time slot t;

[0161] represents the Rayleigh fast fading coefficient between UE i and the base station at time slot t; represents the distance between UE i and the base station at time slot t;

[0162] Indicates whether there is a direct physical link between UE i and UE j at time slot t;

[0163] Indicates whether there is a direct physical link and social connection between UE i and UE j at time slot t;

[0164] represents the set of neighbor users of UE i at time slot t;

[0165] represents the number of neighbor users of UE i at time slot t;

[0166] represents the tanimoto coefficient between UE i and UE j at time slot t; represents the D2D sharing probability between UE i and UE j at time slot t;

[0167] ΔT vali (t) represents the block verification delay deviation;

[0168] represents the transaction size of UE i at time slot t;

[0169] represents the number of signature verification failures received from UE i at time slot t;

[0170] represents the reputation value of UE i in time slot t-1;

[0171] represents the growth index of UE i’s lifetime at time slot t;

[0172] represents the temporary lifetime of each UE in time slot t, and its initial lifetime is 0;

[0173] In each time slot t, the agent determines its action, denoted as in: represents the content placement strategy; A(t) = {a i,j (t)}, Indicates the content acquisition strategy.

[0174] Furthermore, the consensus for calculating the reward in S103: the agent executes the generated action, obtains an immediate reward according to the executed action, and transfers the environment state to the next state is as follows:

[0175]

[0176] Furthermore, the S106: calculates the advantage function:

[0177] A(s t ,a t )=δ t +γδ t+1 +...+γ T-t+1 δ t+1 ,

[0178] Represents the advantage of a truncated trajectory of length T. The advantage function refers to the advantage or relative benefit obtained by an agent taking a certain action relative to other possible actions, where

[0179] δ t =r t +γV(s t+1 )-V(s t ),

[0180] Among them, V(s t ) is the output of the critic network. Based on this, A(s t ,a t ) can be rewritten as

[0181] A(s t ,a t )=R(t)-V(s t ),

[0182] Where R(t) is used to estimate the expected return at time step t.

[0183] Furthermore, in S108: updating the parameters of the critic network, the updating formula is as follows:

[0184]

[0185] in,

[0186]

[0187] Where N represents the number of mini-batches. Indicates the status of the nth sample in the mini-batch.

[0188] Furthermore, in S110: updating the parameters of the actor network, the updating formula is as follows:

[0189]

[0190] Among them, η is a hyperparameter, L clip (θ) is expressed as

[0191]

[0192] In order to elaborate on the blockchain-driven edge computing system joint optimization method, the present invention provides two specific application embodiments, including key details of the implementation scheme.

[0193] Application Example 1: Smart City and IoT Applications

[0194] 1) System initialization

[0195] Device registration: Various IoT devices in smart cities (such as sensors, cameras, smart lamps, etc.) are registered on the blockchain through the network.

[0196] Initialization data upload: The device uploads initialization data to the digital twin system based on its type and location, including device identification, location coordinates, initial state and other information.

[0197] Environmental modeling: Model various equipment, buildings, transportation systems and other entities in the city and synchronize them with the digital twin system to reflect the real environment status in real time.

[0198] Data integration: Integrate data from different devices and sensors to form a comprehensive digital twin model for predicting and simulating urban operation status.

[0199] 2) Agent-environment interaction

[0200] Intelligent agent perception: Obtain real-time data of surrounding devices and environment through D2D connection, including environmental parameters, device status, traffic conditions, etc.

[0201] Information collection: Obtain real-time updated urban data from the digital twin system, such as traffic flow, air quality, energy usage, etc.

[0202] 3) Action execution and decision making

[0203] The intelligent agent performs specific operations and decisions based on the collected information and the current environmental status, such as adjusting light brightness, optimizing traffic flow, and providing safety alerts.

[0204] Using the PPO algorithm, the agent selects the optimal action based on the current state to achieve the desired urban management and optimization goals.

[0205] 4) Rewards

[0206] The reward function is determined, and corresponding rewards or penalties are given according to the performance of the agent's actions and the goals achieved. The agent obtains rewards based on actual operations and system feedback, and the legitimacy of the rewards is recorded and verified through blockchain technology.

[0207] 5) Learning and Optimization

[0208] Through continuous interaction and reward acquisition, the agent can learn to optimize its decision-making strategy and improve urban management efficiency and service quality. The agent uses the real-time updated data of the digital twin system to adjust its strategy to adapt to the dynamic changes of the urban environment, such as holiday traffic control and emergency response.

[0209] Application Example 2: Internet of Vehicles and Autonomous Driving

[0210] 1) Real-time data exchange and secure communication

[0211] Blockchain technology: Blockchain technology is used to ensure the security and credibility of communication and data exchange between vehicles, preventing information tampering and unauthorized access.

[0212] Digital twin technology: Establish a digital twin model to update and reflect the vehicle's operating status, road conditions, weather and other information in real time, providing real-time data support for the autonomous driving system.

[0213] 2) Traffic optimization and navigation coordination

[0214] Intelligent traffic management: Use vehicle sensor data and digital twin systems to analyze urban traffic flow, adjust traffic lights and road conditions in real time, and optimize vehicle driving routes.

[0215] Collaborative navigation: Vehicles share traffic information and the latest navigation suggestions through D2D connections to avoid congestion and provide the best navigation routes.

[0216] 3) Real-time decision making and action execution

[0217] Reinforcement learning and adaptive control: Autonomous vehicles learn and optimize driving strategies through the PPO algorithm and make instant decisions based on real-time environmental data, such as avoiding obstacles and changing lanes.

[0218] The agent interacts with the environment: It perceives surrounding vehicles, pedestrians, and road conditions, performs safe and efficient driving actions, and continuously updates the environmental model through the digital twin system.

[0219] 4) Security and data privacy protection

[0220] Application of blockchain technology: Vehicles share security verification information through blockchain to ensure the security of vehicle identity authentication and traffic data.

[0221] Privacy protection: Use blockchain technology to manage and control access rights to vehicle data, protecting users' personal privacy and driving behavior data.

[0222] 5) Vehicle health monitoring and maintenance

[0223] Remote diagnosis and maintenance: Through the Internet of Vehicles technology, vehicle sensor data can be monitored and collected to predict maintenance needs and arrange services in advance, reducing vehicle failure rates and repair costs.

[0224] Digital twin technology support: Establish a digital twin model of the vehicle to track the vehicle's health status and performance parameters in real time, and provide data support for vehicle maintenance.

[0225] In these two embodiments, the edge cache optimization method in D2D networks supported by blockchain and digital twins provides an efficient, reliable and energy-saving solution suitable for different application scenarios, from smart cities to connected vehicles and autonomous driving, demonstrating its wide application potential and technical advantages.

[0226] In order to comprehensively evaluate the performance of the present invention, the present invention compares it with multiple representative benchmark algorithms. These benchmark algorithms each have their own characteristics and advantages and perform well when dealing with similar problems. By comparing the performance of these algorithms under the same conditions, the present invention can more clearly evaluate the superiority of the present invention and explore its potential room for improvement.

[0227] 1) Random placement: In each time slot, each content has the characteristic of random placement, which means that they may be randomly selected and distributed in different UEs.

[0228] 2) Fixed placement: The location chosen for each content remains unchanged throughout the simulation, meaning that each UE always caches the same content.

[0229] 3) Random acquisition: Each UE acquires content in a random manner, including local acquisition, D2D acquisition, and content server acquisition. In addition, the content acquisition strategy also changes randomly in each time slot.

[0230] exist Figure 4 The convergence performance of the proposed algorithm and three baseline algorithms is shown in Figure 2. All algorithms were evaluated under default parameter settings. The evaluation of the proposed algorithm includes the convergence speed and the reward obtained, providing a comprehensive perspective on the algorithm performance. It is worth noting that the proposed algorithm exhibits excellent convergence performance and reaches a stable state at about the 100th round, with the reward value stabilized at 8.4×10 4 By comparing the curves, the present invention clearly demonstrates this excellent performance, which fully confirms that the method of the present invention is superior to the baseline algorithm.

[0231] Example 1: Real-time data processing and cache optimization in smart manufacturing

[0232] In the field of smart manufacturing, a large number of devices and sensors on the production line will generate massive amounts of data, which need to be processed in real time on edge devices to ensure the continuity and efficiency of production. Traditional edge caching strategies often fail to effectively manage such a large amount of data, resulting in data transmission delays and reduced system response speed.

[0233] The present invention realizes efficient management and transmission of data by applying the D2D network edge cache optimization method supported by blockchain and digital twin in the intelligent manufacturing system. The intelligent agent is deployed at multiple key nodes of the production line, and the deep reinforcement learning algorithm is used to dynamically adjust the cache and acquisition strategy of the content, so that the production data can be processed and transmitted in a timely manner. By introducing TMV-Pt seed synthesis and nanoprobe detection technology, the system can achieve efficient caching and rapid processing of key data without affecting production efficiency. Experiments show that compared with traditional caching methods, the system improves data processing efficiency by 30% and reduces data transmission delay by 50%, significantly improving the operating efficiency and stability of the production line.

[0234] Example 2: Edge computing and cache optimization in smart grids

[0235] In smart grid systems, distributed energy management requires real-time monitoring and optimization of electricity production, distribution and consumption. Due to the large number of nodes in the power grid and their wide distribution, the traditional centralized data processing method is prone to bottlenecks when processing large amounts of real-time data, resulting in long system response times, affecting the timeliness of power dispatch and the stability of the power grid.

[0236] By applying the method of the present invention, the smart grid system can take advantage of the D2D network, distribute data processing on multiple edge nodes, and ensure the security of data transmission through blockchain technology. The intelligent agent is deployed at each power node, and the deep reinforcement learning algorithm is used to dynamically adjust the cache strategy to cope with the ever-changing power demand and supply in the power grid. The system collects the status data of each node in real time, including user power consumption, power generation, equipment health status, etc., and generates optimized content placement and acquisition strategies to ensure that data can be processed and transmitted in a timely manner. The application results show that this method increases the response speed of the smart grid system by 40%, while increasing the optimization efficiency of power distribution by 25%, greatly enhancing the stability and reliability of the power grid.

[0237] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0238] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.

Claims

1. A method for optimizing edge cache in a D2D network supported by blockchain and digital twins, characterized in that: The method applies the actor and critic networks of the intelligent agent and adopts the deep reinforcement learning method to make decisions; initializes the parameters of the policy actor network and the critic network, and sets the hyperparameters related to training; the intelligent agent interacts with the environment based on the current strategy, performs actions and performs state transitions; performs truncation, calculates the estimated return and advantage function; updates the actor network and the critic network; repeats the training until the predefined life stop growth condition is reached, and uses the trained strategy for content placement and content acquisition; The specific steps include: S101, initialize the agent's strategy actor network π(a|s; θ) and its parameters θ, and initialize the critic network V ω (s) and its parameters ω, experience pool M, maximum number of rounds T max ; Initialize the learning rates μ and σ of the actor network and critic network respectively; Initialize the network layout parameters, such as the number of users I, the number of contents F, etc.; S102, the agent interacts with the environment and collects information about the current state s t information, such as user location, estimated processing capacity and remaining energy; S103, based on the current state s t , using the old strategy network π(a t |s t θ old ) Get action a t , get content placement and acquisition strategies; S104, the agent performs action a t , observe the reward r t , and obtain the state s at the next moment t+1 ; S105, the interactive data <s t ,a t ,r t ,s t+1 >Store in the experience pool to form a trajectory sequence; S106, when the trajectory reaches the truncation length T, use the critic network V ω (s) Evaluate the terminal state and get the estimated reward R t Then, the advantage function A(s) is calculated based on the trajectory sequence. t ,a t ); S107. Randomly select a set of experience tuples from the experience pool. Each tuple includes state s t , action a t and the corresponding reward R t ; S108. The selected experience tuple is used to train the critic network. During the training process, the parameters ω of the critic network are updated so that the loss function L critic (ω) minimum; S109, randomly select a set of tuples from the experience pool for training the actor network; S110, constructing the strategy objective function L clip (ω), use the gradient ascent method to update the parameters θ of the actor network so that the policy objective function L clip (ω) maximized; S111, continue to iterate the above steps until the predefined life cycle growth stop condition is reached. In each iteration, the agent contacts the environment, collects trajectory data, and uses this data to adjust the parameters of the actor network, thereby continuously improving the strategy and improving the performance of the agent.

2. The edge cache optimization method in a D2D network supported by blockchain and digital twins as claimed in claim 1, characterized in that: In S102, the agent interacts with the environment and initializes the system state s at the beginning of each cycle. t , specifically expressed as: in, where u i (t-1) = {x i (t-1),y i (t-1)} is the coordinate of UE i at the end of time slot t-1; in Indicates whether UE i requests content f. Note that a UE can only request one content in time slot t; in is the preference of UE i for content f at time slot t; represents the estimated processing capability of UE i in time slot t; represents the remaining energy of each UE at the beginning of time slot t. Specifically, represents the fast fading coefficient of the D2D communication link between UE i and UE j at time slot t; represents the distance between UE i and UE j at time slot t; represents the Rayleigh fast fading coefficient between UE i and the base station at time slot t; represents the distance between UE i and the base station at time slot t; Indicates whether there is a direct physical link between UE i and UE j at time slot t; Indicates whether there is a direct physical link and social connection between UE i and UE j at time slot t; represents the set of neighbor users of UE i at time slot t; represents the number of neighbor users of UE i at time slot t; represents the tanimoto coefficient between UE i and UE j at time slot t; represents the D2D sharing probability between UE i and UE j at time slot t; αT vali (t) represents the block verification delay deviation; represents the transaction size of UE i at time slot t; represents the number of signature verification failures received from UE i at time slot t; represents the reputation value of UE i in time slot t-1; represents the growth index of UE i’s lifetime at time slot t; represents the temporary lifetime of each UE in time slot t, and its initial lifetime is 0; In each time slot t, the agent determines its action, denoted as in: Represents content placement strategy; Indicates the content acquisition strategy.

3. The edge cache optimization method in a D2D network supported by blockchain and digital twins as claimed in claim 1, characterized in that: S103: The agent executes the generated action, obtains an immediate reward according to the executed action, and transfers the environment state to the next state. The calculation consensus of the reward is as follows:

4. The edge cache optimization method in a D2D network supported by blockchain and digital twins as claimed in claim 1 is characterized in that: S106: Calculate the advantage function: A(s t ,a t )=d t +gd t+1 +...+c T-t+1 d t+1 , Represents the advantage of a truncated trajectory of length T. The advantage function refers to the advantage or relative benefit obtained by an agent taking a certain action relative to other possible actions, where δ t =r t +γV(s t+1 )-V(s t ), Among them, V(s t ) is the output of the critic network. Based on this, A(s t ,a t ) can be rewritten as A(s t ,a t )=R(t)-V(s t ), Where R(t) is used to estimate the expected return at time step t.

5. The edge cache optimization method in a D2D network supported by blockchain and digital twins as claimed in claim 1 is characterized in that: S108: Update the parameters of the critic network, and the update formula is as follows: in, Where N represents the number of mini-batches. Indicates the status of the nth sample in the mini-batch.

6. The edge cache optimization method in a D2D network supported by blockchain and digital twins as claimed in claim 1 is characterized in that: S110: Update the parameters of the actor network, and the update formula is as follows: Among them, η is a hyperparameter, L clip (θ) is expressed as 7. A system for edge cache optimization in a D2D network supported by blockchain and digital twins, characterized in that: include: An agent module that configures the parameters used to initialize the policy actor network and the critic network and sets the training-related hyperparameters; an interaction module configured to enable the agent to interact with the environment, perform actions and make state transitions; an evaluation module, configured to truncate trajectories and compute estimated rewards and advantage functions; An update module configured to update the actor network and the critic network until a predefined life stop growth condition is reached and use the trained policies for content placement and content acquisition.

8. The system according to claim 7, characterized in that The agent module further comprises: An initialization unit, which is configured to initialize the agent's policy actor network and its parameters, the critic network and its parameters, the experience pool, and the maximum number of rounds; A network layout unit is configured to initialize network layout parameters, including the number of users and the number of contents.

9. The system according to claim 7, characterized in that The interaction module further comprises: a state collection unit configured to initialize the system state at the beginning of each cycle and collect information about the current state, including user location, estimated processing capacity, remaining energy, etc.; a policy generation unit configured to generate content placement and acquisition policies based on a current state by using an old policy network to obtain actions; A reward observation unit is configured to allow the agent to perform actions and observe rewards, obtain the state at the next moment, and store the interaction data into the experience pool to form a trajectory sequence.

10. The system according to claim 7, characterized in that The evaluation module further comprises: an evaluation unit configured to evaluate the terminal state using a critic network and calculate an advantage function when the trajectory reaches a cutoff length; A training unit is configured to randomly select experience tuples from the experience pool, train the parameters of the critic network and the actor network, and update the parameters of the actor network using the gradient ascent method according to the policy objective function until a predefined life cycle stop growth condition is reached.

Images