Method and system for edge cache optimization in blockchain and digital twin supported d2d network

Abstract

The application relates to the field of wireless communication and discloses an optimization method and system for edge caching in a device-to-device network supported by a blockchain and digital twin. A target function is maximized by jointly optimizing content placement and content acquisition strategies, which comprehensively considers content acquisition gain, network life cycle and user switching cost. The application solves the problems of short network life, long edge caching delay and high cost in a wireless network. In the application, each user can acquire content from his own cache buffer, from other users through a D2D link or from a content server. The digital twin monitors the network operation state by storing key data such as the positions, estimated processing capacity and residual energy of each user. In order to ensure the safety and credibility of transactions between users, the application uses blockchain technology to supervise transactions and continuously update user credit values. For complex problems in a dynamic network, the application uses a proximal policy optimization algorithm for solution.

Description

Technical Field

[0001] The present invention belongs to the field of wireless communication technology, and in particular to a method and system for optimizing edge caching in a D2D network supported by blockchain and digital twins. Background Art

[0002] Currently, with the rapid development of wireless communications, existing mobile networks are facing a rapid increase in mobile data traffic. This unprecedented traffic growth poses a huge challenge to traditional cellular networks due to their reliance on computing, storage, and communication resources. If user equipment (UE) directly accesses or retrieves content from remote data centers such as cloud servers, long delays may occur. Mobile edge caching has become a promising technology that can not only reduce content access latency but also alleviate network traffic load. In an edge caching system, content can be stored in base stations (BSs) and edge servers.

[0003] Device-to-device (D2D) communication technology enables communication between user devices (UEs) and has been used to reduce data transmission latency and for edge caching. Digital twins are an emerging technology that connects the physical and digital realms and can be used for real-time prediction, optimization, monitoring, control, and decision-making, as well as for real-time communication and sustainable computing. Blockchain is a distributed ledger technology that records and verifies transaction data through decentralization and immutability. Integrating digital twins and blockchain technologies into D2D networks requires addressing a number of challenges:

[0004] 1) Issues with data real-timeness and accuracy

[0005] Ensure that user data and network status information obtained from the digital twin system is updated in real time and accurately reflects actual conditions. This includes real-time data collection and transmission, timely data processing and analysis, updating the digital twin model to reflect the latest status changes, and rapidly responding to and adapting to dynamic changes in the network environment and user behavior. This ensures that the system provides accurate and up-to-date information at all times, providing a reliable foundation for optimizing network performance and resource allocation.

[0006] 2) High-dimensional and dynamically changing data processing issues

[0007] Effectively process and analyze complex, high-dimensional user data, adapting to dynamic changes such as user location and device status. The goal is typically defined as a long-term goal. In this case, the present invention requires the design of a higher-performance algorithm to achieve target optimization while maintaining low complexity and good performance. At the same time, it is also necessary to design a flexible system architecture and real-time data processing pipeline that can quickly respond to dynamic changes such as user location changes and device status updates to ensure that network performance and user experience are always optimal. Through these means, the system can maintain efficient and stable operation in complex and changing environments.

[0008] 3) Optimal allocation of network resources

[0009] Optimize edge caching and network resource allocation strategies to improve network performance and user experience. This includes jointly optimizing content placement and retrieval strategies, dynamically adjusting cached content based on user demand, network load, and device status. Specifically, by predicting user content access patterns, frequently accessed content can be pre-cached at edge nodes, reducing data transmission latency and network congestion. Furthermore, the resource capabilities and energy consumption characteristics of different devices must be considered, and computing tasks and data transmission paths must be properly scheduled to maximize resource utilization and user quality of service. Through these optimization measures, network systems can provide a smoother and more satisfying user experience while ensuring high performance and reliability.

[0010] 4) Network lifespan issues

[0011] Users constantly retrieve content from edge nodes, whether from their own caches or from other users via D2D links. This consumes their energy. To extend network life, efficient energy management strategies are needed that balance energy consumption and performance. Specific measures include optimizing content retrieval strategies, reducing unnecessary transmissions, and prioritizing data exchange with nodes with sufficient energy. Summary of the Invention

[0012] In response to the problems existing in the prior art, the present invention provides a method and system for optimizing edge caching in a D2D network supported by blockchain and digital twins. The network system includes two subsystems: an edge caching subsystem and a blockchain subsystem. In the edge caching subsystem, terminals can act as cache nodes and store content locally. Therefore, each UE can obtain content from its own cache buffer, through D2D links, or from other UEs from a content server. The blockchain subsystem manages delays in the block consensus process and continuously updates the UE's reputation value, thereby ensuring the security and reliability of D2D data sharing.

[0013] The present invention is implemented as follows: a method for optimizing edge caching in a D2D network supported by blockchain and digital twins, which applies an intelligent agent's actor and critic network and adopts a deep reinforcement learning method to make decisions; initializes the parameters of the policy actor network and critic network, and sets training-related hyperparameters; the intelligent agent interacts with the environment based on the current strategy, executes actions and performs state transitions; performs truncation, calculates estimated rewards and advantage functions; updates the actor network and critic network; repeats training until a predefined life cycle stops growing, and uses the trained strategy for content placement and content acquisition.

[0014] Further, the following steps are included:

[0015] S101. Initialize the agent's policy actor network π(a|s;θ) and its parameters θ, and initialize the critic network V ω (s) and its parameters ω, experience pool M, maximum number of rounds T max Initialize the learning rates μ and σ of the actor network and critic network respectively; Initialize the network layout parameters, such as the number of users I, the number of contents F, etc.

[0016] S102, the agent interacts with the environment and collects information about the current state s t Information such as user location, estimated processing capacity and remaining energy;

[0017] S103, based on the current state s t , using the old strategy network π(a t |s t θ old ) Get action a t , get content placement and acquisition strategies;

[0018] S104, the agent performs action a t , observe the reward r t , and obtain the state s at the next moment t+1 ;

[0019] S105, the interactive data <s t ,a t ,r t ,s t+1 >Store in the experience pool to form a trajectory sequence;

[0020] S106, when the trajectory reaches the truncation length T, use the critic network V ω (s) Evaluate the terminal state and get the estimated reward R t Then, the advantage function A(s) is calculated based on the trajectory sequence. t ,at );

[0021] S107, randomly select a set of experience tuples from the experience pool. Each tuple includes state s t , action a t and the corresponding reward R t ;

[0022] S108, the selected experience tuple is used to train the critic network. During the training process, the parameters ω of the critic network are updated so that the loss function L critic (ω) minimum;

[0023] S109, randomly select a set of tuples from the experience pool for training the actor network;

[0024] S110, constructing the strategy objective function L clip (ω), use the gradient ascent method to update the parameters θ of the actor network so that the policy objective function L clip (ω) maximized;

[0025] S111. Continue iterating the above steps until the predefined lifecycle growth condition is reached. In each iteration, the agent interacts with the environment, collects trajectory data, and uses this data to adjust the parameters of the actor network, thereby continuously improving the strategy and improving the performance of the agent.

[0026] Furthermore, in S102, the agent interacts with the environment and initializes the system state s at the beginning of each cycle. t , specifically expressed as:

[0027]

[0028] in,

[0029] u(t-1)={u i (t-1)}, where u i (t-1)={x i (t-1),y i (t-1)} is the coordinate of UE i at time slot t-1;

[0030] in Indicates whether UE i requests content f. Note that a UE can only request one content in time slot t;

[0031] in is the preference of UE i for content f at time slot t;

[0032] represents the estimated processing capability of UE i in time slot t;

[0033] represents the remaining energy of each UE at the beginning of time slot t, specifically,

[0034] represents the fast fading coefficient of the D2D communication link between UE i and UE j at time slot t;

[0035] represents the distance between UE i and UE j at time slot t;

[0036] represents the Rayleigh fast fading coefficient between UE i and the base station at time slot t; represents the distance between UE i and the base station at time slot t;

[0037] Indicates whether there is a direct physical link between UE i and UE j at time slot t;

[0038] Indicates whether there is a direct physical link and social connection between UE i and UE j at time slot t;

[0039] represents the set of neighbor users of UE i at time slot t;

[0040] represents the number of neighboring users of UE i at time slot t;

[0041] represents the tanimoto coefficient between UE i and UE j at time slot t; represents the D2D sharing probability between UE i and UE j at time slot t;

[0042] ΔT vali (t) represents the block verification delay deviation;

[0043] represents the transaction size of UE i at time slot t;

[0044] represents the number of signature verification failures received from UE i at time slot t; represents the reputation value of UE i in time slot t-1;

[0045] represents the growth index of UE i’s lifetime at time slot t;

[0046] represents the temporary lifetime of each UE in time slot t, and its initial lifetime is 0;

[0047] In each time slot t, the agent determines its action, denoted as in: Represents content placement strategy; A(t) = {a i,j (t)}, Indicates the content acquisition strategy.

[0048] Furthermore, the consensus calculation of the reward in S103: the agent executes the generated action, obtains an immediate reward based on the executed action, and transfers the environment state to the next state is as follows:

[0049]

[0050] Furthermore, the S106: calculating the advantage function:

[0051] A(s t ,a t )=δ t +γδ t+1 +...+γ T-t+1 δ t+1 ,

[0052] Represents the advantage of a truncated trajectory of length T. The advantage function refers to the advantage or relative benefit obtained by the agent when taking a certain action relative to other possible actions, where

[0053] δ t =r t +γV(s t+1 )-V(s t ),

[0054] Among them, V(s t ) is the output of the critic network. Based on this, A(s t ,a t ) can be rewritten as

[0055] A(s t ,a t )=R(t)-V(s t ),

[0056] Where R(t) is used to estimate the expected return at time step t.

[0057] Furthermore, in step S108, the parameters of the critic network are updated, and the updating formula is as follows:

[0058]

[0059] in,

[0060]

[0061] Where N represents the number of mini-batches. Indicates the status of the nth sample in the mini-batch.

[0062] Furthermore, in S110: updating the parameters of the actor network, the updating formula is as follows:

[0063]

[0064] Among them, η is a hyperparameter, L clip (θ) is expressed as

[0065]

[0066] Another object of the present invention is to provide a system for edge cache optimization in a D2D network supported by blockchain and digital twins, comprising:

[0067] An agent module, which configures the parameters used to initialize the policy actor network and the critic network and sets the training-related hyperparameters;

[0068] an interaction module configured to enable the agent to interact with the environment, perform actions, and perform state transitions;

[0069] an evaluation module configured to truncate trajectories and compute estimated rewards and advantage functions;

[0070] An update module is configured to update the actor network and the critic network until a predefined life stop growth condition is reached and use the trained policy for content placement and content acquisition.

[0071] Furthermore, the intelligent agent module further includes:

[0072] An initialization unit, which is configured to initialize the agent's policy actor network and its parameters, the critic network and its parameters, the experience pool, and the maximum number of rounds;

[0073] A network layout unit is configured to initialize network layout parameters, including the number of users and the number of contents.

[0074] Furthermore, the interaction module further includes:

[0075] a state collection unit configured to initialize the system state at the beginning of each cycle and collect information about the current state, including user location, estimated processing capacity, remaining energy, etc.;

[0076] a policy generation unit configured to generate content placement and acquisition policies based on a current state by utilizing an old policy network to obtain actions;

[0077] A reward observation unit is configured to allow the agent to perform actions and observe rewards, obtain the state at the next moment, and store the interaction data in the experience pool to form a trajectory sequence.

[0078] Furthermore, the evaluation module further includes:

[0079] an evaluation unit configured to evaluate the terminal state using a critic network and calculate an advantage function when the trajectory reaches a truncation length;

[0080] A training unit is configured to randomly select experience tuples from the experience pool, train the parameters of the critic network and the actor network, and update the parameters of the actor network using the gradient ascent method according to the policy objective function until a predefined life cycle stop growth condition is reached.

[0081] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:

[0082] First, the present invention has important benefits in integrating blockchain and D2D networks supported by digital twins, as detailed below:

[0083] The first is efficiency and security. The present invention considers a D2D-assisted cellular network that supports blockchain and edge caching, which includes two subsystems, namely an edge caching subsystem and a blockchain subsystem. The edge caching subsystem is used for content file caching and sharing, and it consists of a remote content server, a BS, and UEs randomly distributed within the coverage of the BS. The blockchain subsystem is used to build a secure and reliable data information sharing and transaction platform, and the UE also serves as a blockchain node. It mainly performs secure processing on content acquisition transactions from the edge caching subsystem, thereby ensuring the traceability of content acquisition. Transactions related to cache and UE reputation updates are packaged into blocks, verified through a consensus mechanism process, and then recorded in the blockchain. The digital twin layer is integrated in the edge caching subsystem to store key data such as terminal location, terminal content requests, estimated processing capacity, remaining energy, etc., and monitor the network operation status in real time.

[0084] Secondly, considering the dynamic nature of tasks, the volatility of network conditions and the random behavior of content requests, the present invention adopts Proximal Policy Optimization (PPO) to learn the environment state and obtain the joint optimization decision of content placement and content acquisition.

[0085] Second, the significant technical advancement achieved by this invention lies in the implementation of an edge cache optimization method in D2D networks supported by blockchain and digital twins. This method has made significant progress in the following key aspects:

[0086] 1) Accurate content placement and content acquisition strategy

[0087] This method significantly improves edge cache hit rates and resource utilization by introducing content placement and retrieval strategies. Specifically, it selects popular content and places it on appropriate UEs, thereby reducing transmission latency. Furthermore, by optimizing content retrieval strategies, it ensures that UEs can efficiently retrieve required content from the nearest, communicable cache node, further improving network performance and user experience.

[0088] 2) Intelligent access control strategy

[0089] By introducing advanced learning mechanisms, the system can intelligently adjust UE content control policies to adapt to dynamically changing network conditions. This not only enhances network stability and user experience, but also effectively balances the load and avoids overloading of individual nodes.

[0090] 3) Integration of reinforcement learning

[0091] Combining DRL algorithms with edge computing systems enables the system to self-learn and adapt based on real-time data, making optimal decisions without explicit instructions. This adaptive capability is crucial for handling complex and dynamically changing edge computing environments, ensuring the system can quickly respond and adjust strategies.

[0092] 4) Maximize gain

[0093] The reward mechanism of this method focuses on the UE's content acquisition gain, and optimizes the strategy to maximize the user's gain when acquiring content. This not only improves user satisfaction, but also optimizes the utilization efficiency of network resources.

[0094] 5) Energy efficiency optimization

[0095] This method takes into account the energy consumption of the device and optimizes the content transmission path and cache location by designing an efficient energy management strategy, reducing unnecessary energy consumption, extending the network life and improving the overall performance.

[0096] 6) Enhanced multi-user collaboration:

[0097] By introducing a multi-user collaboration mechanism, the system allows users to share cache resources and content, further improving resource utilization and content acquisition efficiency. This collaboration mechanism not only reduces network transmission latency but also reduces overall energy consumption.

[0098] 7) Efficient data analysis and prediction

[0099] The system uses historical data and user behavior patterns to make predictions and optimize content acquisition and caching strategies. This helps identify popular content in advance, adjust cache locations in a timely manner, and avoid potential network congestion.

[0100] 8) Improvement of system stability and reliability

[0101] By accurately calculating and optimizing immediate rewards and state transitions after action execution, this approach significantly enhances system stability and reliability. Especially when processing large amounts of data and high-concurrency requests, the system maintains efficient operation, ensuring continuous improvement in service quality and user experience.

[0102] 9) The network's autonomous learning and optimization capabilities

[0103] This approach enables the system to continuously optimize its decision-making process through continuous iterative training and experience-based network updates. This autonomous learning and optimization capability not only improves overall performance but also enables the system to dynamically adapt to environmental changes, ensuring optimal service quality in various usage scenarios.

[0104] These technological advances work together to enable D2D networks supported by blockchain and digital twins to not only improve performance, but also achieve significant improvements in energy efficiency, stability, and adaptability, which are critical to modern complex computing needs.

[0105] Third, the core of the edge cache optimization method in D2D networks supported by blockchain and digital twins provided by this invention is the use of mathematical models to guide the system's behavior and learning process. The technical effects brought about by these mathematical models can be explored based on their characteristics:

[0106] 1) Calculation of instant rewards

[0107] The calculation formula of immediate rewards focuses on the system's content acquisition gain, network lifespan, and switching cost.

[0108] Content acquisition gain: UEs can obtain content from local caches, from other users via D2D links, or directly from content servers. Different content acquisition methods will result in different content acquisition gains, depending on the method selected.

[0109] Network Lifetime: Each UE starts with a full battery charge, which gradually depletes during content caching and retrieval. Once a UE's battery charge falls below a preset threshold, its lifetime stops increasing, becoming the network lifetime. Therefore, even though the incentive mechanism aims to maximize network lifetime, it ensures overall network performance and stability.

[0110] Handover costs: In a dynamic user environment, UE reassociation is a concern due to the UE's volatile location. When reassociation occurs, the UE may experience service interruption and reduced communication quality. Mobile network operators need to update the UE's service profile to accommodate mobility, which incurs handover costs.

[0111] 2) Updates to the Actor and Critic Networks

[0112] Update the current policy network by randomly sampling small batches of experience data and using the gradient ascent method.

[0113] Policy Optimization: By continuously adjusting the parameters of the policy network, the system learns and adopts more effective decision-making strategies. This process ensures that the network can select the best actions in different situations, improving overall performance and efficiency.

[0114] Improved responsiveness: Using small batches of data for updates enables the network to quickly adapt to environmental changes and enhances the system's dynamic adjustment capabilities. This approach not only speeds up learning but also makes the system more flexible and reliable in responding to emergencies and environmental changes.

[0115] Improved stability: Through small batch updates and gradient ascent, fluctuations during training are reduced, ensuring network stability and convergence. This effectively prevents instability caused by large-scale data updates, ensuring a smoother and more efficient training process.

[0116] 3) Parameter update formula

[0117] The target network method for updating the policy actor and critic network parameters is described, involving the parameters of the current network and the target network.

[0118] Smooth policy transition: By introducing a pruning function, the system can smoothly transition to a new policy, avoiding performance fluctuations caused by drastic changes. This approach maintains stability during the update process and gradually improves decision quality.

[0119] Continuous Learning and Adaptation: This continuous parameter update mechanism ensures the system can cope with long-term environmental changes. By constantly adjusting parameters, the system can continue to operate efficiently and optimize its strategies in different situations and dynamic environments.

[0120] Fourth, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the expected benefits and commercial value after the technical solution is transformed:

[0121] This invention utilizes the PPO algorithm to optimize edge caching in D2D networks supported by blockchain and digital twins, significantly improving network performance and service quality. This not only effectively addresses the growing demand for user devices but also optimizes content caching and retrieval strategies, thereby reducing network operating costs. In terms of commercial applications, this invention is applicable to a wide range of scenarios requiring efficient big data processing while ensuring data security and reliability, bringing significant economic and social benefits to related industries. Therefore, this invention has broad market prospects and enormous commercial value.

[0122] Fifth, the present invention solves several technical problems existing in the prior art in edge caching optimization in industrial applications. Traditional D2D (Device-to-Device) network edge caching methods are usually faced with problems such as low resource management efficiency, slow response speed, and insufficient reliability during data transmission. These problems are particularly prominent in complex industrial environments supported by blockchain and digital twins, making it difficult to ensure the real-time and accuracy of data, thereby affecting the performance and stability of the overall system. In addition, traditional methods usually lack intelligent decision-making mechanisms and cannot effectively respond to dynamically changing network environments and user needs, limiting their widespread application in industrial applications.

[0123] By introducing the deep reinforcement learning (DRL) method, the present invention has achieved significant technological progress in industrial applications. First, the use of intelligent agent actor and critic networks can dynamically adapt to complex network environments and make intelligent decisions to optimize content placement and acquisition strategies. This intelligent optimization process not only improves the resource utilization of the system, but also greatly reduces the delay in content transmission, significantly enhancing the response speed and overall performance of edge caches. Especially in industrial scenarios, the ability to respond in real time and dynamically adjust cache strategies is of great significance to ensuring the continuity of production lines and the immediate processing of industrial data.

[0124] Furthermore, this invention enables efficient network resource management in industrial applications. By introducing blockchain and digital twin technologies, the system can optimize cache within a decentralized framework, ensuring the security and integrity of data transmission. Furthermore, through the continuous interaction and learning between the agent and the environment, the system can continuously optimize its strategies to adapt to the changing needs of the industrial environment. Compared with traditional static optimization methods, the dynamic optimization mechanism of this invention can better meet the real-time requirements of industrial scenarios and reduce system maintenance costs.

[0125] Ultimately, the edge cache optimization method of this invention has achieved significant technological advancements in industrial applications, improving the efficiency and stability of D2D networks while also achieving high reliability and low latency in data transmission. This provides a superior solution for data management and communication in industrial production processes, significantly enhancing the intelligence and automation capabilities of industrial systems, and laying a solid foundation for the future development of smart manufacturing and the Industrial Internet of Things. BRIEF DESCRIPTION OF THE DRAWINGS

[0126] The following briefly describes the drawings required for the technical solutions of the embodiments of the present invention to illustrate more clearly. These drawings illustrate some embodiments of the present invention and provide a method for those skilled in the art to obtain other implementation schemes based on these examples without the need for creative work.

[0127] Figure 1 This is a flow chart of a method for optimizing edge cache in a D2D network supported by blockchain and digital twins provided by an embodiment of the present invention;

[0128] Figure 2 This is a scenario diagram for the application of the edge cache optimization method in a D2D network supported by blockchain and digital twins provided by an embodiment of the present invention.

[0129] Figure 3 This is a flowchart of an implementation method of an edge cache optimization method in a D2D network supported by blockchain and digital twins provided by an embodiment of the present invention.

[0130] Figure 4 This is a comparison chart of the comprehensive convergence performance of the edge cache optimization method in the D2D network supported by blockchain and digital twins provided by the embodiments of the present invention. DETAILED DESCRIPTION

[0131] In order to further illustrate the technical means and effects adopted by the present invention to achieve the predetermined purpose, the specific implementation methods, structural features and corresponding effects of the present invention are described in detail below with reference to the accompanying drawings and examples.

[0132] The key parameters, algorithms, and mathematical models of this invention are designed to address the shortcomings of existing technologies in D2D network edge caching optimization. Key parameters include the initial parameter settings of the agent's policy actor network and critic network, the learning rate, the experience pool capacity, and the maximum number of rounds. The precise setting of these parameters directly impacts the system's learning efficiency and decision-making accuracy. By optimizing these parameters, the system can better adapt to complex network environments and ensure stability and efficiency in various industrial application scenarios.

[0133] In terms of algorithms, this invention uses a deep reinforcement learning (DRL) algorithm to dynamically optimize content placement and acquisition strategies through continuous interaction and learning between the agent and the environment. By constructing an actor-critic architecture, the algorithm enables the agent to make optimal decisions under constantly changing network conditions, effectively balancing data transmission latency and cache efficiency, particularly in resource-limited industrial environments. Compared to traditional rule-based or static optimization methods, the DRL algorithm enables the system to be adaptive and self-learning, significantly improving the robustness of the algorithm and the quality of its decisions.

[0134] The mathematical models introduced in this paper, particularly those used to calculate the advantage function and reward mechanism, address the existing challenges of accurately evaluating and predicting system performance. By accurately calculating metrics such as state transitions, rewards, and estimated returns, the system can better adjust policy parameters and improve the performance of intelligent agents in real-world applications. These mathematical models not only enhance the system's predictive capabilities but also increase its adaptability in complex industrial environments, enabling it to continuously optimize edge caching strategies under changing conditions.

[0135] Ultimately, the present invention has achieved significant technological progress in industrial applications, mainly reflected in the improvement of optimization accuracy, response speed and system reliability. By optimizing parameters, improving algorithms and applying mathematical models, the present invention effectively solves the problems of insufficient resource utilization, decision lag and poor system stability in traditional methods. Its intelligent and adaptive edge cache optimization method provides more efficient and reliable solutions for the fields of industrial Internet of Things, smart manufacturing, etc., and significantly promotes the intelligentization and automation of industrial systems. In response to the problems existing in the existing technology, this implementation provides an edge cache optimization method in D2D networks supported by blockchain and digital twins, Figure 2 This is a diagram of a scenario in which the method of the present invention can be applied. It includes two subsystems: the edge caching subsystem and the blockchain subsystem. The edge caching subsystem is used for content file caching and sharing. It consists of a remote content server, a base station (BS), and UEs randomly distributed within the BS's coverage area. The blockchain subsystem is used to build a secure and reliable data information sharing and transaction platform, with UEs also serving as blockchain nodes.

[0136] The BS is connected to the content server via a wired optical fiber link, and all UEs can communicate directly with the BS via a wireless 4G or 5G link. The set of UEs is represented as Its index is denoted as i. In addition to radio stations near stadiums, similar situations also occur around concert venues. In these occasions, a large number of audiences gather together and use mobile devices to access music or video streams, which puts huge traffic pressure on the surrounding radio stations. In order to prevent network congestion caused by repeated downloading of content from the BS, the UE can act as a cache node, store the content locally, and share it with other UEs through D2D links. There are F contents in the content server. The content file is represented as The size of each file is s f In this framework, the content server stores all layers of all F files, and UEs that support caching can cache copies of popular content files. Assuming that the system of the present invention is executed in slots indexed by t, and the length of each slot is Δt, the set of slots is expressed as

[0137] The blockchain subsystem securely processes content sharing transactions from the edge cache subsystem, ensuring the traceability of content sharing. Transactions related to caching and UE reputation updates are packaged into blocks, verified through a consensus mechanism, and then recorded in the blockchain.

[0138] Due to the dynamic characteristics of the network and the uncertainty of information acquisition, the problem becomes extremely complex. To effectively address this challenge, the present invention uses the Markov Decision Process (MDP) to remodel the problem. Considering that the variables involved are discontinuous, the present invention chooses the PPO algorithm as a solution. PPO is a DRL algorithm designed to solve reinforcement learning problems in continuous action spaces, which can support real-time online decision making.

[0139] like Figure 1 As shown, an embodiment of the present invention provides an edge cache optimization method in a D2D network supported by blockchain and digital twins, which specifically includes the following steps:

[0140] S101. Initialize the agent's policy actor network π(a|s;θ) and its parameters θ, and initialize the critic network V ω (s) and its parameters ω, experience pool M, maximum number of rounds T max Initialize the learning rates μ and σ of the actor network and critic network respectively; Initialize the network layout parameters, such as the number of users I, the number of contents F, etc.

[0141] S102, the agent interacts with the environment and collects information about the current state s t Information such as user location, estimated processing capacity and remaining energy;

[0142] S103, based on the current state s t , using the old strategy network π(a t |s t θ old ) Get action a t , get content placement and acquisition strategies;

[0143] S104, the agent performs action a t , observe the reward r t , and obtain the state s at the next moment t+1 ;

[0144] S105, the interactive data <s t ,a t ,r t ,s t+1 >Store in the experience pool to form a trajectory sequence;

[0145] S106, when the trajectory reaches the truncation length T, use the critic network V ω (s) Evaluate the terminal state and get the estimated reward R t Then, the advantage function A(s) is calculated based on the trajectory sequence. t ,a t );

[0146] S107, randomly select a set of experience tuples from the experience pool. Each tuple includes state s t , action a t and the corresponding reward R t ;

[0147] S108, the selected experience tuple is used to train the critic network. During the training process, the parameters ω of the critic network are updated so that the loss function L critic (ω) minimum;

[0148] S109, randomly select a set of tuples from the experience pool for training the actor network;

[0149] S110, constructing the strategy objective function L clip (ω), use the gradient ascent method to update the parameters θ of the actor network so that the policy objective function L clip (ω) maximized;

[0150] S111. Continue iterating the above steps until the predefined lifecycle growth condition is reached. In each iteration, the agent interacts with the environment, collects trajectory data, and uses this data to adjust the parameters of the actor network, thereby continuously improving the strategy and improving the performance of the agent.

[0151] Furthermore, in S102, the agent interacts with the environment and initializes the system state s at the beginning of each cycle. t , specifically expressed as:

[0152]

[0153] in,

[0154] u(t-1)={u i (t-1)}, where u i (t-1)={x i (t-1),y i (t-1)} is the coordinate of UE i at time slot t-1;

[0155] in Indicates whether UE i requests content f. Note that a UE can only request one content in time slot t;

[0156] in is the preference of UE i for content f at time slot t;

[0157] represents the estimated processing capability of UE i in time slot t;

[0158] represents the remaining energy of each UE at the beginning of time slot t, specifically,

[0159] represents the fast fading coefficient of the D2D communication link between UE i and UE j at time slot t;

[0160] represents the distance between UE i and UE j at time slot t;

[0161] represents the Rayleigh fast fading coefficient between UE i and the base station at time slot t; represents the distance between UE i and the base station at time slot t;

[0162] Indicates whether there is a direct physical link between UE i and UE j at time slot t;

[0163] Indicates whether there is a direct physical link and social connection between UE i and UE j at time slot t;

[0164] represents the set of neighbor users of UE i at time slot t;

[0165] represents the number of neighboring users of UE i at time slot t;

[0166] represents the tanimoto coefficient between UE i and UE j at time slot t; represents the D2D sharing probability between UE i and UE j at time slot t;

[0167] ΔT vali (t) represents the block verification delay deviation;

[0168] represents the transaction size of UE i at time slot t;

[0169] represents the number of signature verification failures received from UE i at time slot t;

[0170] represents the reputation value of UE i in time slot t-1;

[0171] represents the growth index of UE i’s lifetime at time slot t;

[0172] represents the temporary lifetime of each UE in time slot t, and its initial lifetime is 0;

[0173] In each time slot t, the agent determines its action, denoted as in: Represents content placement strategy; A(t) = {a i,j (t)}, Indicates the content acquisition strategy.

[0174] Furthermore, the consensus calculation of the reward in S103: the agent executes the generated action, obtains an immediate reward based on the executed action, and transfers the environment state to the next state is as follows:

[0175]

[0176] Furthermore, the S106: calculating the advantage function:

[0177] A(s t ,a t )=δ t +γδ t+1 +...+γ T-t+1 δ t+1 ,

[0178] Represents the advantage of a truncated trajectory of length T. The advantage function refers to the advantage or relative benefit obtained by the agent when taking a certain action relative to other possible actions, where

[0179] δ t =r t +γV(s t+1 )-V(s t ),

[0180] Among them, V(s t ) is the output of the critic network. Based on this, A(s t ,a t ) can be rewritten as

[0181] A(s t ,a t )=R(t)-V(s t ),

[0182] Where R(t) is used to estimate the expected return at time step t.

[0183] Furthermore, in step S108, the parameters of the critic network are updated, and the updating formula is as follows:

[0184]

[0185] in,

[0186]

[0187] Where N represents the number of mini-batches. Indicates the status of the nth sample in the mini-batch.

[0188] Furthermore, in S110: updating the parameters of the actor network, the updating formula is as follows:

[0189]

[0190] Among them, η is a hyperparameter, L clip (θ) is expressed as

[0191]

[0192] In order to elaborate on the blockchain-driven edge computing system joint optimization method, the present invention provides two specific application embodiments, including key details of the implementation scheme.

[0193] Application Example 1: Smart City and Internet of Things Applications

[0194] 1) System initialization

[0195] Device registration: Various IoT devices in smart cities (such as sensors, cameras, smart lamps, etc.) are registered on the blockchain through the network.

[0196] Initialization data upload: The device uploads initialization data to the digital twin system based on its type and location, including device identification, location coordinates, initial state and other information.

[0197] Environmental modeling: Model various devices, buildings, transportation systems and other entities in the city and synchronize them with the digital twin system to reflect the real environment status in real time.

[0198] Data integration: Integrate data from different devices and sensors to form a comprehensive digital twin model for predicting and simulating urban operation status.

[0199] 2) Interaction between the agent and the environment

[0200] Intelligent body perception: Obtain real-time data of surrounding devices and environment through D2D connection, including environmental parameters, device status, traffic conditions, etc.

[0201] Information collection: Obtain real-time updated urban data from the digital twin system, such as traffic flow, air quality, energy usage, etc.

[0202] 3) Action execution and decision making

[0203] Based on the collected information and the current state of the environment, the intelligent agent performs specific operations and decisions, such as adjusting light brightness, optimizing traffic flow, and providing safety alerts.

[0204] Using the PPO algorithm, the agent selects the optimal action based on the current state to achieve the desired urban management and optimization goals.

[0205] 4) Rewards

[0206] A reward function is defined, and rewards or penalties are given based on the effectiveness of the agent's actions and the goals achieved. The agent receives rewards based on actual operations and system feedback, and the legitimacy of the rewards is recorded and verified through blockchain technology.

[0207] 5) Learning and Optimization

[0208] Through continuous interaction and reward acquisition, the agent learns to optimize its decision-making strategies, improving urban management efficiency and service quality. Using real-time data updated by the digital twin system, the agent adjusts its strategies to adapt to dynamic changes in the urban environment, such as holiday traffic control and emergency response.

[0209] Application Example 2: Internet of Vehicles and Autonomous Driving

[0210] 1) Real-time data exchange and secure communication

[0211] Blockchain technology: Blockchain technology ensures the security and credibility of communication and data exchange between vehicles, preventing information tampering and unauthorized access.

[0212] Digital twin technology: Establish a digital twin model to update and reflect the vehicle's operating status, road conditions, weather and other information in real time, providing real-time data support for the autonomous driving system.

[0213] 2) Traffic optimization and navigation collaboration

[0214] Intelligent traffic management: Utilize vehicle sensor data and digital twin systems to analyze urban traffic flow, adjust traffic lights and road conditions in real time, and optimize vehicle driving routes.

[0215] Collaborative navigation: Vehicles share road conditions and the latest navigation suggestions through D2D connections, avoiding congestion and providing the best navigation routes.

[0216] 3) Real-time decision-making and action execution

[0217] Reinforcement learning and adaptive control: Autonomous vehicles learn and optimize driving strategies through the PPO algorithm, making immediate decisions based on real-time environmental data, such as avoiding obstacles and changing lanes.

[0218] The intelligent agent interacts with the environment: It perceives surrounding vehicles, pedestrians, and road conditions, performs safe and efficient driving maneuvers, and continuously updates the environmental model through the digital twin system.

[0219] 4) Security and data privacy protection

[0220] Application of blockchain technology: Vehicles share security verification information through blockchain to ensure the security of vehicle identity authentication and traffic data.

[0221] Privacy protection: Use blockchain technology to manage and control access rights to vehicle data, protecting users' personal privacy and driving behavior data.

[0222] 5) Vehicle health monitoring and maintenance

[0223] Remote diagnosis and maintenance: Through vehicle networking technology, vehicle sensor data can be monitored and collected to predict maintenance needs and arrange services in advance, reducing vehicle failure rates and repair costs.

[0224] Digital twin technology support: Establish a digital twin model of the vehicle to track the vehicle's health status and performance parameters in real time, and provide data support for vehicle maintenance.

[0225] In these two embodiments, the edge cache optimization method in D2D networks supported by blockchain and digital twins provides an efficient, reliable and energy-saving solution suitable for different application scenarios, from smart cities to Internet of Vehicles and autonomous driving, demonstrating its wide application potential and technical advantages.

[0226] To comprehensively evaluate the performance of our invention, we compared it with several representative benchmark algorithms. These benchmark algorithms each have their own unique characteristics and advantages, and excel when addressing similar problems. By comparing the performance of these algorithms under the same conditions, we can more clearly assess the superiority of our invention and explore its potential for improvement.

[0227] 1) Random placement: In each time slot, each content has the characteristic of random placement, which means that they may be randomly selected and distributed among different UEs.

[0228] 2) Fixed placement: The location chosen for each content remains unchanged throughout the simulation, meaning that each UE always caches the same content.

[0229] 3) Random Acquisition: Each UE acquires content in a random manner, including local acquisition, D2D acquisition, and content server acquisition. In addition, the content acquisition strategy also changes randomly in each time slot.

[0230] exist Figure 4 The convergence performance of the proposed algorithm and three baseline algorithms is shown in Figure 2. All algorithms were evaluated under default parameter settings. The evaluation of the proposed algorithm includes the convergence speed and the reward obtained, providing a comprehensive perspective on the algorithm performance. It is worth noting that the proposed algorithm exhibits excellent convergence performance, reaching a stable state around the 100th round, with the reward value stable at 8.4×10 4 By comparing the curves, the present invention clearly demonstrates this excellent performance, fully confirming that the method of the present invention is superior to the baseline algorithm.

[0231] Example 1: Real-time data processing and cache optimization in intelligent manufacturing

[0232] In the field of smart manufacturing, the numerous devices and sensors on production lines generate massive amounts of data. This data needs to be processed in real time on edge devices to ensure production continuity and efficiency. Traditional edge caching strategies often fail to effectively manage such massive amounts of data, resulting in data transmission delays and reduced system response speeds.

[0233] The present invention achieves efficient data management and transmission by applying a D2D network edge cache optimization method supported by blockchain and digital twins in an intelligent manufacturing system. The intelligent agent is deployed at multiple key nodes of the production line and uses a deep reinforcement learning algorithm to dynamically adjust the content caching and acquisition strategy so that production data can be processed and transmitted in a timely manner. By introducing TMV-Pt seed synthesis and nanoprobe detection technology, the system is able to achieve efficient caching and rapid processing of key data without affecting production efficiency. Experiments show that compared with traditional caching methods, the system improves data processing efficiency by 30% and reduces data transmission delay by 50%, significantly improving the operating efficiency and stability of the production line.

[0234] Example 2: Edge Computing and Cache Optimization in Smart Grids

[0235] In smart grid systems, distributed energy management requires real-time monitoring and optimization of electricity production, distribution, and consumption. Due to the large number and widespread distribution of nodes in the grid, traditional centralized data processing methods are prone to bottlenecks when handling large amounts of real-time data. This results in long system response times, impacting the timeliness of power dispatch and grid stability.

[0236] By applying the method of the present invention, the smart grid system can take advantage of the D2D network, distribute data processing on multiple edge nodes, and ensure the security of data transmission through blockchain technology. The intelligent agent is deployed at each power node and uses a deep reinforcement learning algorithm to dynamically adjust the cache strategy to cope with the ever-changing power demand and supply situation in the power grid. The system collects the status data of each node in real time, including user power consumption, power generation, equipment health status, etc., and generates optimized content placement and acquisition strategies to ensure that data can be processed and transmitted in a timely manner. Application results show that this method increases the response speed of the smart grid system by 40%, while increasing the optimization efficiency of power distribution by 25%, greatly enhancing the stability and reliability of the power grid.

[0237] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware portion can be implemented using dedicated logic; the software portion can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. Those skilled in the art will appreciate that the above-mentioned devices and methods can be implemented using computer-executable instructions and / or contained in processor control code, for example, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, or can be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.

[0238] The above description is only a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with this technical field within the technical scope disclosed by the present invention and within the spirit and principles of the present invention should be covered by the scope of protection of the present invention.

Claims

1. A method for optimizing edge cache in a D2D network supported by blockchain and digital twins, characterized in that: The method applies an agent's actor and critic networks to deep reinforcement learning for decision-making; initializes parameters of the policy actor network and critic network, and sets training-related hyperparameters; the agent interacts with the environment based on the current policy, executes actions, and transitions states; performs truncation, calculates estimated rewards and advantage functions; updates the actor network and critic network; and repeatedly trains until a predefined life cycle cessation condition is reached, and uses the trained policy for content placement and content acquisition. The specific steps include: S101. Initialize the agent's policy actor network π(a|s;θ) and its parameters θ, and initialize the critic network V ω (s) and its parameter ω, experience pool Maximum number of rounds T max Initialize the learning rates μ and σ of the actor network and critic network respectively; Initialize the network layout parameters, including the number of users I and the number of contents F; S102, the agent interacts with the environment and collects information about the current state s t information, including user location, estimated processing capacity and remaining energy; S103, based on the current state s t , using the old strategy network π(a t |s t θ old ) Get action a t , get content placement and acquisition strategies; S104, the agent performs action a t , observe the reward r t , and obtain the state s at the next moment t+1 ; S105, the interactive data <s t ,a t ,r t ,s t+1 >Store in the experience pool to form a trajectory sequence; S106, when the trajectory reaches the truncation length T, use the critic network V ω (s) Evaluate the terminal state and get the estimated reward R t , then, the advantage function A(s) is calculated based on the trajectory sequence t ,a t ); S107. Randomly select a set of experience tuples from the experience pool, each tuple includes state s t , action a t and the corresponding reward R t ; S108, the selected experience tuple is used to train the critic network, and the parameters ω of the critic network are updated during the training process so that the loss function L critic (ω) minimum; S109, randomly select a set of tuples from the experience pool for training the actor network; S110, constructing the strategy objective function L clip (θ), use the gradient ascent method to update the parameters θ of the actor network so that the policy objective function L clip (θ) maximized; S111. Continue to iterate the above steps until the predefined life cycle growth stop condition is reached. In each iteration, the agent contacts the environment, collects trajectory data, and uses this data to adjust the parameters of the actor network, thereby continuously improving the strategy and improving the performance of the agent.

2. The edge cache optimization method in a D2D network supported by blockchain and digital twins according to claim 1, characterized in that: In S102, the agent interacts with the environment and initializes the system state s at the beginning of each cycle. t , specifically expressed as: in, where u i (t-1)={x i (t-1),y i (t-1)} is the coordinate of UE i at the end of time slot t-1; in Indicates whether UE i requests content f. Note that a UE can only request one content in time slot t; in is the preference of UE i for content f at time slot t; represents the estimated processing capability of UE i in time slot t; represents the remaining energy of each UE at the beginning of time slot t, specifically, represents the fast fading coefficient of the D2D communication link between UE i and UE j at time slot t; represents the distance between UE i and UE j at time slot t; represents the Rayleigh fast fading coefficient between UE i and the base station at time slot t; represents the distance between UE i and the base station at time slot t; Indicates whether there is a direct physical link between UE i and UE j at time slot t; Indicates whether there is a direct physical link and social connection between UE i and UE j at time slot t; represents the set of neighbor users of UE i at time slot t; represents the number of neighboring users of UE i at time slot t; represents the tanimoto coefficient between UE i and UE j at time slot t; represents the D2D sharing probability between UE i and UE j at time slot t; ΔT vali (t) represents the block verification delay deviation; represents the transaction size of UE i at time slot t; represents the number of signature verification failures received from UE i at time slot t; represents the reputation value of UE i in time slot t-1; represents the growth index of UE i’s lifetime at time slot t; represents the temporary lifetime of each UE in time slot t, and its initial lifetime is 0; In each time slot t, the agent determines its action, denoted as in: Represents content placement strategy; Indicates the content acquisition strategy.

3. The edge cache optimization method in a D2D network supported by blockchain and digital twins according to claim 1, characterized in that: In step S103, the agent executes the generated action, obtains an immediate reward based on the executed action, and transfers the environment state to the next state. The calculation consensus of the reward is as follows:

4. The edge cache optimization method in a D2D network supported by blockchain and digital twins according to claim 1, characterized in that: S106: Calculate the advantage function: A(s t ,a t )=d t +gd t+1 +...+c T-t+1 d t+1 , Represents the advantage situation under the truncated trajectory of length T; the advantage function refers to the advantage or relative benefit obtained by the agent taking a certain action relative to other possible actions, where δ t =r t +γV(s t+1 )-V(s t ), Among them, V(s t ) is the output of the critic network; based on this, A(s t ,a t ) is rewritten as A(s t ,a t )=R(t)-V(s t ), Where R(t) is used to estimate the expected return at time step t.

5. The edge cache optimization method in a D2D network supported by blockchain and digital twins according to claim 1, characterized in that: S108: Update the parameters of the critic network. The update formula is as follows: in, Where N represents the number of mini-batches. Indicates the status of the nth sample in the mini-batch.

6. The edge cache optimization method in a D2D network supported by blockchain and digital twins according to claim 1, characterized in that: S110: Update the parameters of the actor network. The update formula is as follows: Among them, η is a hyperparameter, L clip The (θ) gradient is expressed as 7. A system for edge cache optimization in a D2D network supported by blockchain and digital twins, applied to the edge cache optimization method in a D2D network supported by blockchain and digital twins as described in any one of claims 1 to 3, characterized in that: include: An agent module, which configures the parameters used to initialize the policy actor network and the critic network and sets the training-related hyperparameters; an interaction module configured to enable the agent to interact with the environment, perform actions, and perform state transitions; an evaluation module configured to truncate trajectories and compute estimated rewards and advantage functions; An update module is configured to update the actor network and the critic network until a predefined life stop growth condition is reached and use the trained policy for content placement and content acquisition.

8. The system according to claim 7, wherein: The agent module further comprises: An initialization unit, which is configured to initialize the agent's policy actor network and its parameters, the critic network and its parameters, the experience pool, and the maximum number of rounds; A network layout unit is configured to initialize network layout parameters, including the number of users and the number of contents.

9. The system according to claim 7, wherein: The interaction module further includes: a state collection unit configured to initialize the system state at the beginning of each cycle and collect information about the current state, including user location, estimated processing capacity, remaining energy, etc.; a policy generation unit configured to generate content placement and acquisition policies based on a current state by utilizing an old policy network to obtain actions; A reward observation unit is configured to allow the agent to perform actions and observe rewards, obtain the state at the next moment, and store the interaction data in the experience pool to form a trajectory sequence.

10. The system according to claim 7, wherein: The evaluation module further comprises: an evaluation unit configured to evaluate the terminal state using a critic network and calculate an advantage function when the trajectory reaches a truncation length; A training unit is configured to randomly select experience tuples from the experience pool, train the parameters of the critic network and the actor network, and update the parameters of the actor network using the gradient ascent method according to the policy objective function until a predefined life cycle stop growth condition is reached.

Images