Digital twin deployment method based on multi-target deep reinforcement learning

Abstract

The invention discloses a digital twinborn deployment method based on multi-target deep reinforcement learning, and the method comprises the steps: carrying out the analysis of a digital twinborn edge network, constructing the state, action and reward functions of a multi-target deep reinforcement learning model, introducing a connection relation between a user and an intelligent reflection surface in the state, and obtaining a multi-target deep reinforcement learning model; a penalty function of a constraint term is introduced into a reward function, then a multi-target deep reinforcement learning model is set and trained, state information of a current digital twin edge network is acquired in real time, and target edge servers deployed by each digital twin are determined by adopting the trained multi-target deep reinforcement learning model. According to the method, a plurality of optimization objectives are set, the deployment strategy of the digital twin is determined by adopting multi-objective deep reinforcement learning, and the rationality and the working performance of the digital twin edge network are improved.

Description

Technical Field

[0001] The present invention belongs to the field of edge network technology, and more specifically, relates to a digital twin deployment method based on multi-objective deep reinforcement learning. Background Art

[0002] With the rapid development of technologies such as the Internet of Things (IoT), next-generation mobile communications (6G), and artificial intelligence (AI), urban mobile user data and information are growing exponentially. Mobile users monitor, predict, analyze, and support decisions in physical spaces by interacting with virtual spaces, which has attracted widespread attention from researchers. Digital twins (DT), as a transformative technology that combines the virtual and physical worlds, are considered to be one of the most promising technologies to address the above challenges. However, in actual deployment, it is difficult to simultaneously meet the requirements of digital twins for continuous low latency and large amounts of computing resources, which poses a challenge to the placement of user digital twins.

[0003] In recent years, the rapid rise of mobile edge computing (MEC) has brought new opportunities for the placement of digital twins. By bringing low latency and rich computing resources to the edge of the network close to the user, the distance between the user and the edge server is greatly shortened. In the combination of digital twins and mobile edge computing, the MEC distributed edge nodes provide sufficient resources for the placement of user digital twins while effectively maintaining their continuous low-latency interaction. Therefore, digital twins, as one of the most promising technologies in 6G, will be widely placed in edge networks to reduce the real-time interaction delay between users and their digital twins. Summary of the invention

[0004] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a digital twin deployment method based on multi-objective deep reinforcement learning, set multiple optimization objectives, and use multi-objective deep reinforcement learning to determine the deployment strategy of the digital twin, so as to improve the rationality and working performance of the digital twin edge network.

[0005] In order to achieve the above-mentioned object of the invention, the digital twin deployment method based on multi-objective deep reinforcement learning of the present invention comprises the following steps:

[0006] S1: Get the user set in the digital twin edge network as U = {U1, U2, ..., U N},U n represents the nth user, n=1,2,…,N, N represents the number of users, and the set of intelligent reflective surfaces is I={I1,I2,…,I K}, I k represents the kth smart reflective surface, k = 1, 2, ..., K, K represents the number of smart reflective surfaces, and the edge server set is E = {E1, E2, ..., EM}, E m represents the mth edge server, m = 1, 2, ..., M, where M represents the number of edge servers;

[0007] S2: For each user U n Set up a multi-objective deep reinforcement learning model with the following parameters:

[0008] State: State s(t) = {su1(t),…,su N (t),si1(t),…,si K (t),se1(t),…,se M (t)}, where su n (t) represents user U n The state information at time t includes position, velocity, beamforming vector, si k (t) represents the smart reflective surface I k Status information, including position, reflection parameters, se m (t) represents the edge server E m Status information, including location and total resource quantity;

[0009] Action: Set the action where x n,m =1 indicates user U n The digital twin is deployed to the edge server E m , otherwise x n,m =0;

[0010] Reward function: The calculation formula of the reward function R is as follows:

[0011]

[0012] Where I represents a three-dimensional unit vector, f = [T syn (t),EC com (t),-L Dos (t)] represents the objective function vector, T syn (t) represents the overall synchronization delay, which is calculated as follows:

[0013] Computational intelligent reflective surface assisted user U n and edge server E m The signal transmission rate between n,m (t):

[0014]

[0015] Among them, p n,m (t) represents the user U under the consideration of path loss n For edge server Em The transmission power is calculated as follows:

[0016]

[0017] γ κ Noise ratio threshold, d0 is the reference distance, α is the path loss exponent, d n,m (t) represents the user U at time t n and edge server E m The distance between them; N0 represents the noise power spectrum density, B represents the channel bandwidth, ν n (t) is the time when user U n The beamforming vector of Indicates that user U is assisted by the intelligent reflective surface n and edge server E m The channel vector of:

[0018]

[0019] Among them, h n,m (t) represents user U n and edge server E m Channel gain between:

[0020]

[0021] λ α represents the path fading factor, ε κ Denotes the distortion factor, d n,m (t) represents the user U at time t n and edge server E m The distance between is user U at time t n With smart reflective surface I k The channel gain between n,k (t) represents the user U at time t n To Smart Reflective Surface I k The distance Represents intelligent reflective surface I k and edge server E m The channel gain between k,m Represents intelligent reflective surface I k To the edge server E m The distance between k Represents intelligent reflective surface I k Reflection parameters of

[0022] Calculate the user U at time t n Transmit real-time operation data n To the edge server Em Transmission delay

[0023]

[0024] Among them, |D n |Represents real-time running data D n size;

[0025] Calculate the user U at time t n Transmit real-time operation data n To the edge server E m Processing time on

[0026]

[0027] f n,m (t) represents the user U at time t n Consumption edge server E m computing resources;

[0028] The following formula is used to calculate the user U per unit time at time t n Synchronization delay of digital twin for:

[0029]

[0030] ΔD n Represents user U n The size of data transmitted per unit time;

[0031] The overall interaction delay T is calculated using the following formula: syn (t):

[0032]

[0033] EC com (t) represents the total interaction energy consumption, which is calculated as follows:

[0034] The following formula is used to calculate user U n For edge server E m Energy consumption EC for real-time interactive data n,m (t):

[0035]

[0036] Then the overall interaction energy consumption EC is calculated using the following formula com (t):

[0037]

[0038] LDos (t) represents the total service time, which is calculated as follows:

[0039] Calculate user U n Through the edge server E m Path length J within the service area n,m (t):

[0040]

[0041] Represents user U n The position coordinates at time t, Represents the edge server E m The position coordinates at time t, R n,m Indicates that the edge server E m is the center of the circle and the radius is circle;

[0042] The edge server E is calculated using the following formula m For user U n Length of service provided

[0043]

[0044] in, Represents user U n The moving average speed of

[0045] Then use the following formula to calculate the total service time L Dos (t):

[0046]

[0047] represents the penalty term, which is calculated as follows:

[0048]

[0049] Among them, μ represents the penalty coefficient, KS[g] represents the aggregation term of the constraint term, and its expression is as follows:

[0050]

[0051] ρ is a pre-set positive constant, g i represents the i-th constraint, i = 1, 2, ..., 6, g max Represents the maximum value of all constraints. Each constraint expression is as follows:

[0052] g1:-p n,m (t)≤0

[0053] g2:p n.m (t)-p max ≤0

[0054] g3:

[0055] g4:f n,m (t)-F m ≤0

[0056] g5:

[0057] g6:

[0058] Among them, p max Indicates the maximum wireless transmission power, Indicates the maximum delay, F m Represents the edge server E m The total amount of resources, κ is a constant greater than 0 and less than 1, a n,k is a Boolean variable, a n,k =1 indicates user U n Smart Reflective Surface I k Assist in information interaction with edge servers, otherwise a n,k =0;

[0059] S3: Construct a multi-objective deep reinforcement learning model, including a policy network θ and an evaluation network Q. The evaluation network Q is implemented based on a deep neural network. The state and preference vectors are input into the deep neural network, and the output size is 3N. A The Q value vector, N A Indicates the size of the action space, and then trains the multi-objective deep reinforcement learning model. The specific method is:

[0060] S3.1: Randomly initialize the parameters of the policy network θ and the evaluation network Q, and copy them as the target policy network θ′ and the target evaluation network Q′;

[0061] S3.2: Establish a playback buffer D and initialize it;

[0062] S3.3: Divide the value space Ω of the preference vector W = (w1, w2, w3) into C preference subspaces c=1,2,…,C,w j represents the weight of the jth optimization objective;

[0063] S3.4: Let training round z = 1;

[0064] S3.5: Initialization time t=0, obtain initial state s(0);

[0065] S3.6: For each preference subspace Start a subprocess to generate experience samples. The specific method is:

[0066] From the current subspace A preference vector W = (w1, w2, w3) is sampled from the state s(t); the policy network θ selects the action a(t) according to the state s(t). n (t) obtain the state s(t+1) and reward R(t) of the next moment; store the state transition data (s(t), a(t), R(t), W, s(t+1)) as experience samples in the replay buffer D; if the number of experience samples in the experience replay buffer pool D exceeds its predetermined capacity, delete the excess experience samples according to the established rules;

[0067] Then randomly sample N from all experience samples in the current experience replay cache pool D W Preference vector W v ,v=1,2,…,N W ; Then N W state transition data (s(t), a(t), R(t), W v ,s(t+1)) is stored as an experience sample in the playback buffer D;

[0068] S3.7: Sample B transition state data from the experience replay buffer pool D;

[0069] S3.8: using an interpolator to interpolate each preference vector in the B conversion state data, aligning the dimension of the preference vector with the dimension of the Q value vector, and obtaining an interpolated preference vector;

[0070] S3.9: For the B transition state data after preference vector interpolation, use the target evaluation network Q′ to estimate the action and obtain the value y of the action b :

[0071]

[0072] Where γ represents the discount factor, R b represents the reward in the bth transition state data, represents the interpolated preference vector in the b-th transition state data, a b ′、s b ′ respectively represent the action and state of the b-th transition state data at the next moment, Q′() represents the Q value vector obtained by the target evaluation network Q′, S c () indicates similarity, and the superscript T indicates transposition;

[0073] Calculate the loss function L(θ) and update the parameters of the policy network;

[0074] S3.10: Determine whether t<T, where T represents the preset maximum time step. If so, proceed to step S3.11; otherwise, proceed to step S3.12;

[0075] S3.11: Let t=t+1, and return to step S3.6;

[0076] S3.12: Calculate the gradient of the evaluation network Q and update the parameters of the evaluation network;

[0077] S3.13: Determine whether z < Z, where Z represents the preset maximum number of training rounds. If yes, proceed to step S3.14, otherwise the training ends.

[0078] S3.14: Determine whether z%η=0, where η represents the target network update period. If yes, proceed to step S3.15, otherwise proceed to step S3.16;

[0079] S3.15: Update target network parameters:

[0080] Update the target evaluation network and target strategy network parameters:

[0081] Q′←τQ+(1-τ)Q′

[0082] θ′←τθ+(1-τ)θ′

[0083] Among them, τ represents the preset weight.

[0084] S3.16: Let z=z+1, and return to step S3.5;

[0085] S4: Collect the status information of each user, each intelligent reflective surface and each edge server in the current digital twin edge network to construct the state s, and then use the policy network trained in step S3.3 to determine the optimal action a, so as to determine the edge server serial number where each user's digital twin is deployed and obtain the digital twin deployment strategy.

[0086] The present invention is based on a digital twin deployment method based on multi-objective deep reinforcement learning. It analyzes the digital twin edge network, constructs the state, action and reward function of the multi-objective deep reinforcement learning model, introduces the connection relationship between the user and the intelligent reflective surface in the state, introduces the penalty function of the constraint item in the reward function, and then sets and trains the multi-objective deep reinforcement learning model to obtain the current state information of the digital twin edge network in real time, and uses the trained multi-objective deep reinforcement learning model to determine the target edge server for each digital twin deployment.

[0087] The present invention has the following beneficial effects:

[0088] 1) The digital twin edge network model constructed by the present invention combines user digital twins, mobile edge computing and intelligent reflective surface technology (IRS), uses IRS technology to optimize the wireless communication link of the system, and uses distributed computing and storage resources in the edge network to optimize the maintenance link of the digital twin;

[0089] 2) The present invention sets interaction delay, energy consumption and service duration as optimization targets in the reward function setting of the multi-objective deep reinforcement learning model, so that the final digital twin deployment strategy is more reasonable;

[0090] 3) In the training process of the multi-objective deep reinforcement learning model, the present invention adopts the Kreisselmeier-Steinhauser (KS) constraint aggregation method to aggregate multiple constraints into a single constraint, and further transforms the single constraint into a penalty function in the objective function to achieve the transformation of the unconstrained problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0091] Figure 1 It is a schematic diagram of the digital twin edge network structure of the present invention;

[0092] Figure 2 It is a user wireless communication diagram based on the smart reflective surface in the present invention;

[0093] Figure 3 It is a flowchart of a specific implementation method of the digital twin deployment method based on multi-objective deep reinforcement learning of the present invention;

[0094] Figure 4 It is a training flow chart of the multi-objective deep reinforcement learning model in the present invention;

[0095] Figure 5 It is a flow chart of dynamic adjustment of penalty coefficient in this embodiment. DETAILED DESCRIPTION

[0096] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0097] The present invention considers a digital twin edge network that combines digital twin, mobile edge computing and smart reflective surface technologies. Figure 1 This is a schematic diagram of the digital twin edge network structure of the present invention. Figure 1As shown. The architecture mainly includes a physical layer and a digital twin network layer. The physical layer includes the cloud-edge-end three layers in the mobile edge computing architecture. The terminal devices include mobile users and intelligent reflective surfaces (IRS) installed on the building surface. The edge layer only considers base stations with edge servers. In this system, the intelligent reflective surface is responsible for assisting the wireless communication service between mobile users and edge servers, while the edge server provides maintenance services for users and IRS digital twins. The digital twin network is located in the core cloud layer and is used to store service digital twins of all users, intelligent reflective surfaces, and edge servers. It relies on a large amount of computing and storage resources configured in the core cloud to execute the process of target server selection and the formulation of the strategy for associating the intelligent reflective surface with mobile users. The present invention defines the user set as U={U1,U2,...,U N},U i represents the nth user, n=1,2,…,N, N represents the number of users, and the set of intelligent reflective surfaces is I={I1,I2,…,I K}, I k represents the kth smart reflective surface, k = 1, 2, ..., K, K represents the number of smart reflective surfaces, and the edge server set is E = {E1, E2, ..., E M}, E m represents the mth edge server, m = 1, 2, ..., M, and M represents the number of edge servers.

[0098] Since the core cloud layer where the digital twin network is located is far away from the end user, real-time updating of the digital twin of the end user requires a lot of resources (computing, storage and communication resources), and it is difficult to meet the delay constraints between the user and the digital twin. In order to meet the requirements of digital twins for delay and resources, the present invention divides the digital twins of users and smart reflective surfaces into two types: entity digital twins and service digital twins.

[0099] The entity digital twin is a complete copy of the user's status information. The present invention stores it in the edge server as the user's main digital twin to meet the needs of continuous low-latency information interaction of the digital twin. The entity digital twin keeps the status information synchronized with the user by interacting with the user in real time to achieve real-time monitoring of the user's status. Then the user U n The entity digital twin of can be written as:

[0100] νκ dt (u n )={su n (t),su n (t+1)}

[0101] Among them, n(t),su n (t+1) represents user U n The state information at time t and t+1, s n (t)=(P n (t),V n (t),R n (t)), where Represents user U n The position coordinates at time t, V n (t) represents user U n The velocity at time t, R n (t) represents user U n The path at time t.

[0102] The service digital twin only synchronizes part of the user's status information and stores it in the digital twin network. With the advantage of information interaction between digital twins, different twins in the digital twin network exchange information with each other to assist in the selection of target servers during placement and the formulation of association strategies between smart reflective surfaces and mobile users. n The service digital twin can be written as:

[0103]

[0104] in, Respectively represent the user U in the service digital twin n Status information at time t and t+1. For different service types The digital twin can have status information of different parts of the physical digital twin to reduce system resource loss and improve system operation efficiency.

[0105] Similar to the above, the present invention defines a smart reflective surface I k and edge server E m The digital twins are νκ dt (I k ) and νκ dt (E m ), then the digital twin network in the core cloud layer can be expressed as:

[0106]

[0107] In order to realize the deployment of user entity digital twins, the present invention establishes the system's communication model, interaction cost model and service duration model, and then formulates a multi-objective optimization problem based on the interaction cost and service duration between the user and the edge server.

[0108] ·Communication model

[0109] The present invention considers the signal attenuation problem caused by dense urban buildings, installs intelligent reflective surfaces on the building surfaces, and the wireless communication between users and edge servers is completed with the assistance of the intelligent reflective surfaces near the edge servers. Figure 2 is a user wireless communication diagram based on the intelligent reflective surface in the present invention. Figure 2 As shown, at time t, user U n and edge server E m The direct channel vector between (i.e., wireless transmission channels between users and edge servers without the assistance of smart reflective surfaces), including line-of-sight (LoS) path components and non-line-of-sight (NLoS) path components UserU n and edge server E m The channel gain h n,m (t) is as follows:

[0110]

[0111] Among them, ζ represents the Rice factor, λα represents the path fading factor, and ε κ Denotes the distortion factor, d n,m (t) represents the user U at time t n and edge server E m The distance between.

[0112] The present invention introduces a smart reflective surface to establish a wireless channel between the user and the edge server. The smart reflective surface can provide a channel gain to suppress the signal attenuation problem caused by the building. n and edge server E m The channel vector It can be expressed as:

[0113]

[0114] in, is user U at time t n With smart reflective surface I k The channel gain between n,k (t) represents the user U at time t n To Smart Reflective Surface I k The distance Represents intelligent reflective surface I k and edge server E m The channel gain between k,m Represents intelligent reflective surface I k To the edge server E m The distance betweenk Represents intelligent reflective surface I k Reflection parameters:

[0115]

[0116] θ r,p and Represent smart reflective surface I k The phase shift and amplitude reflection coefficient of element p, θ r,p ∈[0,2π], p=1, 2, ..., P, where P represents the number of elements of the smart reflective surface.

[0117] For the user U under the assistance of intelligent reflective surface n and edge server E m The signal transmission rate between n,m (t) can be expressed as:

[0118]

[0119] Among them, p n,m (t) represents the user U under the consideration of path loss n For edge server E m The transmission power is calculated as follows:

[0120]

[0121] Among them, γ κ Noise ratio threshold, d0 is the reference distance, and α is the path loss exponent.

[0122] N0 represents the noise power spectral density, B represents the channel bandwidth, ν n (t) is the time when user U n The beamforming vector (e.g., ν n (t)h n,m (t)=1).

[0123] Interaction cost model

[0124] The main optimization goal of this invention is to minimize the interaction cost between the user and his entity digital twin, which mainly considers the synchronization delay and energy consumption. n The entity digital twin is placed on the edge server E m To provide maintenance services. User U n Through the intelligent reflective surface I r , real-time operation data D n Transmit to edge server E m . Edge Server E m Receive real-time operation data Dn After that, it is processed and analyzed, and the processed data is used to update the entity digital twin to support the entity digital twin and user U n The synchronization delay of the system mainly includes the transmission delay of real-time interactive data between users and edge servers and the calculation delay of processing data on edge servers for model synchronization.

[0125] 1) Transmission delay: When user U n The entity digital twin is stored on the edge server E m During maintenance, user U n Low-latency real-time information interaction with the digital twin. n Real-time operation data D n Transmit to edge server E m At time t, user U n Transmit real-time operation data n To the edge server E m Transmission delay It can be expressed as:

[0126]

[0127] Among them, |D n |Represents real-time running data D n size.

[0128] 2) Calculation delay: User U n The real-time running data D n After being transmitted to edge server j, edge server E m The data needs to be processed and analyzed to update the digital twin to keep the status of the user and the digital twin synchronized. Assume that the edge server E m The total computing resources are F m , it can provide digital twin maintenance services to multiple users at the same time. n Consumption edge server E m The computing resources are f n,m (t), then at time t, user U n Transmit real-time operation data n To the edge server E m Processing time on It can be expressed as:

[0129]

[0130] The present invention assumes that user U n The size of the data transmitted per unit time is ΔD n , then the user U per unit time at time t nSynchronization delay of digital twin for:

[0131]

[0132] As the user moves, the synchronization delay may continue to increase. In order to ensure that the user can continue to interact with low latency, the present invention sets the maximum delay that the user and his digital twin can tolerate to be When the synchronization delay meets When the migration action is triggered, the target server selection strategy will be formulated.

[0133] Considering the interaction delay of the system, the energy consumption generated by real-time interaction data is also an important indicator to measure the performance of the system. To this end, the present invention defines the energy consumption of the system real-time interaction data EC n,m (t) is:

[0134]

[0135] Service time model

[0136] Since maintaining digital twins requires a lot of computing resources, frequent migrations may significantly increase the risk of state deviation between users and physical digital twins, thereby reducing the reliability of the service and also bringing additional cost overhead to the system. In order to solve the above problems, the present invention proposes a service duration model to evaluate the length of time that different edge servers provide maintenance services for users' physical digital twins. Taking the service duration of each edge server as the optimization target, an edge server that can provide a longer service duration is selected to reduce the number of migrations of the user's digital twin.

[0137] The wireless communication model between the user and the entity digital twin is: user-intelligent reflective surface-edge server. Among them, the intelligent reflective surface determines the scope of service provided by the edge server to the user and its digital twin. Based on the analysis of the above communication model and interaction cost model, it can be concluded that the system delay is related to the user U n To the edge server E m The distance d n,m (t) is positively correlated.

[0138] The present invention defines the maximum delay that the system can tolerate Next, user U n With edge server E m The distance is Based on this definition, the edge server E m For User U n Service Range n,m It is defined as the edge server E in the two-dimensional plane m is the center of the circle and the radius is circle.

[0139] According to the service range of each edge server, it is assumed that the path length of the user within the service range of the edge server is a set of multiple Euclidean distances. n Through the edge server E m The total path length J within the service area n,m (t) can be written as:

[0140]

[0141] Represents user U n The position coordinates at time t, Represents the edge server E m The position coordinates at time t.

[0142] Assume that user U n The moving average speed is The edge server E m Length of service provided It can be calculated by the following formula:

[0143]

[0144] in, Represents user U n The moving average speed of .

[0145] Based on the interaction cost between users and digital twins assisted by intelligent reflective surfaces, the present invention takes service time as an important indicator to reduce the migration frequency, aiming to weigh the relationship between interaction cost and service time, and formulate a target server selection strategy adaptively according to user preferences. The strategy formulation of the present invention is only used when the edge server where the current digital twin is located cannot meet the delay constraint. Executed under the circumstances.

[0146] Considering the user's synchronous interaction delay and the overall synchronization delay T of the computing delay system syn (t) can be written as:

[0147]

[0148] Similarly, the total interaction energy consumption EC com (t) can be written as:

[0149]

[0150] In order to measure the migration frequency of digital twins, this paper introduces the overall service time L Dos(t) can be expressed as an evaluation indicator:

[0151]

[0152] The target edge server selection problem in the digital twin placement process aims to balance the relationship between interaction delay, energy consumption and service duration, and consider the constraints of edge server computing resources, transmission energy consumption and interaction delay to maximize user experience. Based on the above, the multi-objective optimization problem of the present invention can be expressed as:

[0153] P1:min[T syn (t),EC com (t)]

[0154] max[L Dos (t)]

[0155] stC1:0≤p n,m (t)≤p max

[0156] C2:

[0157] C3:f n,m (t)≤F m

[0158] C4:

[0159] Among them, C1 represents the upper and lower bounds of the user's wireless transmission power, and C2 represents the constraint on the synchronization delay between the user and the digital twin to ensure the synchronization delay The maximum delay that can be tolerated In addition, C3 and C4 are used to ensure that the computing resources used by users on the edge server do not exceed its total resources.

[0160] In order to deal with the signal attenuation problem caused by dense urban buildings, the present invention formulates the association problem between users and smart reflective surfaces to maintain the smart reflective representation to continuously optimize the quality of the wireless channel between users and edge servers. Since it is necessary to ensure that the communication between users and MEC servers is connected through IRS, the connectivity between users and IRS needs to be considered. For this, the weight matrix A = [a n,k ] to represent user U n and smart reflective surfaces I k The relationship between:

[0161]

[0162] when a n,k =1, indicating that user U n Smart Reflective Surface I kAssist in information interaction with edge servers, otherwise a n,k =0.

[0163] According to the description of the relationship between the user and the smart reflective surface, the optimization problem P1 can be rewritten as:

[0164] P2:min[T syn (t),EC com (t)]

[0165] max[L Dos (t)]

[0166] stC1:0≤p n,m (t)≤p max

[0167] C2:

[0168] C3:f n,m (t)≤F m

[0169] C4:

[0170] C5:

[0171] C6:a n,k ={0,1},

[0172] C5 ensures that there is an association between the user and the smart reflective surface. C6 indicates a n,k A Boolean variable can only exist in two states: 0 and 1.

[0173] As can be seen from question P2, the goal of the present invention is to achieve a balance and compromise between the three goals of interactive synchronization delay, energy consumption and service duration under the constraints of computing resources, transmission power and interactive synchronization delay. However, the current multi-objective deep reinforcement learning (MODRL) method is mainly used to solve unconstrained optimization problems. When designing the objective function in reinforcement learning, corresponding penalty terms are usually added to deal with the various constraints in the problem. By setting appropriate constraint violation values ​​to change the value of the objective function, it is ensured that the strategy obtained by reinforcement learning can optimize the target while also taking into account the requirements of meeting all problem constraints.

[0174] To this end, the present invention introduces the Kreisselmeier-Steinhauser (KS) function, a constraint aggregation method based on gradient optimization, which aggregates each original constraint into a single constraint containing all constraints through a continuous and smooth function according to the degree of violation of each original constraint. The transformation of this process needs to be carried out under the constraint term in standard form, that is, the inequality gi ≤0, i=1,2,…,6. According to the C1-C6 constraints in the optimization problem P2, it is noted that the constraints C5 and C6 cannot be directly converted into the standard form. In this regard, the present invention rewrites the constraints C5 and C6 as Here κ is a constant greater than 0 and less than 1. Then the optimization problem P2 can be rewritten as:

[0175] P3:min[T syn (t),EC com (t)]

[0176] max[L Dos (t)]

[0177] stC1:0≤p n,m (t)≤p max

[0178] C2:

[0179] C3:f n,m (t)≤F m

[0180] C4:

[0181] C7:

[0182] Further rewrite the constraints C1-C4, C7 into standard form, that is, the constraints are expressed as follows:

[0183] g1:-p n,m (t)≤0

[0184] g2:p n,m (t)-p max ≤0

[0185] g3:

[0186] g4:f n,m (t)-F j ≤0

[0187] g5:

[0188] g6:

[0189] According to the constraints in the above standard form, they are aggregated based on the gradient optimized constraint aggregation method, and the aggregation function is as follows:

[0190]

[0191] Among them, gi represents the i-th constraint, g max represents the maximum value of all constraints, and ρ is a preset positive constant, which controls the degree of aggregation of different constraint functions.

[0192] The properties of the KS function on the boundary are as follows:

[0193]

[0194] That is, by limiting the value of the constraint function KS[g] to the loosest constraint g of the original problem max After considering the maximum contribution of all constraints This ensures that the comprehensive constraints do not exceed the influence of the single most stringent constraint, and at the same time can fuse multiple discrete constraints of the original problem into a continuous and smooth comprehensive constraint function.

[0195] Since the optimization objective in the P3 problem involves both maximization and minimization, in order to unify the problems, the present invention maximizes L Dos (t) is the minimization optimization objective - L Dos (t), then the optimization problem P3 can be rewritten as:

[0196] P4:min[T syn ,E com ,-L Dos ]

[0197] stC1~C4,C7

[0198] In order to further eliminate the comprehensive single constraint function, the penalty term is set in the objective function based on the constraint function as follows:

[0199]

[0200] Among them, μ represents the penalty coefficient. By setting an appropriate μ value, the approximate optimal penalty function value is determined. Indicates that the current deployment plan meets all constraints. On the contrary, if The larger the value of , the more serious the violation of the constraint, and the adverse impact of the violation of the constraint on the overall system optimization goal will also increase. Adding the penalty term to the objective function can rewrite the optimization problem as

[0201] P5:

[0202] Where I represents a three-dimensional unit vector.

[0203] In this regard, the present invention defines an auxiliary function to represent the unconstrained optimization problem P5:

[0204]

[0205] Among them, f j (X) represents the value of the jth optimization objective function under the current deployment scheme X, j = 1, 2, 3, f1(X) = T syn , f2(X)=E com , f2(X)=L Dos .

[0206] Based on the above derivation and analysis, the present invention proposes a digital twin deployment method based on multi-objective deep reinforcement learning. Figure 3 This is a flowchart of a specific implementation method of the digital twin deployment method based on multi-objective deep reinforcement learning of the present invention. Figure 3 As shown, the specific steps of the digital twin deployment method based on multi-objective deep reinforcement learning of the present invention include:

[0207] S301: Obtain digital twin edge network information:

[0208] Get the user set in the digital twin edge network as U = {U1, U2, ..., U N},U i represents the nth user, n=1,2,…,N, N represents the number of users, and the set of intelligent reflective surfaces is I={I1,I2,…,I K}, I k represents the kth smart reflective surface, k = 1, 2, ..., K, K represents the number of smart reflective surfaces, and the edge server set is E = {E1, E2, ..., E M}, E m represents the mth edge server, m = 1, 2, ..., M, where M represents the number of edge servers;

[0209] S302: Setting multi-objective deep reinforcement learning model parameters:

[0210] In the present invention, a multi-objective deep reinforcement learning model is set up to approximately learn the Pareto frontier strategy covering the entire preference space Ω by learning the value of each optimization objective (including interaction delay, energy consumption and service duration) under different preference conditions. The multi-objective deep reinforcement learning model can essentially be regarded as a multi-objective Markov decision process (MOMDP), which can be expressed as follows:

[0211]

[0212] in, represents the state space of the system, represents the space of actions that the system can take, represents the probability distribution of taking action a to transfer to s′ in state s, that is, the state transition probability. In addition, since multi-objective deep reinforcement learning needs to simultaneously learn multiple task objectives with different rewards, the reward signal is no longer a scalar, but a vector form To indicate that here Represent the rewards corresponding to Z different optimization objectives. Ω represents all possible preference spaces of the system. Each preference vector W can be regarded as a target weight vector, which consists of Z weight elements w j It is used to measure the priority of different objectives in the optimization process and meets In addition, f W Indicates that under a given preference vector W, the multi-objective reward vector The scalarization process can be written as

[0213] The parameters of the multi-objective deep reinforcement learning model in the present invention are set as follows:

[0214] State: In the present invention, the state s(t) = {su1(t), ..., su N (t),si1(t),…,si K (t),se1(t),…,se M (t)}, where su n (t) represents user U n The state information at time t includes position, velocity, beamforming vector, si k (t) represents the smart reflective surface I k Status information, including position, reflection parameters, se m (t) represents the edge server E m Status information of the resource, including location and total resource quantity.

[0215] Action: In the digital twin edge network of the present invention, the action performed by the edge server as an agent includes the target edge server selection vector for the user's digital twin placement, i.e., action where x n,m =1 indicates user U n The digital twin is deployed to the edge server E m , otherwise x n,m =0.

[0216] Reward function: The reward function in this invention mainly includes the penalty term converted from the constraint term And the three optimization goals of the system, namely, the interaction delay T syn (t), interaction energy consumption EC com (t) and service duration L Dos (t), so the reward function R is calculated as follows:

[0217]

[0218] Where I represents a three-dimensional unit vector, f = [T syn (t),EC com (t),-L Dos (t)] represents the objective function vector, T syn (t) represents the overall synchronization delay, EC com (t) represents the total interaction energy consumption, L Dos (t) represents the total service time. The total interaction delay is T syn The calculation method of (t) is as follows:

[0219] Computational intelligent reflective surface assisted user U n and edge server E m The signal transmission rate between n,m (t):

[0220]

[0221] in,

[0222] p n,m (t) represents the user U under the consideration of path loss n For edge server E m The transmission power is calculated as follows:

[0223]

[0224] γ κ Noise ratio threshold, d0 is the reference distance, α is the path loss exponent, d n,m (t) represents the user U at time t n and edge server E m The distance between them; N0 represents the noise power spectrum density, B represents the channel bandwidth, ν n (t) is the time when user U n The beamforming vector of Indicates that user U is assisted by the intelligent reflective surface n and edge server E m The channel vector of:

[0225]

[0226] Among them, h n,m (t) represents user U n and edge server E m Channel gain between:

[0227]

[0228] λ α represents the path fading factor, ε κ Denotes the distortion factor, d n,m (t) represents the user U at time t n and edge server E m The distance between is user U at time t n With smart reflective surface I r The channel gain between n,r (t) represents the user U at time t n To Smart Reflective Surface I r The distance Represents intelligent reflective surface I r and edge server E m The channel gain between r,m Represents intelligent reflective surface I r To the edge server E m The distance between r Represents intelligent reflective surface I r Reflection parameters of

[0229] Calculate the user U at time t n Transmit real-time operation data n To the edge server E m Transmission delay

[0230]

[0231] Among them, |D n |Represents real-time running data D n size;

[0232] Calculate the user U at time t n Transmit real-time operation data n To the edge server E m Processing time on

[0233]

[0234] f n,m (t) represents the user U at time t n Consumption edge server E m computing resources.

[0235] The following formula is used to calculate the user U per unit time at time t n Synchronization delay of digital twins for:

[0236]

[0237] ΔD n Represents user U n The size of data transmitted per unit time;

[0238] The overall interaction delay T is calculated using the following formula: syn (t):

[0239]

[0240] Total interactive energy consumption EC com The calculation method of (t) is as follows:

[0241] The following formula is used to calculate user U n For edge server E m Energy consumption EC for real-time interactive data n,m (t):

[0242]

[0243] Then the overall interaction energy consumption EC is calculated using the following formula com (t):

[0244]

[0245] Overall service time L Dos The calculation method of (t) is as follows:

[0246] Calculate user U n Through the edge server E m Path length J within the service area n,m (t):

[0247]

[0248] Represents user U n The position coordinates at time t, Represents the edge server E m The position coordinates at time t, R n,m Indicates that the edge server E m is the center of the circle and the radius is circle.

[0249] The edge server E is calculated using the following formula m For user U n Length of service provided

[0250]

[0251] in, Represents user U n The moving average speed of .

[0252] Then use the following formula to calculate the total service time L Dos (t):

[0253]

[0254] represents the penalty term, which is calculated as follows:

[0255]

[0256] Among them, μ represents the penalty coefficient, KS[g] represents the aggregation term of the constraint term, and its expression is as follows:

[0257]

[0258] ρ is a pre-set positive constant, g i represents the i-th constraint, i = 1, 2, ..., 6, g max Represents the maximum value of all constraints. Each constraint expression is as follows:

[0259] g1:-p n,m (t)≤0

[0260] g2:p n.m (t)-p max ≤0

[0261] g3:

[0262] g4:f n,m (t)-F m ≤0

[0263] g5:

[0264] g6:

[0265] Among them, p max Indicates the maximum wireless transmission power, Indicates the maximum delay, F m Represents the edge server E m The total amount of resources, κ is a constant greater than 0 and less than 1, a n,k is a Boolean variable, a n,k =1 indicates user U n Smart Reflective Surface I k Assist in information interaction with edge servers, otherwise a n,k =0.

[0266] S303: Training a multi-objective deep reinforcement learning model:

[0267] Construct a multi-objective deep reinforcement learning model, including a policy network θ and an evaluation network Q. The evaluation network Q is implemented based on a deep neural network (DNN). The state and preference vectors are input into the deep neural network, and the output size is 3N. A The Q value vector, N A Indicates the size of the action space. Figure 4 It is a training flow chart of the multi-objective deep reinforcement learning model in the present invention. Figure 4 As shown, the specific steps of training the multi-objective deep reinforcement learning model in the present invention include:

[0268] S401: Initialize model parameters:

[0269] Randomly initialize the parameters of the policy network θ and the evaluation network Q, and copy them as the target policy network θ′ and the target evaluation network Q′.

[0270] S402: Initialize playback buffer:

[0271] Create a playback buffer D and initialize it.

[0272] S403: Dividing preference space:

[0273] The multi-objective deep reinforcement learning model of the present invention adopts a single policy network to solve the multi-objective optimization problem. The present invention expands the single-objective Q function space in the traditional Q learning framework into a multi-objective Q function space, where the multi-objective Q function space contains all the The bounded Q function Q(s,a,W) can describe the expected total reward under different preferences. By fitting this expanded complete Q function space, which contains all the possibilities of multi-objective Pareto fronts, the single policy network can approximate the global Pareto front in the multi-objective optimization problem. Therefore, the present invention first divides the value space Ω of the preference vector W = (w1,w2,w3) into C preference subspaces c=1,2,…,C,w j Represents the preference weight of the jth optimization objective.

[0274] S404: Set training round z=1.

[0275] S405: Initialize the training parameters for this round:

[0276] Initialization time t=0, obtain the initial state s(0).

[0277] S406: Generate experience samples:

[0278] For each preference subspace Start a subprocess to generate experience samples. The specific method of generating experience samples is:

[0279] From the current subspace A preference vector W = (w1, w2, w3) is sampled from the state s(t); the policy network θ selects the action a(t) according to the state s(t). n (t) obtain the state s(t+1) and reward R(t) of the next moment; store the state transition data (s(t), a(t), R(t), W, s(t+1)) as experience samples in the replay buffer D; if the number of experience samples in the experience replay buffer pool D exceeds its predetermined capacity, delete the excess experience samples according to the established rules.

[0280] Then randomly sample N from all experience samples in the current experience replay cache pool D W Preference vector W v ,v=1,2,…,N W ; Then N W state transition data (s(t), a(t), R(t), W v ,s(t+1)) is stored in the playback buffer D as an experience sample.

[0281] In the prior art, the policy network θ usually adopts the ε-greedy strategy to select actions. The ε-greedy strategy is a common random strategy method. Its core idea is to randomly select actions with probability ε in each state and select the current optimal action with probability 1-ε. Among them, ε is a positive number less than 1, usually with a smaller value. The advantage of this is that it balances the relationship between exploration and utilization to a certain extent, which can not only ensure a certain degree of randomness in the exploration process, but also select more optimal actions to obtain rewards.

[0282] S407: Sampling transfer status data:

[0283] Sample B transition state data from the experience replay buffer pool D.

[0284] S408: Preference vector interpolation:

[0285] An interpolator is used to interpolate each preference vector in the B conversion state data, and the dimension of the preference vector is aligned with the dimension of the Q value vector to obtain the interpolated preference vector.

[0286] S409: Evaluation network parameter update:

[0287] For the B transition state data after preference vector interpolation, the target evaluation network Q′ is used to estimate the action and obtain the value y of the action. b :

[0288]

[0289] Where γ represents the discount factor, R b represents the reward in the bth transition state data, represents the interpolated preference vector in the b-th transition state data, a b ′、s b ′ respectively represent the action and state of the b-th transition state data at the next moment, Q′() represents the Q value vector obtained by the target evaluation network Q′, S c () indicates similarity, and the superscript T indicates transposition. The loss function L(θ) is calculated and the parameters of the policy network are updated. In this embodiment, the loss function L(θ) uses mean square error, and the specific calculation formula is as follows:

[0290]

[0291] S410: Determine whether t<T, where T represents the preset maximum time step. If so, proceed to step S411, otherwise proceed to step S412.

[0292] S411: Let t=t+1, and return to step S406.

[0293] S412: Update evaluation network parameters:

[0294] Calculate the gradient of the evaluation network Q and update the parameters of the evaluation network.

[0295] S413: Determine whether z<Z, where Z represents the preset maximum number of training rounds. If yes, proceed to step S414, otherwise the training ends.

[0296] S414: Determine whether z%η=0, where η represents the target network update period. If so, proceed to step S415, otherwise proceed to step S416.

[0297] S415: Update target network parameters:

[0298] Update the target evaluation network and target strategy network parameters:

[0299] Q′←τQ+(1-τ)Q′

[0300] θ′←τθ+(1-τ)θ′

[0301] Among them, τ represents the preset weight.

[0302] S416: Let z=z+1, and return to step S405.

[0303] The penalty coefficient μ is required in the calculation of the penalty term in the present invention. If the penalty coefficient is set too small or too large, the agent will fall into a local minimum or converge slowly. In order to solve this problem, the present invention introduces a dynamic penalty algorithm with an adaptive penalty coefficient to dynamically adjust the value of the penalty coefficient μ during the training process. Figure 5 is a flowchart of the dynamic adjustment of the penalty coefficient in this embodiment. Figure 5 As shown, the specific steps of dynamically adjusting the penalty coefficient in this embodiment include:

[0304] S501: Initialization parameters:

[0305] Set the minimum value of the penalty coefficient μ min and the maximum value μ max , penalty coefficient multiplication parameter φ, where φ>1, statistical threshold λ.

[0306] S502: Initialize penalty coefficient μ=μ min .

[0307] S503: Initialize counter u=0.

[0308] S504: Monitor the loss function value:

[0309] During the training process of the multi-objective deep reinforcement learning model, monitor and record its loss value, and determine whether the current loss value is less than (100-χ)% of the maximum loss value after the parameter update, where χ represents a preset constant. If so, go to step S505, otherwise go directly to step S506.

[0310] S505: Let μ = φ × μ min , u=u+1.

[0311] S506: Determine whether the current training round is finished. If not, return to step S504; otherwise, go to step S507.

[0312] S507: Calculate the ratio value:

[0313] Calculate the ratio β1 = u / u max , where u max Indicates the maximum number of iterations for each training round.

[0314] S508: Determine whether the ratio value β1≥λ. If so, take the current training round as a qualified training round. Otherwise, proceed to step S510.

[0315] S509: Calculate the statistical value of qualified training rounds:

[0316] Preset the training round sliding window size Δz, and count the statistical values ​​of qualified training rounds in the last Δz training rounds Calculate the proportion of statistical values

[0317] S510: Determine whether β2≥1-ε or μ≥μ max , ε represents the preset exploration rate. If it is satisfied, the penalty coefficient adjustment is completed, otherwise it returns to step S503.

[0318] S304: Obtain digital twin deployment strategy:

[0319] The status information of each user, each intelligent reflective surface and each edge server in the current digital twin edge network is collected to construct the state s, and then the policy network trained in step S303 is used to determine the optimal action a, thereby determining the edge server serial number where each user's digital twin is deployed, and obtaining the digital twin deployment strategy.

[0320] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A digital twin deployment method based on multi-objective deep reinforcement learning, characterized in that: The following steps are involved: S1: Get the user set in the digital twin edge network as U = {U1, U2, ..., U N },U n represents the nth user, n=1,2,…,N, N represents the number of users, and the set of intelligent reflective surfaces is I={I1,I2,…,I K }, I k represents the kth smart reflective surface, k = 1, 2, ..., K, K represents the number of smart reflective surfaces, and the edge server set is E = {E1, E2, ..., E M }, E m represents the mth edge server, m = 1, 2, ..., M, where M represents the number of edge servers; S2: For each user U n Set up a multi-objective deep reinforcement learning model with the following parameters: State: State s(t) = {su1(t),…,su N (t),si1(t),…,si K (t),se1(t),…,se M (t)}, where su n (t) represents user U n The state information at time t includes position, velocity, beamforming vector, si k (t) represents the smart reflective surface I k Status information, including position, reflection parameters, se m (t) represents the edge server E m Status information, including location and total resource quantity; Action: Set the action where x n,m =1 indicates user U n The digital twin is deployed to the edge server E m , otherwise x n,m =0; Reward function: The calculation formula of the reward function R is as follows: Where I represents a three-dimensional unit vector, f = [T syn (t),EC com (t),-L Dos (t)] represents the objective function vector, T syn (t) represents the overall synchronization delay, which is calculated as follows: Computational intelligent reflective surface assisted user U n and edge server E m The signal transmission rate between n,m (t): Among them, p n,m (t) represents the user U under path loss consideration n For edge server E m The transmission power is calculated as follows: γ κ Noise ratio threshold, d0 is the reference distance, α is the path loss exponent, d n,m (t) represents the user U at time t n and edge server E m The distance between them; N0 represents the noise power spectrum density, B represents the channel bandwidth, ν n (t) is the time when user U n The beamforming vector of Indicates that user U is assisted by the intelligent reflective surface n and edge server E m The channel vector of: Among them, h n,m (t) represents user U n and edge server E m Channel gain between: λ α represents the path fading factor, ε κ Denotes the distortion factor, d n,m (t) represents the user U at time t n and edge server E m The distance between is user U at time t n With smart reflective surface I k The channel gain between n,k (t) represents the user U at time t n To Smart Reflective Surface I k The distance Represents intelligent reflective surface I k and edge server E m The channel gain between k,m Represents intelligent reflective surface I k To the edge server E m The distance between k Represents intelligent reflective surface I k Reflection parameters of Calculate the user U at time t n Transmit real-time operation data n To the edge server E m Transmission delay Among them, |D n |Represents real-time running data D n size; Calculate the user U at time t n Transmit real-time operation data n To the edge server E m Processing time on f n,m (t) represents the user U at time t n Consumption edge server E m computing resources; The following formula is used to calculate the user U per unit time at time t n Synchronization delay of digital twins for: ΔD n Represents user U n The size of data transmitted per unit time; The overall interaction delay T is calculated using the following formula: syn (t): EC com (t) represents the total interaction energy consumption, which is calculated as follows: The following formula is used to calculate user U n For edge server E m Energy consumption EC for real-time interactive data n,m (t): Then the overall interaction energy consumption EC is calculated using the following formula com (t): L Dos (t) represents the total service time, which is calculated as follows: Calculate user U n Through the edge server E m Path length J within the service area n,m (t): Represents user U n The position coordinates at time t, Represents the edge server E m The position coordinates at time t, R n,m Indicates that the edge server E m is the center of the circle and the radius is circle; The edge server E is calculated using the following formula m For user U n Length of service provided in, Represents user U n The moving average speed of Then use the following formula to calculate the total service time L Dos (t): represents the penalty term, which is calculated as follows: Among them, μ represents the penalty coefficient, KS[g] represents the aggregation term of the constraint term, and its expression is as follows: ρ is a pre-set positive constant, g i represents the i-th constraint, i = 1, 2, ..., 6, g max Represents the maximum value of all constraints. Each constraint expression is as follows: Among them, p max Indicates the maximum wireless transmission power, Indicates the maximum delay, F m Represents the edge server E m The total amount of resources, κ is a constant greater than 0 and less than 1, a n,k is a Boolean variable, a n,k =1 indicates user U n Smart Reflective Surface I k Assist in information interaction with edge servers, otherwise a n,k =0; S3: Construct a multi-objective deep reinforcement learning model, including a policy network θ and an evaluation network Q. The evaluation network Q is implemented based on a deep neural network. The state and preference vectors are input into the deep neural network, and the output size is 3N. A The Q value vector, N A Indicates the size of the action space, and then trains the multi-objective deep reinforcement learning model. The specific method is: S3.1: Randomly initialize the parameters of the policy network θ and the evaluation network Q, and copy them as the target policy network θ′ and the target evaluation network Q′; S3.2: Establish a playback buffer D and initialize it; S3.3: Divide the value space Ω of the preference vector W = (w1, w2, w3) into C preference subspaces c=1,2,…,C,w j represents the weight of the jth optimization objective; S3.4: Let training round z = 1; S3.5: Initialization time t=0, obtain initial state s(0); S3.6: For each preference subspace Start a subprocess to generate experience samples. The specific method is: From the current subspace A preference vector W = (w1, w2, w3) is sampled from the state s(t); the policy network θ selects the action a(t) according to the state s(t). n (t) obtain the state s(t+1) and reward R(t) of the next moment; store the state transition data (s(t), a(t), R(t), W, s(t+1)) as experience samples in the replay buffer D; if the number of experience samples in the experience replay buffer pool D exceeds its predetermined capacity, delete the excess experience samples according to the established rules; Then randomly sample N from all experience samples in the current experience replay cache pool D W Preference vector W v ,v=1,2,…,N W ; Then N W state transition data (s(t), a(t), R(t), W v ,s(t+1)) is stored as an experience sample in the playback buffer D; S3.7: Sample B transition state data from the experience replay buffer pool D; S3.8: using an interpolator to interpolate each preference vector in the B conversion state data, aligning the dimension of the preference vector with the dimension of the Q value vector, and obtaining an interpolated preference vector; S3.9: For the B transition state data after preference vector interpolation, use the target evaluation network Q′ to estimate the action and obtain the value y of the action b : Where γ represents the discount factor, R b represents the reward in the bth transition state data, represents the interpolated preference vector in the b-th transition state data, a b ′、s b ′ respectively represent the action and state of the b-th transition state data at the next moment, Q′() represents the Q value vector obtained by the target evaluation network Q′, S c () indicates similarity, and the superscript T indicates transposition; Calculate the loss function L(θ) and update the parameters of the policy network; S3.10: Determine whether t<T, where T represents the preset maximum time step. If so, proceed to step S3.11; otherwise, proceed to step S3.12; S3.11: Let t=t+1, and return to step S3.6; S3.12: Calculate the gradient of the evaluation network Q and update the parameters of the evaluation network; S3.13: Determine whether z < Z, where Z represents the preset maximum number of training rounds. If yes, proceed to step S3.14, otherwise the training ends. S3.14: Determine whether z%η=0, where η represents the target network update period. If yes, proceed to step S3.15, otherwise proceed to step S3.16; S3.15: Update target network parameters: Update the target evaluation network and target strategy network parameters: Q′←τQ+(1-τ)Q′ θ′←τθ+(1-τ)θ′ Among them, τ represents the preset weight. S3.16: Let z=z+1, and return to step S3.5; S4: Collect the status information of each user, each intelligent reflective surface and each edge server in the current digital twin edge network to construct the state s, and then use the policy network trained in step S3.3 to determine the optimal action a, so as to determine the edge server serial number where each user's digital twin is deployed, and obtain the digital twin deployment strategy.

2. The digital twin deployment method according to claim 1, characterized in that: The penalty coefficient in step S2 is dynamically adjusted during the training process, and the specific method is as follows: 1) Set the minimum value of the penalty coefficient μ min and the maximum value μ max , penalty coefficient multiplication parameter φ, where φ>1, statistical threshold λ; 2) Initialize the penalty coefficient μ = μ min ; 3) Initialize counter u=0; 4) During the training process of the multi-objective deep reinforcement learning model, monitor and record its loss value, and determine whether the current loss value is less than (100-χ)% of the maximum loss value after the parameter update, where χ represents a preset constant. If so, proceed to step 5), otherwise proceed directly to step 6); 5) Orderμ=φ×μ min , u = u + 1; 6) Determine whether the current training round has ended. If not, return to step 4), otherwise go to step 7); 7) Calculate the ratio β1 = u / u max , where u max Indicates the maximum number of iterations for each training round; 8) Determine whether the ratio value β1≥λ, if yes, take this training round as a qualified training round, otherwise go to step 10); 9) Preset the training round sliding window size Δz, and count the statistical values ​​of qualified training rounds in the last Δz training rounds Calculate the proportion of statistical values 10) Determine whether β2 ≥ 1-ε or μ ≥ μ max , ε represents the preset exploration rate. If it is satisfied, the penalty coefficient adjustment ends, otherwise returns to step 3).

Images