Edge digital twin information synchronization optimization method in vehicle-to-everything scenario

Abstract

The application relates to an edge digital twin information synchronization optimization method in a vehicle networking scene, and belongs to the field of mobile communication, and comprises the following steps: S1, establishing a digital twin assisted vehicle networking architecture, and realizing real-time monitoring and service decision of a vehicle user and an edge server by using a digital twin technology; S2, determining a digital twin body association strategy of the user by the edge server; S3, directly obtaining a direct satisfaction degree of the user to the digital twin body construction according to direct interaction experience between the user and the edge server; S4, considering the influence of the digital twin body association of the user on other service quality, that is, an indirect satisfaction degree; S5, evaluating a satisfaction utility of the user to the digital twin body, and optimizing the digital twin body association relationship of the user by taking the maximum average satisfaction utility of the digital twin body of all vehicle users in the network as a target; and S6, adopting a branch and bound Q network algorithm to solve an optimal user digital twin body and server association scheme.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of mobile communication, and relates to an edge digital twin information synchronization optimization method in a vehicle networking scene. BACKGROUND

[0002] Vehicle networking is gradually becoming a key application scenario of 5G and super fifth-generation mobile communication. As a key link of intelligent transportation, vehicle networking technology is an important engine for promoting the construction of smart cities. This technology takes vehicle nodes as the main body of mobile sensing, and realizes seamless connection in the vehicle networking communication network with the help of advanced wireless communication technology, so as to realize the interconnection and sharing of multi-level and large-scale data. The rapid development of automatic driving and connected vehicles also brings a large number of vehicle applications, including services related to safety services and entertainment, which leads to heterogeneous QoS requirements in vehicle networking.

[0003] The digital twin model is not only a simple digital representation of a physical object, but also a multi-level, multi-scale and multi-field comprehensive model throughout the life cycle through intelligent processing based on networked perception and full-factor information acquisition. Digital twin technology creates a virtual twin physical network through modeling, computing, communication, data processing and other technologies, and realizes the co-evolution of virtual space and physical space. With these advanced functions, digital twin is considered one of the most important enabling technologies for 6G. In the vehicle networking scene, reasonable application of digital twin technology can accurately simulate traffic flow and effectively manage network resources, thereby supporting the stable operation of traffic-related applications and improving the safety and intelligence of road traffic.

[0004] In actual vehicle networking scenarios, the synchronization process of the digital twin may be affected by server resource limitations and communication bottlenecks. At the same time, when synchronizing user information of the digital twin, physical resources are consumed, and the system utility and other services are affected, so it is necessary to quantify the process of associating the user digital twin with the server from different dimensions. SUMMARY

[0005] Therefore, the purpose of the present application is to provide an edge digital twin information synchronization optimization method in a vehicle networking scene to ensure the timeliness and reliability of user information transmission and to realize the accuracy of strategy simulation. The digital twin association relationship of vehicle users is optimized to maximize the system utility of all vehicle users in the network.

[0006] To achieve the above purpose, the present application provides the following technical scheme:

[0007] An edge digital twin information synchronization optimization method in a vehicle networking scene, comprising the following steps:

[0008] S1: Establish a digital twin-assisted vehicle networking architecture, including vehicle users, base stations, and edge servers; use digital twin technology to realize real-time monitoring and business decision-making between vehicle users and edge servers;

[0009] S2: The edge server determines the digital twin association strategy of the vehicle user in the network;

[0010] S3: According to the direct interaction experience between the vehicle user and the edge server, the direct satisfaction of the vehicle user to the construction of its digital twin is obtained;

[0011] S4: Consider the impact of the digital twin association of the vehicle user on the quality of other business services, i.e. indirect satisfaction;

[0012] S5: Evaluate the satisfaction utility of the vehicle user to its digital twin, and optimize the digital twin association relationship of the vehicle user with the goal of maximizing the average satisfaction utility of all vehicle users in the network;

[0013] S6: Use the branch and bound Q-network algorithm to solve the optimal vehicle user digital twin and server association scheme.

[0014] Further, the vehicle user will collect real-time state data through the base station into the digital twin deployed in the edge server to update the user's latest state in real time; the vehicle user set is The edge server is connected to the base station one by one, and the sequence set is Therefore, the edge server set is The base station set is When the user's twin is deployed in the edge server ES, real-time information is transmitted to the edge server ES through the base station BS for twin update; the user set whose twin is deployed in the same edge server ES is U m .

[0015] Further, in step S2, the vehicle user establishes a twin association relationship in the edge server according to the construction requirements and network state, including:

[0016] (1) The vehicle user u transmits state information data to the edge server ES m through the base station BS m , i.e. is associated with the edge server ES m ; The association between the twin of the vehicle user and the edge server at time slot t is represented by a binary variable matrix Φ(t), where Φ(t) represents that the twin of the vehicle user u is deployed in the edge server ES m to synchronize real-time data, otherwise ​

[0017] (2) When the dynamic change of edge server resources and the movement of vehicle users cause the distance to increase, migrate the twin to other edge servers, and use binary variable δ m (t) to represent the twin migration associated with edge server ES m , that is:

[0018]

[0019] wherein represents the deployment of the twin of vehicle user u at time slot t-1 to edge server ES m synchronizes real-time data;

[0020] (3) The maximum number of maintainable vehicle users of the edge server is:

[0021]

[0022] wherein M m represents the total storage resources of edge server ES m , t d represents a fixed duration, represents the storage resource size required by the edge server to establish 1-bit data storage and management, D H represents the historical data of the vehicle user stored by the edge server;

[0023] (4) The twin of each user can only be associated with one edge server, that is:

[0024]

[0025] Further, the step S3 specifically comprises the following steps:

[0026] S31: Use the demand supply model to quantify the congestion degree of the server:

[0027]

[0028] wherein π0 and π1 are the coefficients of performance degradation, |U m | is the number of users deploying the twin in ES m ;

[0029] S32: Calculate the digital twin mapping granularity, which is jointly determined by the edge server congestion degree and the user distance, and is represented as:

[0030]

[0031] wherein λ and ο are constants, ζ m (t) is the server ES​m the congestion degree, η is the packet loss rate per unit congestion degree, d(m, u) is the distance between the edge server ES m and the user u, and a is the distance scaling factor.

[0032] S33: Calculate the information synchronization delay of the twin in updating the state of the user u, including the transmission delay and the calculation delay; then the synchronization delay is recorded as:

[0033]

[0034] where D u is the size of the user transmission state update data, τ is the calculation size required per unit data size, and are the computing resources and bandwidth resources provided by the edge server and its wired connected base station respectively; SNR m,u (t) represents the achievable signal-to-noise ratio at time t.

[0035] The timeliness of the digital twin of the user u associated with the edge server ES m is represented as:

[0036]

[0037] where T D represents the synchronization delay threshold of the vehicle user;

[0038] S34: Adopt a sliding window mechanism, i.e. a decay function represents the decay degree of the satisfaction degree obtained in the kth interaction compared to the current interaction time slot satisfaction degree, i.e.:

[0039]

[0040] wherein is a parameter, t k represents the end time of the kth interaction time slot.

[0041] When associated with the edge server ES m , the direct satisfaction degree of the digital twin of the user u is represented as:

[0042]

[0043] where P represents the number of valid interactions within the sliding window, k represents the number of interactions, μ1 and μ2 represent the weights of the digital twin timeliness and the mapping granularity respectively, Sy m,u (t k ) represents the mapping granularity of the digital twin in the kth interaction.

[0044] Further, in the step S4, a resource residual rate is used to quantify the relationship between the resources required for twin maintenance and the resources required for other services of the user, wherein the resource residual rate is determined by the adaptive allocation of computing resources of the edge server and the total computing resources, the adaptive allocation of bandwidth of the base station and the total bandwidth, and is expressed as:

[0045]

[0046] wherein κ1 and κ2 are weights, B m represents the total bandwidth of the BS m , C m represents the total computing resources of the ES m .

[0047] Further, the step S5 specifically comprises the following steps:

[0048] S51: Weighting the direct satisfaction and the indirect satisfaction to obtain the global satisfaction of the user u to the digital twin after the current interaction time slot t ends:

[0049] Gs m,u (t) = ωDR m,u (t) + (1-ω)IR m,u (t)

[0050] wherein ω is a weight coefficient;

[0051] The global satisfaction is normalized as:

[0052]

[0053] S52: The revenue obtained by the edge synchronization of the digital twin is positively correlated with the satisfaction of the user to the digital twin, and is expressed as:

[0054]

[0055] wherein υ in is the unit income of information synchronization;

[0056] S53: The instantiation cost CO ins (t) is the instantiation cost of each server downloading the corresponding software module from the cloud server to support the instantiation of a new digital twin, and is expressed as:

[0057]

[0058] wherein l ins represents the unit instantiation cost, D soft represents the size of the software module required for instantiation, and d(m, G) represents the distance between the ES m and the cloud server G;

[0059] The synchronization cost is the cost of real-time data transmission of the user twin on the appropriate server, denoted as:

[0060]

[0061] where p is the cost of transmitting data per unit distance; d(m, u) represents the distance between user u and ES m .

[0062] Therefore, when migration occurs, the migration cost is:

[0063]

[0064] δ m (t) represents the migration status of the twin associated with the edge server ES m ; d(m, m') is the distance between edge servers ES m and ES m' ; c M is the unit migration cost.

[0065] To maximize the long-term utility of the system, the problem of optimizing the association of the digital twin of the vehicle user is solved, that is:

[0066]

[0067] In the formula, Φ(t) is the association of the twin of the vehicle user with the server, where represents the deployment of the twin of the vehicle user to the edge server for synchronization of real-time data, that is, the establishment of the association between the twin of the user and the server. and T D are the synchronization delay and the synchronization delay threshold of the vehicle user, respectively, is the maximum number of twins that can be maintained by the server; U D (t) represents the information synchronization utility, ω1 represents the revenue weight, ω2 represents the total cost weight, π1, π2 and π3 represent the weights of the instantiation cost, the synchronization cost and the migration cost, respectively.

[0068] C1 ensures that each user can only deploy a twin on one server; C2 represents the number constraint of the association between each server and the twin; C3 represents that the DT synchronization delay cannot exceed the maximum synchronization delay threshold.

[0069] Furthermore, in step S6, the optimization model is first converted into a Markov decision process, and then the branch duel Q network algorithm is used to solve the optimal vehicle user digital twin and server association scheme; the Markov decision process is as follows: (1) State space: The system state is composed of the user's association state with each edge server at the previous moment, the user's location and the remaining status of each resource; (2) Action space: The action space is the association action with the user twin; (3) Reward function: The reward function is the difference between the benefit and cost in the current network, that is, the system utility.

[0070] Furthermore, the branched dueling Q-network adopts a structure based on the dueling dual-depth Q-network. The dueling network separates the original dual-depth Q-network structure into a value branch and an advantage branch, and simultaneously trains these two branches through experience replay. The branched dueling Q-network divides multi-dimensional actions into multiple sub-actions for separate processing while maintaining the sharing of input state, providing a certain degree of autonomy for each sub-action. The network is trained and updated through the interaction between the intelligent agent and the environment, ultimately obtaining a converged optimal association strategy between the connected vehicle user twin and the server.

[0071] The Q value Q of each sub-action of the branch duel Q network d (s,a d ) is composed of the value branch and the corresponding advantage branch, namely:

[0072]

[0073] Where V(s) represents the value branch under state s, Represents a collection of sub-actions. d (s,a d ) indicates a sub-action Advantage function, M represents the number of decomposable sub-actions, A d (s,a' d ) indicates a sub-action Advantage function of

[0074] The ε-greedy strategy is used to select actions, namely:

[0075]

[0076] in Represents the first decomposed sub-action, Indicates sub-action The Q-value function, represents the Mth sub-action of the decomposition, Indicates sub-action Q-value function of

[0077] The loss function is the expected value of the mean square error between each branch, that is:

[0078]

[0079] Ε (s,a,r,s') denotes the mean of the following formula, M denotes the number of sub-actions, d denotes the sub-action sequence value, Q d (s,a d ,ω) denotes the Q value function of sub-action a d , r denotes the reward value, λ D denotes the learning rate, Q d (s',a' d ,ω) denotes the Q value function of sub-action a' d . denotes the target network parameter;

[0080] The TD-error formula is:

[0081]

[0082] where Q d (s,a d ,ω) denotes the Q value function of sub-action a d , and a' d denotes the action that maximizes the Q value function.

[0083] Further, the branch duel Q network algorithm is used to solve the optimal vehicle user digital twin and server association scheme, which specifically includes the following steps:

[0084] S61: initialize the learning rate, discount factor, soft update parameter, experience pool, neural network parameter, exploration probability, and global satisfaction of users to edge servers, and randomly assign the edge server index with unstable satisfaction;

[0085] S62: the agent observes the environment to obtain the initial state;

[0086] S63: after the agent selects an action using the ε-greedy strategy at each time slot, the agent obtains a reward and enters the next state;

[0087] S64: store the experience in the experience replay pool;

[0088] S65: extract samples from the experience replay pool, calculate the TD-error, and update the neural network parameter;

[0089] S66: update the target network parameter according to the soft update method;

[0090] S67: stop when the number of iterations reaches the maximum, otherwise return to step S66.

[0091] ​The application has the beneficial effect that the mapping granularity and timeliness of the digital twin are considered, and the influence on other services is also considered, thereby improving the satisfaction of the digital twin deployment in the vehicle networking scene.

[0092] Other advantages, objects, and features of the application will be set forth in part in the following specification taken in conjunction with the accompanying drawings, and in part will become apparent to those skilled in the art from a consideration of the following specification and drawings. The objects and other advantages of the application will be realized and attained by means of the instrumentalities and combinations pointed out in the following specification. BRIEF DESCRIPTION OF DRAWINGS

[0093] In order to make the objects, technical solutions and advantages of the application clearer, the preferred detailed description of the application will be made below in conjunction with the drawings, in which:

[0094] Figure 1 A digital twin assisted vehicle networking architecture diagram according to the application;

[0095] Figure 2 A BDQ algorithm network architecture diagram according to the application;

[0096] Figure 3 A BDQ algorithm training flowchart according to the application. DETAILED DESCRIPTION

[0097] The embodiments of the application are described below through specific concrete examples, and those skilled in the art can easily understand other advantages and effects of the application from the disclosure of the specification. The application can also be implemented or applied through other different specific embodiments, and each detail in the specification can be modified or changed based on different views and applications without departing from the spirit of the application. It should be noted that the diagrams provided in the following examples only illustrate the basic concept of the application in a schematic manner, and the following examples and features in the examples can be combined with each other without conflict.

[0098] The drawings are only used for exemplary illustration, and the representation is only a schematic diagram, not a physical diagram, and cannot be understood as a limitation on the application; in order to better illustrate the embodiments of the application, some components in the drawings may be omitted, enlarged or reduced, and do not represent the size of the actual product; it can be understood by those skilled in the art that some known structures and their descriptions in the drawings may be omitted.

[0099] The same or similar reference numerals in the drawings of the embodiments of the present application correspond to the same or similar components; in the description of the present application, it is understood that if the orientations or positional relationships indicated by the terms "upper", "lower", "left", "right", "front", "back" and the like are based on the orientations or positional relationships shown in the drawings, they are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, therefore the terms describing the positional relationship in the drawings are only used for exemplary illustration, and cannot be understood as a limitation on the present application, for those skilled in the art, the specific meanings of the above terms can be understood according to the specific circumstances.

[0100] As shown in Figures 1-3 , the present application proposes an edge digital twin information synchronization optimization method in the Internet of Vehicles scene to guarantee the timeliness and reliability of user information transmission to realize the accuracy of strategy simulation. The digital twin body association relationship problem of vehicle users is optimized to maximize the system utility of all vehicle users in the network;

[0101] S1: Establish a digital twin assisted Internet of Vehicles architecture, realize real-time monitoring and business decision of vehicle users and servers by using digital twin technology;

[0102] As shown in Figure 1 , it is a digital twin assisted Internet of Vehicles architecture diagram; the digital twin assisted Internet of Vehicles architecture includes vehicle users, base stations and edge servers; the information (position, speed, service quality requirement, etc.) of the vehicle users is transmitted to the edge server through the base station, wherein the vehicle users provide data support for the association of the digital twin body and the server, the base station provides wireless access service, and the edge server provides computing and storage service.

[0103] The vehicle users will transmit the collected real-time state data to the digital twin body deployed in the edge server through the base station to update the latest state of the users in real time. The set of vehicle users is

[0104] Suppose there are M edge servers, and the sequence set is The edge servers correspond to the base stations one by one and are connected through optical fibers, so the set of edge servers is The set of base stations is When the twin body of the user is deployed in the edge server ES, the real-time information is transmitted to the edge server ES through the base station BS for twin body update; the set of users whose twin bodies are deployed in the same edge server is U m ;

[0105] S2: The edge server determines the digital twin association strategy of the vehicle user in the network; in order to capture the dynamic changes of the service state of the vehicle user in time, the user needs to establish a twin association relationship in the edge server according to the construction needs and the network state, while considering the completeness, timeliness and influence on other services in the information synchronization process; wherein the network state includes vehicle position, speed, server remaining resources, wireless channel state, etc.

[0106] The vehicle user u passes through the base station BS m The state information data is transmitted into the edge server ES m , that is, associated with the edge server ES m ;

[0107] The association of the twin of the vehicle user u at time slot t with the edge server is represented by a binary variable matrix Φ(t), wherein Φ(t) represents that the twin of user u is deployed to the edge server ES m Synchronization real-time data, otherwise

[0108] Due to the dynamic changes of server resources and the increase of communication distance caused by user movement, the user's synchronization information may face a high packet loss rate or a large synchronization delay, thereby reducing the timeliness and accuracy of the digital twin synchronization process. In order to ensure the reliability of the user synchronization information, thereby ensuring the accuracy of the subsequent digital twin network distribution strategy, the system migrates the twin to other edge servers. A binary variable δ m (t) is used to represent the migration of the twin associated with the edge server ES m , that is:

[0109]

[0110] Since the server computing capability is very strong, the main influencing factor of the maximum number of maintainable users is the storage resources consumed by historical data and future data, that is:

[0111]

[0112] Wherein, M m represents the total storage resources of the edge server ES m , t d represents a fixed duration, represents the storage resource size required for the server to establish 1-bit data storage and management, D H represents the historical data of the edge server storing the user, for convenience, it is assumed that D H is a constant value;

[0113] At the same time, it is stipulated that the twin of each user can only be associated with one edge server, that is:

[0114]

[0115] S3: Considering the direct satisfaction of the vehicle user to the construction of its digital twin, S3 is derived from the direct interaction experience between the vehicle user and the edge server;

[0116] The direct satisfaction is derived from the direct interaction experience between the vehicle user and the edge server, which is jointly determined by dynamic factors such as digital twin mapping granularity and digital twin timeliness;

[0117] Associating too many user twins on the same edge server can cause resource competition and server congestion, because these twins need to share the underlying infrastructure's resource storage history information, resulting in a decline in the server's storage performance. Therefore, in order to measure the completeness of the user's synchronization with its twin information, a demand-supply model is used to quantify the degree of server congestion, that is:

[0118]

[0119] Where π0 and π1 are the coefficients of performance degradation, |U m is the number of users whose twins are deployed on ES m ;

[0120] Since the congestion level of the edge server can cause packet loss problems between the user and its twin, and the distance between the user and its twin can cause errors in the mapping process, when the user u associates its twin with the edge server ES m , the digital twin mapping granularity is jointly determined by the edge server congestion level and the user distance, which can be expressed as:

[0121] Sy m,u (t)=λ(1-ζ m (t)η) αd(m,u) -ο

[0122] Where λ and ο are constants, ζ m (t) is the congestion degree of the server ES m , η is the packet loss rate per unit congestion degree, d(m,u) is the distance between the edge server ES m and the user u, and α is the distance scaling factor

[0123] When the twin of the user u is associated with the edge server ES mIn association, in order to measure the timeliness of user twin body information synchronization, the time delay of twin body updating user u state is composed of two parts, which are data transmission time delay and edge server computing time delay, wherein the transmission time delay is determined by vehicle update task size, vehicle transmission power, signal to noise ratio and bandwidth adaptively allocated by base station, and the computing time delay is determined by CPU cycle required by vehicle update task and computing resources adaptively allocated by edge server.

[0124] Then the synchronization time delay can be recorded as:

[0125]

[0126] Wherein, D u is the data size of user transmission state update, τ is the computing size required by unit data size, and are the computing resources and bandwidth resources adaptively provided by edge server and its wired connected base station respectively; SNR m,u (t) represents the signal to noise ratio that can be achieved at t time.

[0127] The timeliness of digital twin of user u associated with edge server ES m can be expressed as:

[0128]

[0129] Wherein T D represents the synchronization time delay threshold of vehicle user;

[0130] Considering that the user interacts frequently with the edge server where the DT is deployed, and the interaction experience that has been long apart lags behind the current satisfaction update, more attention should be paid to recent interaction behavior, therefore, the sliding window mechanism is adopted to update direct satisfaction. Considering the sliding window mechanism, i.e. the decay function is used to represent the decay degree of the satisfaction obtained by the kth interaction compared with the current interaction time slot satisfaction, i.e.: Wherein, is a parameter, t k represents the end time of the kth interaction time slot;

[0131] P represents the number of effective interactions in the sliding window. Therefore, when associated with edge server ES m , the direct satisfaction of digital twin of user u can be expressed as:

[0132]

[0133] k represents the number of interactions, μ1 and μ2 represent the weights of digital twin timeliness and mapping granularity respectively, Sy m,u (t krepresents the mapping granularity of the kth interaction digital twin.

[0134] S4: Consider the influence of the digital twin association of the vehicle user on the quality of other service, i.e. indirect satisfaction;

[0135] The resource surplus rate is used to quantify the relationship between the resources required for twin maintenance and the resources required for user services, so as to reduce network load while dealing with sudden conditions such as large flow services of users. The resource surplus rate is determined by the adaptive allocation of computing resources of the edge server and the total computing resources, the adaptive allocation of bandwidth of the base station and the total bandwidth, i.e.:

[0136]

[0137] Where κ1 and κ2 are weights, B m represents the total bandwidth of the BS m , C m represents the total computing resources of the ES m .

[0138] S5: Evaluate the satisfaction utility of the vehicle user to its digital twin, and optimize the digital twin association relationship problem of the vehicle user with the goal of maximizing the system utility of all vehicle users in the network;

[0139] The direct satisfaction and indirect satisfaction obtained from the above are weighted to obtain the global satisfaction of the user u to the digital twin after the current interaction time slot t ends, and ω is the weight coefficient:

[0140] Gs m,u (t)=ωDR m,u (t)+(1-ω)IR m,u (t)

[0141] After normalization, we can get:

[0142] Assuming that the revenue obtained by the digital twin edge synchronization is positively correlated with the satisfaction of the user to the digital twin, it can be represented as:

[0143]

[0144] Where, υ in is the unit income of information synchronization;

[0145] The instantiation cost CO ins (t) is the instantiation cost of each server from the cloud server to download the corresponding software module to support the instantiation of a new digital twin, which can be represented as:

[0146]

[0147] Where l insdenotes the unit instantiation cost, D soft denotes the software module size needed for instantiation, d(m, G) denotes the ES m distance to the cloud server G;

[0148] The synchronization cost is the cost of real-time data transmission of the user twin on the appropriate server, which can be expressed as:

[0149]

[0150] where p is the cost of transmitting data per unit distance; d(m, u) denotes the distance between user u and ES m ;

[0151] Since the server stores a large amount of historical data and twin data, when migration occurs, the user's historical data also needs to be migrated, and the migration cost is as shown in the formula:

[0152]

[0153] δ m (t) represents the migration situation of the twin associated with the edge server ES m ; d(m, m') is the distance from the edge server ES m to ES m' ; c M is the unit migration cost;

[0154] To maximize the long-term utility of the system, the problem of optimizing the association relationship of the digital twin of the vehicle user is solved, that is:

[0155]

[0156] In the formula, Φ(t) is the association of the twin of the vehicle user with the server, where represents the synchronization of real-time data of the twin of the vehicle user deployed to the edge server, that is, the association between the twin of the user and the server, and T D are the synchronization delay and synchronization delay threshold of the vehicle user, respectively, is the maximum number of twins that the server can maintain; U D (t) represents the information synchronization utility, ω1 represents the revenue weight, ω2 represents the total cost weight, π1, π2 and π3 represent the weights of instantiation cost, synchronization cost and migration cost, respectively;

[0157] C1 ensures that each user can only deploy a twin on one server; C2 represents the number constraint of the association between each server and the twin; C3 represents that the DT synchronization delay cannot exceed the maximum synchronization delay threshold;

[0158] S6: adopt branch duel Q network algorithm to solve the optimal vehicle user digital twin and server association scheme;

[0159] First, the optimization model is converted into a Markov decision process, and the branch duel Q network algorithm is used to solve the optimal vehicle user digital twin and server association scheme. The Markov decision process is as follows:

[0160] (1) State space: the system state is composed of the association state of the user with each edge server at the last time Φ(t-1), the user location d, and the remaining resources of each resource, which can be represented as:

[0161] (2) Action space: the action space is the association action of the user twin

[0162] (3) Reward function: the reward function is the difference between the current network income and cost, i.e. the system utility

[0163]

[0164] The parameter penalty κ is a very large negative constant. The purpose of setting the reward is to promote the optimal configuration of resource management and DT association to maximize the reward. If the selected behavior violates the constraints, the reward will be penalized;

[0165] The branch duel Q network adopts the structure based on duel double deep Q network. The duel network separates the original double deep Q network structure into value branch and advantage branch, and trains the two branches simultaneously through experience replay. The branch duel Q network divides the multi-dimensional action into multiple sub-actions for processing, while maintaining the sharing of input states, providing a certain degree of autonomy for each sub-action. The network is trained and updated through the interaction between the agent and the environment;

[0166] The BDQ algorithm is similar to the Dueling DDQN, and the Q value of each sub-action Q d (s,a d ) is aggregated by the value branch and the corresponding advantage branch, i.e.

[0167]

[0168] where V(s) represents the value branch under state s, A d (s,a d ) represents the advantage function of sub-action , and M represents the number of decomposable sub-actions. A d (s,a' d ) represents the advantage function of sub-action ;

[0169] The action is selected by using an epsilon-greedy strategy, that is:

[0170]

[0171] wherein represents the first decomposed sub-action, represents the Q value function of the sub-action , and represents the Mth decomposed sub-action, represents the Q value function of the sub-action .

[0172] The loss function is the expected value of the mean square error between branches, that is:

[0173]

[0174] E (s,a,r,s') represents the mean value of the following formula, M represents the number of sub-actions, d represents the value of the sub-action sequence, Q d (s,a d , ω) represents the Q value function of the sub-action a d , r represents the reward value, λ D represents the learning rate, Q d (s',a' d , ω) represents the Q value function of the sub-action a' d , and represents the target network parameter;

[0175] The TD-error formula is:

[0176]

[0177] wherein Q d (s,a d , ω) represents the Q value function of the sub-action a d , and represents the action a' d that maximizes the Q value function.

[0178] Therefore, the BDQ algorithm comprises the following steps:

[0179] S61: initialize the learning rate, discount factor, soft update parameter, experience pool, neural network parameter, exploration probability, global satisfaction of users to the edge server, and randomly assign the index of the edge server with unstable satisfaction;

[0180] S62: the agent observes the environment to obtain an initial state;

[0181] S63: after the agent selects an action by using an epsilon-greedy strategy at each time slot, a reward is obtained, and the next state is entered;

[0182] S64: store the experience into an experience replay pool;

[0183] S65: draw a sample from the experience replay pool, calculate a TD-error to update the neural network parameters;

[0184] S66: update the target network parameters according to a soft update manner;

[0185] S67: stop when the number of iterations reaches a maximum, otherwise return to step S66.

[0186] A person of ordinary skill in the art can understand that all or part of the steps in the above-mentioned embodiment methods can be completed by programs instructing relevant hardware, and the programs can be stored in a computer readable storage medium. When the programs are executed, the steps of the method can be implemented. The storage medium is, for example, ROM / RAM, a magnetic disk, an optical disk, etc.

[0187] Finally, it should be pointed out that the above embodiments are only used to illustrate the technical solutions of the present application and not to limit it. Although the present application has been described in detail with reference to the preferred embodiments, a person of ordinary skill in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the purpose and scope of the technical solutions, which should be covered in the scope of the claims of the present application.

Claims

1. A method for synchronous optimization of edge digital twin information in a connected vehicle scenario, characterized by: The following steps are involved: S1: Establish a digital twin-assisted Internet of Vehicles architecture, including vehicle users, base stations, and edge servers; use digital twin technology to enable real-time monitoring and business decision-making of vehicle users and edge servers; S2: The edge server determines the digital twin association strategy of vehicle users in the network; S3: The direct satisfaction of vehicle users with their digital twin construction is obtained based on the direct interaction experience between vehicle users and edge servers; S4: Consider the impact of the digital twin association of vehicle users on the quality of other business services, that is, indirect satisfaction; S5: Evaluate the satisfaction utility of vehicle users with their digital twins, and optimize the digital twin association problem of vehicle users with the goal of maximizing the average satisfaction utility of the digital twins of all vehicle users in the network; S6: Using the branching dueling Q-network algorithm to solve the optimal vehicle-user digital twin and server association scheme; The vehicle user transmits the collected real-time status data to the digital twin deployed in the edge server through the base station to update the user's latest status in real time; the vehicle user set is The edge servers are connected to the base stations one by one, and the sequence set is Therefore, the set of edge servers is The base station set is When the user's twin is deployed on the edge server ES, the real-time information is transmitted to the edge server ES through the base station BS via the wireless link for twin update; the twin is deployed to the same edge server The user set is U m ; In step S2, the vehicle user establishes a twin association relationship on the edge server based on the construction requirements and network status, including: (1) Vehicle user u passes through base station BS m Transmit state information data to the edge server ES m , that is, with the edge server ES m The association between the twin of the vehicle user in time slot t and the edge server is represented by a binary variable matrix Φ(t), where The twin representing vehicle user u is deployed to the edge server ES m Synchronize real-time data, otherwise (2) When the dynamic changes in edge server resources and the movement of vehicle users lead to an increase in signal distance, the twin is migrated to other edge servers and the binary variable δ m (t) represents the connection to the edge server ES m The twin migration situation is: in The twin representing vehicle user u at time slot t-1 is deployed to the edge server ES m Synchronize real-time data; (3) Maximum number of maintainable vehicle users on the edge server for: Among them, M m Indicates edge server ES m Total storage resources, t d represents a fixed duration, represents the size of storage resources required by the edge server to establish and manage 1 bit of data, D H Indicates that the edge server stores historical data of vehicle users; (4) Each user's twin can only be associated with one edge server, that is: The step S3 specifically includes the following steps: S31: Use the demand-supply model to quantify the server congestion level: Among them, π0 and π1 are the coefficients of performance degradation, |U m To deploy the twin on ES m the number of users; S32: Calculate the digital twin mapping granularity, which is determined by the edge server congestion level and the user distance, expressed as: Sy m,u (t)=λ(1-ζ m (t)h) αd(m,u) -o Among them, λ and o are constants, ζ m (t) is the server ES m The congestion degree of the edge server ES is η, which is the packet loss rate per unit congestion degree. m The distance between user u and α is the distance scaling factor; S33: Calculate the information synchronization delay of the twin updating the user u state, including the transmission delay and the calculation delay; the synchronization delay is recorded as: Among them, D u is the amount of data that updates the user's state transmission, τ is the amount of computation required per unit data volume, and The computing resources and bandwidth resources adaptively provided to the edge server and its wired base station; SNR m,u (t) represents the signal-to-noise ratio achievable at time t; With edge server ES m The timeliness of the digital twin of the associated user u is expressed as: Where T D Indicates the synchronization delay threshold of the vehicle user; S34: Using a sliding window mechanism, i.e., a decay function It represents the degree of attenuation of the satisfaction obtained from the k-th interaction compared to the satisfaction of the current interaction time slot, that is: in, is a parameter, t k represents the end time of the kth interaction time slot; When using edge server ES m When associated, the direct satisfaction of user u’s digital twin is expressed as: Where P represents the number of effective interactions in the sliding window, k represents the number of interactions, μ1 and μ2 represent the weights of digital twin timeliness and mapping granularity respectively, Sy m,u (t k ) represents the mapping granularity of the k-th interactive digital twin; In step S4, the resource remaining rate is used to quantify the relationship between the resources required for twin maintenance and the resources required for other user services. The resource remaining rate is determined by the computing resources and total computing resources adaptively allocated by the edge server, and the bandwidth and total bandwidth adaptively allocated by the base station, and is expressed as: Where κ1 and κ2 are weights, B m Indicates BS m The total bandwidth, C m Indicates ES m Total computing resources; The step S5 specifically includes the following steps: S51: Weight the direct satisfaction and indirect satisfaction to obtain the global satisfaction of user u with the digital twin after the current interaction time slot t ends: Gs m,u (t)=ωDR m,u (t)+(1-ω)IR m,u (t) Where ω is the weight coefficient; Global satisfaction is normalized to: S52: The benefits of digital twin edge synchronization are positively correlated with user satisfaction with the digital twin, expressed as: Among them, in Synchronize unit income for information; S53: Instantiation cost CO ins (t) is the cost for each server to download the corresponding software module from the cloud server to support the instantiation of the new digital twin, which is expressed as: where l ins represents the unit instantiation cost, D soft Indicates the size of the software module required for instantiation, and d(m,G) represents ES m Distance to cloud server G; The synchronization cost is the cost of real-time data transmission between the user twin and the appropriate server, which is expressed as: Where ρ is the cost of transmitting data per unit distance; d(m,u) represents the cost of transmitting data per unit distance between user u and ES. m distance; Therefore, when migration occurs, the migration cost is: δ m (t) indicates the connection to the edge server ES m Twin migration situation, d(m,m') is the edge server ES m to ES m' The distance, c M is the unit migration cost; With the goal of maximizing the long-term utility of the system, the digital twin relationship between vehicle users is optimized, namely: Where Φ(t) is the association between the vehicle user’s twin and the server, where The twin of the vehicle user is deployed to the edge server to synchronize real-time data, that is, the user's twin establishes an association relationship with the server; and T D are the synchronization delay and synchronization delay threshold of vehicle users respectively, is the maximum number of twins that can be maintained by the server; U D (t) represents the information synchronization utility, ω1 represents the benefit weight, ω2 represents the total cost weight, π1, π2 and π3 represent the weights of instantiation cost, synchronization cost and migration cost respectively; C1 ensures that each user can only deploy twins on one server; C2 represents the quantity constraint associated with each server and twins; C3 indicates that the DT synchronization delay must not exceed the maximum synchronization delay threshold.

2. The method for synchronous optimization of edge digital twin information in the Internet of Vehicles scenario according to claim 1 is characterized by: In step S6, the optimization model is first converted into a Markov decision process, and then the branch duel Q network algorithm is used to solve the optimal vehicle user digital twin and server association scheme; the Markov decision process is as follows: (1) state space: the system state is composed of the user's association state with each edge server at the previous moment, the user's location and the remaining status of each resource; (2) action space: the action space is the association action with the user twin; (3) reward function: the reward function is the difference between the benefit and cost in the current network, that is, the system utility.

3. The method for synchronous optimization of edge digital twin information in the Internet of Vehicles scenario according to claim 2 is characterized by: The branched dueling Q network adopts a structure based on the dueling dual-depth Q network. The dueling network separates the original dual-depth Q network structure into a value branch and an advantage branch, and simultaneously trains these two branches through experience replay; The Branch Duel Q Network divides multi-dimensional actions into multiple sub-actions for separate processing while maintaining shared input state, providing a degree of autonomy for each sub-operation. The network is trained and updated through interaction between the agent and the environment, ultimately achieving a converged optimal association strategy between the connected vehicle user twin and the server. The Q value Q of each sub-action of the branch duel Q network d (s,a d ) is composed of the value branch and the corresponding advantage branch, namely: Where V(s) represents the value branch under state s, Represents a set of sub-actions, A d (s,a d ) indicates a sub-action Advantage function, M represents the number of decomposable sub-actions, A d (s,a' d ) indicates a sub-action Advantage function of The ε-greedy strategy is used to select actions, namely: in Represents the first decomposed sub-action, Indicates sub-action The Q-value function, represents the Mth sub-action of the decomposition, Indicates sub-action Q-value function of The loss function is the expected value of the mean square error between each branch, that is: E (s,a,r,s') represents the mean of the following formula, M represents the number of sub-actions, d represents the sub-action sequence value, Q d (s,a d ,ω) represents sub-action a d The Q value function, r represents the reward value, λ D represents the learning rate, Q d (s',a' d ,ω) represents the sub-action a' d The Q-value function, represents the target network parameters; The TD-error formula is: where Q d (s,a d ,ω) represents sub-action a d The Q-value function, Represents the action a' that maximizes the Q value function d .

4. The method for synchronous optimization of edge digital twin information in the Internet of Vehicles scenario according to claim 3 is characterized by: The branching dueling Q network algorithm is used to solve the optimal vehicle user digital twin and server association scheme, which includes the following steps: S61: Initialize the learning rate, discount factor, soft update parameters, experience pool, neural network parameters, exploration probability, user's global satisfaction with edge servers, and randomly assign edge server indexes with unstable satisfaction. S62: The agent observes the environment and obtains the initial state; S63: In each time slot, the agent uses the ε-greedy strategy to select an action, obtains a reward, and enters the next state; S64: Store the experience into the experience replay pool; S65: Extract samples from the experience replay pool and calculate TD-error to update the neural network parameters; S66: Update the target network parameters according to the soft update method; S67: Stop when the number of iterations reaches the maximum, otherwise return to step S66.

Images