Wireless network digital twin edge association method for multi-mode terminal

Abstract

The invention discloses a wireless network digital twin edge association method for a multi-mode terminal, and belongs to the technical field of wireless communication, and the method comprises the steps: obtaining system parameters of a communication system of a to-be-constructed terminal and server association relationship; determining constraint conditions when the association relationship between the terminal and the server is constructed; combining the generated diffusion model with a deep reinforcement learning model to construct a digital twin edge correlation model; training the digital twinborn body edge correlation model to obtain a trained digital twinborn body edge correlation model; and realizing wireless network digital twin edge association by using the trained edge association model. According to the method, the low-delay requirement of the twinborn bodies of multiple types of equipment on virtual-real interaction is met, the resource use is optimized, and the future development of the 6G network is accelerated.

Description

Technical Field

[0001] The present invention relates to the field of wireless communication technologies, and in particular to a wireless network digital twin edge association method for multimodal terminals. Background Art

[0002] The sixth generation (6G) mobile communication system is moving towards a new era of interconnected intelligence, with the vision of realizing innovative Internet of Everything (IoE) applications, including but not limited to extended reality (XR), industrial manufacturing, autonomous driving, and tactile networks. These IoE applications have unprecedentedly high requirements for data transmission rate, reliability, low latency and other performance. For example, immersive multi-sensory experience pursues a data transmission rate of at least 10Gbit / s and an end-to-end latency of less than 1 millisecond, while industrial automation requires a control plane latency as low as 0.1 millisecond and a reliability of up to 99.999999%. As user behavior and information environment continue to change, 6G networks must demonstrate a high degree of adaptability and self-optimization capabilities to meet the dynamic and extreme service needs of end users. This indicates that 6G networks are moving towards full automation and intelligence, and are committed to building a more efficient, reliable and flexible wireless ecosystem to achieve the intelligent connection of all things. In order to achieve this goal, wireless network digital twin (DT) technology has emerged and has become the key to promoting the intelligent development of 6G networks.

[0003] As a cutting-edge technology, digital twins show great potential in 6G networks. Based on real-time data communication and information transmission between physical networks and their digital networks, it can comprehensively analyze, estimate, simulate and control various elements of physical networks and their dynamic changes. By building 6G digital twins, network visibility can be enhanced, data sharing can be facilitated, dynamic simulation can be accurate, and intelligent analysis and decision-making can be efficient. This technical feature not only helps to reduce trial and error costs and accelerate the innovation process, but also enables predictive maintenance and improves network operation and maintenance efficiency. However, with the development of new paradigms such as integrated sky-land-sea networks, the high mobility and unknown distribution of terminal devices have brought higher complexity to the management of digital twins. In addition, the application of millimeter wave and terahertz communications has led to a sharp increase in the amount of sensing and communication information data, posing severe challenges to real-time data processing capabilities. With the popularization of multimodal terminals, how to efficiently associate the digital twins of these terminals on edge servers has become an urgent problem to be solved.

[0004] With the continuous development of 6G networks, multimodal mobile terminals such as wearable devices, drones, and self-driving vehicles are becoming increasingly popular. The amount of data generated by these devices is huge and diverse, which puts higher requirements on the real-time performance and reliability of the network. In order to realize the intelligent connection of all things in a complex and changeable network environment, creating twins that can synchronize the operating status of physical devices in real time in the edge server connected to the base station has become an effective solution. However, the edge association of twins of multimodal terminals faces many challenges. On the one hand, creating and running digital twins of massive terminal devices requires a lot of computing resources, communication resources, and storage resources, which brings a heavy burden to the network. On the other hand, the twins of different types of terminal devices have different requirements for low latency of virtual-real interaction. How to achieve flexible twin edge association for different requirements has become an urgent problem to be solved. In addition, with the continuous increase in the number and types of terminal devices, how to efficiently manage these twins and optimize resource usage is an important direction of current research. Summary of the invention

[0005] The present invention provides a wireless network digital twin edge association method for multimodal terminals to solve the problem of deploying twins of different types of terminal devices on the edge side in building wireless network digital twins.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] On the one hand, the present invention provides a wireless network digital twin edge association method for multimodal terminals, which uses a generative diffusion model to select twin servers and generate twin placement strategies, and combines deep reinforcement learning to reduce the latency of deploying multimodal terminal twins. In this regard, the method includes:

[0008] Obtaining system parameters of a communication system for establishing a terminal-server association relationship;

[0009] Based on system parameters, determine the constraints when establishing the association relationship between the terminal and the server;

[0010] Combine the generative diffusion model with the deep reinforcement learning model to build a digital twin edge association model;

[0011] Under the constraints, the digital twin edge association model is trained to meet the delay requirements between different device twins and terminal devices, minimize the average delay between all twins and terminal devices, and maximize the use of network resources to obtain a trained digital twin edge association model;

[0012] The trained digital twin edge association model is used to realize the edge association of wireless network digital twins.

[0013] Furthermore, the system parameters include: the maximum computing resources that can be provided by the base station side edge server, the number of multimodal terminal devices connected to the current base station, the number of base station side edge servers, the amount of data transmitted from the terminal to the server, and the maximum delay that the terminal can accept.

[0014] Furthermore, the determining of the constraint conditions for establishing the association relationship between the terminal and the server based on the system parameters includes:

[0015] Based on system parameters and different relationships between terminals and servers, the minimum computing resources required by the server to meet the maximum latency requirement of the terminal are calculated. The minimum latency of terminal data transmission is calculated when the computing resources of the server are maximized.

[0016] Determine the constraints when establishing the association relationship between the terminal and the server; wherein the constraints include: for all terminals and servers, the data transmission delay cannot exceed the maximum delay that the terminal can accept; and the total computing resources allocated by the server to the terminal cannot exceed its maximum computing resources.

[0017] Furthermore, the generation diffusion model is combined with the deep reinforcement learning model to construct a digital twin edge association model, including:

[0018] The Markov decision process is used to represent the placement and edge association relationship of digital twin servers. The generative diffusion model is used as the policy generation network of the deep reinforcement learning model to construct the edge association model of the digital twin. Among them, the policy generation network obtains the current environmental state through observation, and uses the generative diffusion model to obtain the action after removing the noise from the Gaussian noise through the back diffusion denoising algorithm according to the current environmental state.

[0019] Furthermore, the state space of the deep reinforcement learning model consists of three parts: terminal state, server state, and communication environment state;

[0020] The terminal status includes: the current terminal transmission power and the amount of terminal data transmitted to build the digital twin;

[0021] The server status includes: the currently available computing resources and communication environment status of the server; wherein the communication environment status includes: the current channel bandwidth and the current channel power gain;

[0022] The action in the model refers to the operation of placing the digital twins of multiple terminals on different twin servers; the action space includes the digital twin server placement strategy and association strategy; among them, the goal of each action is to minimize the average delay and maximize the total utility of maintaining the digital twin within its coverage area.

[0023] Furthermore, the reward function R of the deep reinforcement learning modelt It is expressed as:

[0024] R t =μ·R L +ν·R U

[0025] Among them, R L is the delayed reward; R U is the utility reward; μ and ν represent R L and R U Further, the delayed reward R L It is expressed as:

[0026] R L =-T(p,φ(i,k))

[0027] in,

[0028]

[0029]

[0030] Among them, u i represents the i-th terminal; d k represents the kth digital twin server; N is the total number of terminals; m is the number of edge servers on the base station side selected as digital twin servers; φ(i,k) represents the association relationship between the i-th terminal and the k-th digital twin server. When the i-th terminal is associated with the k-th digital twin server, φ(i,k)=1, otherwise φ(i,k)=0; is the total delay of the ith terminal twin; is the twin update delay per time slot; η represents the number of time slots required for twin update; and The expression is:

[0031]

[0032] in, Δt represents the time interval between two updates; ΔD i Indicates the size of the updated data; D i is the amount of data transmitted from the i-th terminal to the server; r ij is the maximum uplink rate between the i-th terminal and the j-th base station; is the delay caused by transmitting one unit of data within each unit distance; d(j,k) is the distance between the jth base station and the kth twin server; f ik It represents the computing resources allocated by the k-th twin server to the i-th terminal to build the digital twin; p represents the placement strategy of the digital twin server.

[0033] Furthermore, the utility reward R U It is expressed as:

[0034] R U =U p (t)

[0035] Among them, U p (t) = U(t) - λ·m; λ represents the cost required to place a unit digital twin server; m is the number of edge servers selected as digital twin servers on the base station side; N is the total number of terminals; e is the natural number base; τ ui The maximum delay that the terminal device can accept.

[0036] On the other hand, the present invention further provides an electronic device, comprising a processor and a memory; wherein the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the above method.

[0037] In yet another aspect, the present invention further provides a computer-readable storage medium, wherein at least one instruction is stored in the storage medium, and the instruction is loaded and executed by a processor to implement the above method.

[0038] The technical solution provided by the present invention is aimed at multimodal mobile terminals such as wearable devices, drones, and autonomous driving vehicles with significant feature differences, and creates a twin in the edge server connected to the base station that can synchronize the operating status of physical devices in real time. Considering that maintaining the digital twins of massive terminal devices requires a large amount of resources, including computing resources, communication resources, storage resources, etc., the present invention uses a generative artificial intelligence algorithm to achieve flexible twin edge associations based on the differences in the requirements of twins of different types of terminal devices, meet the low latency requirements of multiple types of twins for virtual-real interaction, and optimize the use of multiple types of resources. In this regard, the present invention combines the generative diffusion model with the deep reinforcement learning algorithm DRL, and uses the policy generation network and the policy evaluation network to generate the optimal solution, providing the optimal twin placement position for multiple types of terminals in the 6G wireless network's intelligent interconnection of all things scenario, and finding the optimal edge association relationship between them and the edge server. Its beneficial effects include at least:

[0039] The present invention can be used to build wireless network digital twins in the 6G intelligent interconnection scenario, realize zero-contact and autonomous wireless network management, and use the generative diffusion model technology combined with DRL to realize the twin edge association of multimodal terminals, solve the twin placement problem, meet the differentiated delay requirements of virtual-real interaction between twins and physical entities, and optimize the use efficiency of communication resources and computing resources. It can not only meet the low-latency requirements of virtual-real interaction of multi-type device twins, but also optimize resource usage, thereby accelerating the future development of 6G networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 is a schematic diagram of an overall system model provided by an embodiment of the present invention;

[0042] Figure 2 It is a framework diagram of a wireless network digital twin edge association method for multimodal terminals provided in an embodiment of the present invention;

[0043] Figure 3 It is a schematic diagram of the execution flow of the wireless network digital twin edge association method for multimodal terminals provided in an embodiment of the present invention;

[0044] Figure 4 It is a system block diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0045] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0046] First of all, it should be noted that in the embodiments of the present invention, words such as "exemplarily" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "exemplarily" is intended to present the concept in a concrete way. In addition, in the embodiments of the present invention, the meaning expressed by "and / or" can be both, or it can be either of the two.

[0047] First embodiment

[0048] In order to solve the problem of deploying twins of different types of terminal devices on the edge side in building wireless network digital twins, this embodiment provides a wireless network digital twin edge association method for multimodal terminals. Figure 1 It shows the deployment of the twins of terminal devices on the edge side in the wireless network digital twin, selects a suitable base station side edge server as the digital twin server, and selects a suitable twin edge association method to associate the terminal with the twin server, and finally creates a digital twin of the running terminal on the server.

[0049] Among them, it should be noted that choosing the optimal twin placement and the optimal edge association can not only meet the delay requirements between different device twins and terminal devices, but also minimize the average delay between all twins and terminal devices, and maximize the use of network resources. The delay in building and maintaining digital twins mainly includes two parts: 1) construction delay; 2) synchronization delay. Among them, the construction delay can be divided into three aspects, namely: wireless transmission delay, wired transmission delay, and data calculation delay. Wireless transmission: transmit the operating data of terminal i to the base station; wired transmission: transmit from base station j to digital twin server k; data calculation delay: the calculation time required for digital twin server k to build the terminal digital twin.

[0050] Define U to represent the terminal device set, u i represents the i-th terminal, the total number of multimodal terminals accessing the base station is N, B represents the base station set, b j represents the jth base station, the total number of base stations and their edge servers is M, DT represents the set of digital twin servers, and d k represents the kth digital twin server, and the number of edge servers on the base station side selected as digital twin servers is m, m≤M.

[0051] Definition D i is the amount of data transmitted from the i-th terminal to the server. The maximum uplink rate r between terminal i and base station j is calculated according to the Shannon formula. ij for

[0052]

[0053] Among them, p ij represents the transmit power of terminal i, h ij represents the channel power gain of terminal i, and w is the bandwidth of the transmission channel.

[0054] The wireless transmission delay T is obtained by the uplink rate ij com for

[0055]

[0056] definition The wired transmission delay is the delay caused by transmitting one unit of data per unit distance. for

[0057]

[0058] Among them, d(j,k) is the distance between base station j and twin server k.

[0059] Define f ikrepresents the computing resources allocated by the twin server k to the terminal i to build the digital twin. Then the computing time of the digital twin server k is for

[0060]

[0061] In summary, the total delay in building the terminal i-twin is for

[0062]

[0063] Since the digital twin server k needs to continuously interact with the terminal i to keep consistent with the current operating state of the terminal, the size of the update data is defined as ΔD i , you can get the delay of an update for

[0064]

[0065] Twin update delay per time slot for

[0066]

[0067] Where Δt represents the time interval between two updates.

[0068] Define the association relationship between terminal i and digital twin server k as φ(i,k), which has the following two results:

[0069]

[0070] Among them, if the i-th terminal is associated with the k-th digital twin server, φ(i,k)=1, otherwise it is 0.

[0071] Since one terminal device can only correspond to one digital twin server,

[0072]

[0073] Define p to represent the placement strategy of the digital twin server, expressed as

[0074] p={p k |k∈[1,m]}

[0075] Among them, p k Represents the digital twin server d k location.

[0076] Under this p placement strategy, the average delay is calculated by

[0077]

[0078] Among them, η represents the number of time slots required for twin update; Indicates the delay required for twin update.

[0079] Definition U i (t) is the terminal utility, which is determined by the actual delay and the maximum delay that the terminal can accept.

[0080]

[0081] The utility sum U(t) of all terminals is

[0082]

[0083] Define the total utility U of the placement process p (t) is

[0084] U p (t) = U (t) - λ·m

[0085] Among them, λ represents the cost required to place a unit digital twin server, and m is the number of edge servers selected as digital twin servers on the base station side.

[0086] The goal of the digital twin placement problem is to find a placement location and relationship solution that minimizes average latency and maximizes utility. It must meet the following constraints:

[0087] Constraints: 1) The interaction time t between the terminal device U and its corresponding digital twin server DT ui Should not exceed the delay requirement of the terminal device τ ui

[0088] 2) For all twin servers, ensure that the computing resources consumed do not exceed the total computing capacity of the server F k ;

[0089] minT(p,φ(i,k)),maxU p (t)

[0090]

[0091] Now there is a group of multimodal terminals U, a group of base stations B with edge servers, and a group of twin servers DT used to build digital twins. The goal is to select appropriate edge servers as twin servers in a way that minimizes latency and maximizes overall utility, and to place the built twins on suitable twin servers. Due to resource constraints, a single digital twin server can only build digital twins for a limited number of terminals, and the digital twin of a single terminal can only be built by a single server.

[0092] To this end, a Markov decision process is defined to solve the twin placement and edge association problems.

[0093] Given an initial state s t , the agent takes an action a in the environment at each step T = 0, 1, ... t t ∈A(t) and obtain reward r t From a state t Transition to the next state s t+1 The action decision is selected according to the strategy. Define π θ is the action strategy. The goal is to maximize the cumulative reward r(s t ,a t ) makes the total reward R as large as possible. The MDP model of the placement and edge association problem can be described as a tuple (s t ,a t ,r t ,s t+1 ).

[0094] 1) State space: The state space consists of three parts: terminal state (current terminal transmission power p ij , the amount of terminal data D transmitted by building a digital twin i ); server status (the current available computing resources of the kth server ), communication environment status (current channel bandwidth w, current channel power gain h ij ).Right now

[0095]

[0096] 2) Action space: The operation of placing the digital twins of multiple terminals on different twin servers is defined as A(t), including the digital twin server placement strategy and the association strategy. The operation of digital twin placement is defined as p, that is, p = {p k |k∈[1,m]} means placing the digital twin server k on the edge server. The association strategy between terminal i and server k is represented as φ(i,k), and the action space can be expressed as:

[0097]

[0098] 3) Reward function: Design the reward function so that each action can make the best decision for edge association. The goal of each action is to minimize the average delay of maintaining the digital twin within its coverage area and maximize the total utility. The reward function mainly consists of two parts: delay reward R L and utility reward R U .

[0099] Where the delayed reward R Lis related to the total latency of the placement strategy, defined as

[0100] R L =-T(p,A)

[0101] The utility reward R U It is related to the total utility of all terminals and the cost of placing servers, defined as

[0102] R U =U p (t)

[0103] Define the total reward R t for

[0104] R t =μ·R L +ν·R U

[0105] μ and ν represent the delayed reward R L and utility reward R U The weight factor of .

[0106] Figure 2 The figure shows a framework diagram of an algorithm proposed in the present invention that combines the Generative Diffusion Model (GDM) with DRL to solve the digital twin server placement and association problems in order to minimize the delay between the terminal and the digital twin, improve the terminal utility, and reduce energy consumption.

[0107] The purpose of using GDM is to use its powerful generative ability to model complex data distributions and generate high-quality samples. In network optimization tasks, GDM can accurately and quickly find the best solution through dynamic online learning and optimization in the absence of expert datasets.

[0108] The algorithm that combines GDM and DRL has many advantages over the traditional DRL algorithm. The traditional DRL algorithm requires a lot of interaction with the environment to learn effectively, which consumes a lot of resources and time. Relying on GDM's powerful ability to model complex data distributions, the number of samples required can be reduced; traditional DRL algorithms have difficulty coping with environments characterized by complex high-dimensional states and action spaces, while GDM can provide a more effective environmental representation by accurately capturing the underlying data distribution; GDM can also help solve the problems of instability and slow convergence that the DRL algorithm may encounter, providing a more stable and efficient learning process.

[0109] The overall algorithm architecture consists of four parts: environment, replay buffer, strategy generation network, and strategy evaluation network.

[0110] The environment consists of the terminal, the edge server on the base station side, and the communication environment, and its state space is composed of the states of each part.

[0111] Replay buffer: First, the environment provides a state to the policy generation network, then the policy generation network generates an action and acts on the environment, obtains the action reward, and enters the next state. This process is recorded in the replay buffer so that it can be used as a sample for training the policy evaluation network later.

[0112] Define the strategy generation network as N θ , N θ By denoising the initial distribution, the optimal placement and association strategy is generated in the environment, where θ is N θ The network is trained with parameters of , and outputs the optimal solution through the back diffusion process of GDM.

[0113] GDM gradually introduces Gaussian noise through a forward diffusion process to generate targets for the denoising neural network. The neural network is then trained to reverse the noise process and recover the original data and content.

[0114] 1) Forward diffusion process

[0115] The forward diffusion process can be modeled as a Markov chain with T steps. x0 represents the original data. At each step of the Markov chain, i.e., step t, a vector with variance β t Gaussian noise is added to x t-1 So the distribution is q(x t |x t-1 ) t This process is expressed as

[0116]

[0117] Among them, q(x t |x t-1 ) is a normal distribution with mean μ t and variance Σ are the features, Ι is the identity matrix, indicating that each dimension has the same standard deviation β t .

[0118] From the original data x0 to the final x T The posterior probability is expressed as

[0119]

[0120] Define α t =1-β t , x t It can be expressed as

[0121]

[0122] Among them, ∈0…∈ t-1 ~N(0,Ι). According to the distribution, we can get x t :

[0123]

[0124] 2) Reverse diffusion process

[0125] When T is large, x T Approximately isotropic Gaussian distribution, using parameterized algorithm model π θ For q(x t-1 |x t ) is estimated, and then x is sampled from N(0,Ι) T , perform the reverse process and obtain the sample of q(x0).

[0126] π θ (x t-1 |x t )=N(x t-1 ;μ θ (x t ,t),Σ θ (x t ,t))

[0127] So we can get from x T The trajectory to x0 is

[0128]

[0129] After adding conditional information s in the denoising process, π θ (x t-1 |x t ,s) can be simulated as a noise prediction model, and its covariance matrix is ​​fixed as

[0130] Σ θ (x t ,s,t)=β t Ι

[0131] Average value μ θ (x t ,s,t) is expressed as

[0132]

[0133] Then first x T ~N(0,Ι) is sampled, and then sampled from the reverse diffusion chain with θ as the parameter, that is,

[0134]

[0135] Where ∈~N(0,Ι),t=1…T, the loss function is

[0136]

[0137] The action decision is based on the strategy π θ Generated. Using the GDM diffusion model, according to the environmental state through the strategy π θ After removing the noise, we get the action. θ (s t ) means that under this strategy, in state s t The action to be taken when the target is to maximize the cumulative reward is defined as R(s t ,π θ ): indicates that in π θ Strategy from s t The total reward accumulated until the end of the state.

[0138]

[0139] Among them, γ∈(0,1] is the reward discount factor, the purpose of which is to highlight the impact of current rewards on future rewards. The closer to the front, the more important the reward obtained by the action.

[0140] In the training strategy generation network, adjust the parameter θ so that θ The reward and maximum are obtained, that is

[0141]

[0142] Define the strategy evaluation network as N Q , N Q The Q-Learning algorithm is used, which is defined as Q(s t ,a t ), denoted as s t Take action a in state t Then the total reward that can be obtained until the end is expressed as:

[0143]

[0144] And N Q The network goal is to be in a certain state s t After taking the best action, the reward is maximized, and the N obtained by training Q The network can obtain the best action to take in each state.

[0145]

[0146] Training N QThe network samples are sampled from the playback buffer using the temporal difference method TD (Temporal Difference), and the parameters are updated using the dual Q function: Q function and target The function interval is updated. In two consecutive states s t ,s t+1 In the definition of a state s t Take a t The total reward obtained is y t ,y t =Q(s t ,a t ), using a t The reward for the action is r t , define the next state s t+1 Take the best action t+1 The total reward obtained is y t+1 ,Right now

[0147] The loss function L1 is defined as: L1 = E|y t -(r t +γy t+1 )|

[0148] You can train N by minimizing L1 Q Network, that is

[0149]

[0150] And after step C, update Right now

[0151] According to N Q Network training N θ , according to N θ Strategy π in the network θ The generated action a t Not all are necessarily optimal. Q The network updates the original policy.

[0152]

[0153] The loss function L2 is defined as: L2 = E|Q(s t ,a t )-R(s t ,π θ )|

[0154] Training N θ Minimize L2, that is

[0155]

[0156] The specific execution process of the above method is as follows Figure 3 As shown, the following steps are included:

[0157] Step 1: Obtain the maximum computing resource F that the edge server on the base station side can provide k , the number of terminal devices N, the number of edge servers on the base station side M, the amount of data transmitted from the terminal to the server D i , the maximum delay τ that the terminal can accept ui .

[0158] Step 2: According to the different relationships between the terminal and the server, calculate the minimum computing resources that the server needs to provide while meeting the maximum latency requirement of the terminal, and calculate the minimum latency for the terminal to transmit data when maximizing the utilization of the server computing resources.

[0159] Among them, it should be noted that when building a digital twin for a multimodal terminal, it is necessary to select a base station side edge server that matches the differentiated needs of the terminal as the twin server, and select a suitable server and terminal association relationship, so as to build and run a digital twin for the terminal on the server. The terminal sends the data required to build the twin to the twin server through wireless transmission between it and the base station and wired transmission between the base station and the twin server, completes the data calculation required for twin construction on the twin server, and then returns the relevant data for feedback. When choosing to place the twin server and the association relationship between the terminal and the server, for all terminals and servers, the data transmission delay is required not to exceed the maximum delay τ that the terminal can accept. ui , the total computing resources allocated by the server to the terminal cannot exceed its maximum computing resource F k .

[0160] Step 3: Initialize the GDM-based strategy generation network N θ Weight θ, playback buffer, N Q Q-function and objective in the network function.

[0161] It should be noted that the use of traditional deep reinforcement learning (DRL) algorithms is prone to instability or even non-convergence when the state space and action space are large and high in dimension. Therefore, the present invention uses an algorithm combining GDM and DRL to solve the edge association method of wireless network digital twins, and uses Markov decision process (MDP) to represent the placement of digital twin servers and edge association relationships.

[0162] Step 4: Initialize the current environment state p ijis the current terminal device transmit power, D i The amount of terminal data transmitted to build the digital twin, is the current available computing resources of the kth server, the current channel bandwidth w and the channel power gain h ij .

[0163] Step 5: Calculate the current state s t Next, place the twin server and build a twin for the terminal to get the instant reward R t , R t =μ·R L +ν·R U , where R L is the delay reward generated by the terminal transmitting data under the current placement and association state, which is negatively correlated with the delay. U It is the utility reward under the current placement and association state, which is determined by the total utility of the terminal and the cost incurred by placing the twin server.

[0164] Step 6: Strategy Generation Network N θ Get the current environment state s by observing t , through the reverse diffusion denoising algorithmπ θ Remove the noise from the Gaussian noise to get the action p means placing the digital twin server on the edge server, φ(i,k) represents the association between the i-th terminal and the k-th server. When the value is 1, the two are associated with each other, otherwise they are not associated with each other.

[0165] It should be noted that the above steps obtain the current state s by observing t , the network N is generated by the strategy θ Using the back diffusion algorithmπ θ After removing the noise in the environment that affects the generation of the best action, the action A(t) is obtained. The present invention generates a diffusion model based on GDM to obtain the action according to the current environmental state. Compared with the traditional action-evaluation deep learning network that uses an exploratory algorithm to randomly generate actions, the traditional method requires a lot of learning time and is not easy to converge. The use of the GDM method can generate higher quality actions, thereby providing a more stable and efficient learning process.

[0166] Step 7: Generate action a from the strategy generation network t With the current environment t Interact and get a new state s t+1 , calculate the immediate reward r generated by this action t , and this process (s t ,a t ,r t ,s t+1 ) is stored in the playback buffer.

[0167] It should be noted that the goal of the edge association problem of wireless network digital twins is to minimize the delay and maximize the utility while meeting the transmission delay requirements between multimodal terminals and digital twins. The total reward R is composed of the delay reward R L and utility reward R U The delay reward function is negatively correlated with the delay generated under the placement strategy, that is, R L = -T(p,A), and the utility reward function is determined by the total utility of the terminal U(t) and the cost of placing the twin server, that is, R U =U(t)-λ·m.

[0168] Step 8: Randomly extract small batches of samples from the playback buffer and put them into the policy evaluation network N Q , N Q Using the Q function, we can get Q(s t ,a t ), denoted as s t Take action a in state t Then the total reward accumulated until the end. Calculate Q(s) obtained by all samples t ,a t ).

[0169] Step 9: Q(s) obtained through multiple samples t ,a t ), using the temporal difference method with the Q function and the objective The network parameters are updated by updating the function interval. Every C time steps, the parameters of the current Q function are copied to the target function.

[0170] Step 10: Continuously iterate and optimize, update parameters until N Q The parameter no longer changes, and Q(s t ,a t ) can reach the maximum value, and the training is finished. Q network.

[0171] It should be noted that, through the above steps, according to the previous state s t , the strategy generation network outputs action a t , after interacting with the environment, get an immediate reward r t and the next state s t+1 , this process (s t ,a t ,r t ,s t+1) is recorded in the playback buffer. Then samples are extracted and the time difference method is used to compare the Q function and the target The function interval update method is used to train and update N Q The network parameters. Find and update the optimal action strategy based on the reward obtained, repeat this cycle, and finally converge to the optimal state. At the end of the training, N Q network.

[0172] Step 11: Evaluate the network N using the trained strategy Q To train N θ . Best N θ The network enables the cumulative reward Q(s t ,a t ) is maximum, that is, the input state s t Output the best action a t . The mean-variance loss function is used to update N through gradient back propagation. θ Parameters.

[0173] Step 12: Iterate and optimize until N θ The parameters no longer change, and the output action according to a certain state can make Q(s t ,a t ) is the largest, the training ends, and according to N θ The network generation action completes the placement and edge association of the optimal digital twin server.

[0174] It should be noted that the trained strategy evaluation network N Q To train the policy generation network N θ , the purpose is to make the N obtained by training θ The network is in input state s t After that, an optimal action a can be output t So that Q(s t ,a t ) is the largest, and the mean variance loss function is used to update N through gradient back propagation. θ Then continue to iterate and optimize until N θ The parameters no longer change, and its action generation strategy is consistent with the realization of maximizing Q(s t ,a t ) has the same action strategy, and the training ends. θ The network completes the optimal placement and edge association of digital twin servers.

[0175] In summary, this embodiment provides a wireless network digital twin edge association method for multimodal terminals, which can be used to build wireless network digital twins in 6G all-in-one intelligent scenarios, realize zero-contact, autonomous wireless network management, and use the generation diffusion model technology combined with DRL to realize the twin edge association of multimodal terminals, solve the twin placement problem, meet the differentiated latency requirements of virtual-real interaction between twins and physical entities, and optimize the use efficiency of communication resources and computing resources. It can not only meet the low latency requirements of multi-type device twins for virtual-real interaction, but also optimize resource usage, thereby accelerating the future development of 6G networks.

[0176] Second embodiment

[0177] This embodiment provides an electronic device, such as Figure 4 As shown, the electronic device includes: a processor and a memory; wherein the processor and the memory can be connected via a communication bus; the memory stores at least one instruction, and the instruction is loaded and executed by the processor to implement the method of the first embodiment. In addition, the electronic device may also include a transceiver, the processor and the transceiver can be connected via a communication bus, and the transceiver is used to communicate with other devices.

[0178] Next, combine Figure 4 The following is a detailed introduction to the various components of the electronic device:

[0179] Among them, the processor is the control center of the electronic device, and the electronic device may include multiple processors, each of which may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may be a processor or a general term for multiple processing elements. For example, the processor is one or more central processing units (CPUs), or other general-purpose processors, application specific integrated circuits (ASICs), or one or more integrated circuits configured to implement an embodiment of the present invention, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (field programmable gate arrays, FPGAs), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. The processor may execute various functions of the electronic device by running or executing software programs stored in the memory and calling data stored in the memory.

[0180] In a specific implementation, as an embodiment, the processor may include one or more CPUs, such as Figure 4 The CPU0 and CPU1 shown in the figure are, of course, only exemplary.

[0181] The memory is used to store the software program for executing the solution of the present invention, and the execution is controlled by the processor. The specific implementation method can refer to the above method embodiment and will not be repeated here.

[0182] Optionally, the memory may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory may be integrated with the processor or exist independently and accessed through the interface circuit ( Figure 4 (not shown) is coupled to the processor, which is not specifically limited in this embodiment of the present invention.

[0183] The transceiver may include a receiver and a transmitter ( Figure 4 The receiver is used to implement the receiving function, and the transmitter is used to implement the sending function. The transceiver can be integrated with the processor or exist independently and communicate with the electronic device through the interface circuit ( Figure 4 (not shown) is coupled to the processor, which is not specifically limited in this embodiment of the present invention.

[0184] In addition, it should be noted that Figure 4 The structure of the electronic device shown in the figure does not constitute a limitation on the device, and the actual device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently. In addition, the technical effects achieved by the electronic device when executing the method of the first embodiment above can refer to the technical effects described in the first embodiment above, so they are not repeated here.

[0185] Third embodiment

[0186] This embodiment provides a computer-readable storage medium, which stores at least one instruction, and the instruction is loaded and executed by a processor to implement the method of the first embodiment. The computer-readable storage medium may be a ROM, a random access memory, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc. The instructions stored therein may be loaded by a processor in a terminal to execute the method.

[0187] In addition, it should be noted that the present invention can be provided as a method, an apparatus or a computer program product. Therefore, the embodiment of the present invention can be in the form of a full or partial hardware embodiment, a full or partial software embodiment or an embodiment combining software and hardware. Moreover, when implemented using software, the embodiment of the present invention can be in the form of a computer program product implemented on one or more computer-usable storage media containing computer-usable program codes. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center containing one or more available media sets. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a DVD), or a semiconductor medium. The semiconductor medium may be a solid state hard disk.

[0188] The embodiments of the present invention are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0189] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, so that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable terminal device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0190] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of more restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements. In addition, the term "and / or" is only an association relationship describing the associated objects, indicating that there can be three relationships, for example, A and / or B, which can represent: A exists alone, A and B exist at the same time, and B exists alone, wherein A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding. "At least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can be represented by: a, b, c, ab, ac, bc or abc, where a, b, c can be single or plural.

[0191] In addition, it can be understood that in various embodiments of the present invention, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0192] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0193] In several embodiments provided by the present invention, it should be understood that the disclosed equipment, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of functional modules / units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The unit described as a separate component may or may not be physically separated, and the component displayed as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, each functional unit in each embodiment of the present invention can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit.

[0194] If the method is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0195] Finally, it should be noted that the above is only a preferred embodiment of the present invention. It should be pointed out that although the preferred embodiment of the present invention has been described, for ordinary technicians in this technical field, once the basic creative concept of the present invention is known, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the protection scope of the present invention. Therefore, the attached claims are intended to be interpreted as including the preferred embodiment and all changes and modifications that fall within the scope of the embodiments of the present invention.

Claims

1. A wireless network digital twin edge association method for multimodal terminals, characterized in that: The wireless network digital twin edge association method for multimodal terminals includes: Obtaining system parameters of a communication system for establishing a terminal-server association relationship; Based on system parameters, determine the constraints when establishing the association relationship between the terminal and the server; Combine the generative diffusion model with the deep reinforcement learning model to build a digital twin edge association model; Under the constraints, the digital twin edge association model is trained to meet the delay requirements between different device twins and terminal devices, minimize the average delay between all twins and terminal devices, and maximize the use of network resources to obtain a trained digital twin edge association model; The trained digital twin edge association model is used to realize the edge association of wireless network digital twins.

2. The wireless network digital twin edge association method for multimodal terminals according to claim 1, characterized in that: The system parameters include: the maximum computing resources that the base station side edge server can provide, the number of multimodal terminal devices connected to the current base station, the number of base station side edge servers, the amount of data transmitted from the terminal to the server, and the maximum delay that the terminal can accept.

3. The wireless network digital twin edge association method for multimodal terminals according to claim 2, characterized in that: The determining of the constraint conditions for establishing the association relationship between the terminal and the server based on the system parameters includes: Based on system parameters and different relationships between terminals and servers, the minimum computing resources required by the server to meet the maximum latency requirement of the terminal are calculated. The minimum latency of terminal data transmission is calculated when the computing resources of the server are maximized. Determine the constraints when establishing the association relationship between the terminal and the server; wherein the constraints include: for all terminals and servers, the data transmission delay cannot exceed the maximum delay that the terminal can accept; and the total computing resources allocated by the server to the terminal cannot exceed its maximum computing resources.

4. The wireless network digital twin edge association method for multimodal terminals according to claim 1, characterized in that: The generation diffusion model is combined with the deep reinforcement learning model to construct a digital twin edge association model, including: The Markov decision process is used to represent the placement and edge association relationship of digital twin servers. The generative diffusion model is used as the policy generation network of the deep reinforcement learning model to construct the edge association model of the digital twin. Among them, the policy generation network obtains the current environmental state through observation, and uses the generative diffusion model to obtain the action after removing the noise from the Gaussian noise through the back diffusion denoising algorithm according to the current environmental state.

5. The wireless network digital twin edge association method for multimodal terminals according to claim 4, characterized in that: The state space of the deep reinforcement learning model consists of three parts: terminal state, server state, and communication environment state; The terminal status includes: the current terminal transmission power and the amount of terminal data transmitted to build the digital twin; The server status includes: the currently available computing resources and communication environment status of the server; wherein the communication environment status includes: the current channel bandwidth and the current channel power gain; The action in the model refers to the operation of placing the digital twins of multiple terminals on different twin servers; the action space includes the digital twin server placement strategy and association strategy; among them, the goal of each action is to minimize the average delay and maximize the total utility of maintaining the digital twin within its coverage area.

6. The wireless network digital twin edge association method for multimodal terminals according to claim 5, characterized in that: Reward function R of deep reinforcement learning model t It is expressed as: R t =μ·R L +v·R U Among them, R L is the delayed reward; R U is the utility reward; μ and ν represent R L and R U The weight factor of .

7. The wireless network digital twin edge association method for multimodal terminals according to claim 6, characterized in that: Delayed Reward R L It is expressed as: R L =-T(p,φ(i,k)) in, Among them, u i represents the i-th terminal; d k represents the kth digital twin server; N is the total number of terminals; m is the number of edge servers on the base station side selected as digital twin servers; φ(i,k) represents the association relationship between the i-th terminal and the k-th digital twin server. When the i-th terminal is associated with the k-th digital twin server, φ(i,k)=1, otherwise φ(i,k)=0; is the total delay of the ith terminal twin; is the twin update delay per time slot; η represents the number of time slots required for twin update; and The expression is: in, Δt represents the time interval between two updates; ΔD i Indicates the size of the updated data; D i is the amount of data transmitted from the i-th terminal to the server; r ij is the maximum uplink rate between the i-th terminal and the j-th base station; is the delay caused by transmitting one unit of data within each unit distance; d(j,k) is the distance between the jth base station and the kth twin server; f ik It represents the computing resources allocated by the k-th twin server to the i-th terminal to build the digital twin; p represents the placement strategy of the digital twin server.

8. The wireless network digital twin edge association method for multimodal terminals according to claim 6, characterized in that: Utility Reward R U It is expressed as: R U =U p (t) Among them, U p (t) = U(t) - λ·m; λ represents the cost required to place a unit digital twin server; m is the number of edge servers selected as digital twin servers on the base station side; N is the total number of terminals; e is the natural number base; τ ui The maximum delay that the terminal device can accept.

Images