A NOMA-assisted cloud-edge communication computing method based on digital twins

Abstract

The present invention belongs to the field of information technology and discloses a NOMA-assisted cloud-edge-terminal communication computing method based on digital twins, including constructing a cloud-edge-terminal collaborative communication computing architecture, considering the terminal heterogeneity and edge server load balancing problems, partially offloading computing tasks, and choosing to transmit the offloaded part to the local edge server for calculation, or choosing to use the local edge server as a relay node and further transmit it to a third-party edge server or a cloud server for collaborative computing. At the same time, considering the security issues of edge computing, the problem is formulated, and the optimization objectives and corresponding constraints of resource allocation and task offloading decisions are constructed, and converted into a multi-agent MDP; the MADDPG algorithm is used to solve the problem at the digital layer, and real-time interaction with the physical layer is achieved to complete policy issuance, realize cloud-edge-terminal collaborative communication computing, improve the security and accuracy of the problem model, and meet the QoS requirements of the terminal.

Description

Technical Field

[0001] The present invention belongs to the field of information technology, and specifically relates to a NOMA-assisted cloud-edge communication computing method based on digital twins. Background Art

[0002] With the rapid adoption of the Industrial Internet of Things (IIoT) and the continuous development of intelligent manufacturing, massive heterogeneous terminals are generating a large number of computing tasks in real time, significantly increasing the demand for computing resources and making the scheduling and allocation of computing resources increasingly important. Emerging computing paradigms such as edge computing and cloud computing offer effective options for collaborative computing, significantly alleviating the computing pressure on resource-constrained terminals and improving task processing capabilities. Currently, the mainstream task offloading solution utilizes edge computing to address resource allocation and task offloading. However, the computational pressure of massive resource-intensive and latency-sensitive tasks makes a single offloading model inadequate. Therefore, cloud-edge-device collaborative architectures, which combine the advantages of cloud and edge computing, are gaining increasing research and application.

[0003] Against this backdrop, cloud-edge-device collaborative solutions place higher demands on low-latency and highly reliable communication methods. Furthermore, the unpredictable nature of industrial internet scenarios poses challenges to collaborative computing. Non-Orthogonal Multiple Access (NOMA) communication technology is considered a promising approach for enabling multiple users to operate in the same frequency band, mitigating co-channel interference by utilizing Successive Interference Cancellation (SIC). Digital twin technology enables real-time monitoring of physical entities and the construction of highly realistic virtual models, offering a new solution for the precise and real-time scheduling of communication and computing resources in intelligent manufacturing. However, integrating these new technologies with cloud-edge-device communication and computing methods remains challenging, and complex problem models require intelligent and efficient optimization algorithms.

[0004] For example, the Chinese invention patent with publication number CN 116886703 A discloses a cloud-edge collaborative computing offloading method based on priority and reinforcement learning, and proposes a cloud-edge collaborative computing offloading method applied to the Internet of Things scenario. In terms of problem model construction, this patent (1) does not consider the heterogeneity of terminals and edge servers in terms of computing resources and QoS requirements, and lacks consideration for the load balancing problem of edge servers. As a result, in actual applications, some edge servers are overloaded while others are idle, resulting in low resource utilization and possibly affecting the offloading success rate, further affecting the overall performance of the system. (2) The offloading method is binary offloading, not proportional offloading, which does not fully reflect the ability of cloud-edge multi-layer collaborative task processing. The simple offloading method cannot adapt to complex application scenarios, limiting the flexibility and scalability of the system. There is no innovation in communication technology, and traditional wireless communication methods are used. The communication security issues between the edge and the terminal are not considered, and the low-latency and high-reliability communication required for large-scale links cannot be met. (3) The use of traditional deep learning algorithms lacks dynamic adaptability and scalability. Faced with the real-time dynamic changes and large-scale connections of the industrial Internet, it is impossible to complete the scheduling strategy formulation in a short time, limiting the application potential of its technology.

[0005] For example, the Chinese invention patent with publication number CN 117591297 A discloses a DDPG-based edge computing offloading method for smart communities. It proposes an edge computing offloading method for smart communities. However, this patent (1) does not fully consider the heterogeneity of terminal devices. All device tasks are offloaded using binary offloading. This simple offloading method cannot adapt to the computing power and resource requirements of different device tasks in complex applications, resulting in low resource utilization. (2) When offloading tasks to collaborative edge servers, the authentication issue is not considered, which poses a potential security risk. In addition, the communication method used is only simple basic wireless communication, which lacks innovation and technological breakthroughs and fails to provide sufficient security and communication efficiency. (3) When the community server obtains global information, it lacks the necessary technical support, resulting in untimely updates of global information. In addition, the DDPG algorithm used in the patent has insufficient processing efficiency in the face of large-scale access scenarios due to the existence of high-dimensional state space and action space. The timeliness issue limits the practical application of the algorithm.

[0006] Therefore, it is necessary to design cloud-edge communication computing methods to deal with the computing-intensive and resource-saving computing tasks generated in real time by heterogeneous terminals with limited resources in the industrial Internet. Summary of the Invention

[0007] In response to the deficiencies of the above-mentioned background technologies, the present invention provides a NOMA-assisted cloud-edge communication computing method based on digital twins, and proposes a cloud-edge collaborative communication computing method for industrial Internet. This method combines NOMA communication technology to build a multi-layer communication computing structure, and constructs a problem model based on terminal heterogeneity and edge server load balancing. Real-time monitoring and simulation of physical entities are achieved through digital twins, and the improved multi-agent deep reinforcement learning algorithm MADDPG algorithm is used to learn and formulate resource allocation and task offloading strategies, ultimately reducing the system's time delay and energy consumption, and achieving safe and efficient resource allocation and task offloading.

[0008] The purpose of the present invention is achieved through the following technical solutions, which provide a NOMA-assisted cloud-edge communication computing method based on digital twins, and the method includes the following steps:

[0009] Step S1: Build a cloud-edge-end collaborative communication computing architecture.

[0010] S1.1, establish the physical layer in industrial scenarios, including three layers: cloud, edge and terminal. The cloud layer is a cloud server with powerful computing capabilities, the edge layer includes edge servers with different computing capabilities and load capacities, and the terminal layer includes terminals in different industrial scenarios.

[0011] S1.2, through real-time monitoring of physical entities, a corresponding digital twin is constructed based on the data mapping of the physical entity, i.e., the digital layer, the physical layer, and the digital layer interact with each other in real time;

[0012] Step S2: Based on the constructed cloud-edge collaborative communication computing architecture, taking into account the heterogeneity of terminals and the load balancing of edge servers, partially offload the computing tasks generated by the terminals in real time. The offloaded tasks can be transferred to the local edge server for calculation, or the local edge server can be used as a relay node and further transferred to a third-party edge server or cloud server for collaborative computing. At the same time, the security issues of edge computing are considered, the problem is formulated, and the optimization objectives and corresponding constraints for resource allocation and task offloading decisions are constructed.

[0013] Step S3, converting the optimization objective and corresponding constraints into a multi-agent MDP;

[0014] In step S4, the multi-agent deep reinforcement learning algorithm MADDPG is used at the digital layer to solve the multi-agent MDP, implement the scheduling strategy formulation that takes load balancing into consideration, and interact with the physical layer in real time to complete the strategy issuance, realizing cloud-edge-end collaborative communication computing.

[0015] Preferably, in step S1, the physical layer includes terminals, edge servers and a cloud server , terminals and edge servers use sets and Indicates that the digital layer is constructed as .

[0016] Preferably, in step S2, the terminal heterogeneity and edge server load balancing issues are considered, as follows:

[0017] The terminal generates computing tasks in real time. The task parameters generated on the terminal are expressed as , Indicates the amount of generated task data, Indicates the number of CPU cycles required for the task to perform the calculation. Indicates the maximum computational time delay that the task can accept;

[0018] Task offloading implements partial ratio offloading, splitting the task into two unrelated parts, and offloading the tasks divided according to the task split ratio to the edge server or cloud server for collaborative computing. The local computing task is defined as ,

[0019]

[0020]

[0021] in, Represents a local computing task The amount of data, Represents a local computing task The number of CPU cycles required,

[0022] The tasks offloaded to edge servers or cloud servers are defined as:

[0023]

[0024]

[0025] in, Indicates the task split ratio, , Indicates the amount of data unloaded by the task. Indicates the number of CPU cycles required for the offloaded computation;

[0026] Each edge server covers a fixed number of terminals. The terminal first offloads part of the task offload data to its corresponding local edge server through NOMA communication transmission, and then chooses to calculate the task on the local edge server, or use the local server as a relay node to further offload the task to a third-party edge server or cloud server for calculation. The task can only be calculated on one edge server or cloud server in the end. The task offloading correlation coefficient is expressed as Indicates that the binary variable It is used to indicate the migration of edge servers and offloading to cloud servers. The specific expression is as follows:

[0027] .

[0028] Preferably, in step S2, the optimization objectives and corresponding constraints of the resource allocation and task offloading decision are as follows:

[0029]

[0030]

[0031]

[0032]

[0033]

[0034]

[0035]

[0036]

[0037] in, The total cost of offloading cloud-edge collaborative tasks, is the total time delay, is the total energy consumption, and are the delay weight factor and energy consumption weight factor respectively, and , For the The maximum computational delay that a task generated by a terminal can accept, is the task split ratio, Binary variables for edge server migration and offloading to cloud servers, Indicates the number of CPU cycles required to offload the computation. For the The maximum amount of computing power that an edge server can handle, is the power allocation coefficient, for A collection of terminals, for A collection of edge servers;

[0038] is the maximum delay constraint, represents the task split ratio constraint, Binary association for the device, Represents a single uninstall association, represents the edge load constraint, and Represents the power allocation coefficient constraint.

[0039] Preferably, in step S2, the optimization objectives and corresponding constraints for resource allocation and task offloading decisions are constructed, and the specific steps are as follows:

[0040] S2.1, build an authentication model to offload computing tasks to a third-party edge server for collaborative computing using digital certificates and unique signatures for bilateral authentication;

[0041] S2.2, Building a Communication Model

[0042] 1) Terminal to edge server offloading

[0043] The terminals under the same edge server coverage are grouped into the same NOMA cluster. The terminals in the same NOMA cluster are multiplexed on the same sub-channel for task data offloading transmission. Assume that The terminal in Within the coverage area of ​​the edge server, the terminals in the NOMA cluster are distributed according to the channel gain. Sort ascending , the terminal sends the signal , then the signal received by the edge base station is:

[0044]

[0045] in, For the The transmission power of a NOMA cluster to the edge server is calculated by multiplying the power allocation coefficient by the total transmission power. is the Gaussian white noise at the receiver;

[0046] The receiving end uses interference cancellation technology to receive information sequentially. It first decodes the terminal with good channel quality, subtracts it from the superimposed signal after decoding, and then decodes the terminal with poor channel quality. The signal-to-noise ratio of a terminal is:

[0047]

[0048] in, is the noise power of additive white Gaussian noise,

[0049] No. The terminal sends the The communication rate of the local edge server is:

[0050]

[0051] No. The terminal sends the Communication data transmission delay of local edge servers for:

[0052]

[0053] in, Indicates the amount of data offloaded by the task;

[0054] If the task is migrated from the local edge server to a third-party edge server to offload the computation, the data transmission delay required for the migration needs to be considered. , edge servers communicate using optical fiber links:

[0055]

[0056] in, Indicates the transmission rate of the optical fiber link;

[0057] The terminal offloads the task to the third-party edge server for computing. This requires identity authentication between the terminal and the third-party edge server. Therefore, the total migration delay is:

[0058] in, Indicates the time consumed to obtain authentication;

[0059] The total communication delay when the terminal offloads the task to the edge server for calculation is:

[0060]

[0061] in, Binary variables migrated for edge servers,

[0062] Communication transmission energy consumption is expressed as:

[0063]

[0064] in, Indicates the energy consumption required to obtain certification, Energy consumption per unit time of transmission from the local edge server to the third-party edge server;

[0065] 2) Uninstall from terminal to cloud server:

[0066] The terminal offloads part of the task to the cloud server. The calculation includes two stages of transmission: from the terminal to the local edge server and from the local edge server to the cloud server. The total data transmission delay when a terminal offloads tasks to the cloud server for:

[0067]

[0068] The transmission from edge server to cloud server also uses optical fiber link transmission. The data transmission delay from edge server to cloud server is :

[0069]

[0070] in, Indicates the communication rate between the edge server and the cloud server;

[0071] Energy consumption of transferring tasks from edge servers to cloud servers :

[0072]

[0073] in, Indicates the power transmitted from the edge server to the cloud server optical fiber link;

[0074] S2.3, building a computational model

[0075] 1) Local computing

[0076] When building a computational model, it is necessary to consider the data deviation between the digital layer and the physical layer. Indicates the CPU frequency of local computing in the digital layer. The deviation between the computing resources of the digital layer and the physical layer is , then the local task calculation time is:

[0077]

[0078] The energy consumption of local computing is calculated as:

[0079]

[0080] in, Indicates the The effective switching capacitance of the chip for local computing of each terminal; Represents a local computing task The number of CPU cycles required;

[0081] 2) Edge computing

[0082] The task computing offloading time and energy consumption of the local edge server are:

[0083]

[0084]

[0085] in, represents the energy consumption of the local edge server processing one CPU cycle, Indicates the CPU frequency of the local edge server calculation in the digital layer, Deviation between digital layer and physical layer computing resources;

[0086] The time and energy consumption of migrating the third-party edge server to calculate the offloading task are:

[0087]

[0088]

[0089] in, Indicates the CPU frequency of the third-party edge server calculation in the digital layer. Calculate the resource deviation between the digital layer and the physical layer. represents the energy consumption of a CPU cycle processed by the third-party edge server;

[0090] Therefore, the total time delay of edge computing is:

[0091]

[0092] The total energy consumption of edge computing is:

[0093]

[0094] 3) Cloud computing

[0095] The cloud server calculates the offloading task time as:

[0096]

[0097] in, Indicates the CPU frequency of the cloud server calculation in the digital layer. The deviation value of the computing resources of the digital layer and the physical layer is ;

[0098] The computing energy consumption of the cloud server is:

[0099]

[0100] in, It represents the energy consumption of a cloud server processing one CPU cycle;

[0101] S2.4, considering that computing tasks and data transmission and communication are performed in parallel in the cloud-edge-device collaborative method, the total time delay is:

[0102]

[0103] in, is the total time delay of the terminal layer, is the total time delay of the edge layer, is the total time delay of the cloud layer;

[0104]

[0105]

[0106]

[0107] The total energy consumption is:

[0108]

[0109] in, is the energy consumption for data transmission, is the energy consumption of task execution,

[0110]

[0111]

[0112] Preferably, step S2.1 constructs an authentication model, specifically as follows:

[0113] Step S2.1: The terminal and the third-party edge server send a certificate request to the trusted authority;

[0114] Step S2.2: The trusted authority generates and issues a certificate, which includes the expiration date, user ID, and user public key.

[0115] Step S2.3: After obtaining the certificate, the terminal and the third-party edge server exchange certificates;

[0116] Step S2.4: The terminal signs the authentication information and random number and sends it to the third-party edge server;

[0117] Step S2.5: The third-party edge server receives and verifies the message and responds.

[0118] Step S2.6: After receiving the reply, the terminal verifies and responds, completing the authentication.

[0119] Preferably, in step S3, the optimization objective and corresponding constraints are converted into a multi-agent MDP, specifically:

[0120] In step S3.1, the digital twin of each terminal acts as a separate intelligent entity, and ultimately the entire entity collaborates to achieve the minimum system cost. The MDP of an agent is represented by a tuple express, Describe the state of the environment and deploy global information shared through digital twins. represents the set of action spaces of all agents, is the reward value set of the individual agent, representing the benefit of performing actions in the current state, is the probability of transitioning to the next state;

[0121] Step S3.2, state space :The state space is the observation value of all agents on the environment The observation value includes the amount of data generated by the terminal task and tasks , a set of weighted coefficients of latency and energy consumption generated based on each terminal’s sensitivity to them, the computing power of the terminal, edge server, and cloud server, the deviation between the digital twin and the actual physical entity, the distribution of transmission power, and the associated terminal status returned from the edge server and cloud server digital twins;

[0122] Step S3.2, action space :The action space is the set of all agent action spaces. The action space of each agent includes the task split ratio, power allocation coefficient, and associated unloading method:

[0123]

[0124] in, γ m (t) ∈[0,1] , λ k (t) ∈[0,1] , ; Representing an agent In time The task split ratio, Indicates the subchannels at time The power distribution coefficient, A binary variable representing edge server migration and offloading to cloud servers within time t;

[0125] Step S3.3, reward function :

[0126]

[0127] in, is the total time delay, is the total energy consumption, 、 are respectively the delay weight factor and the energy consumption weight factor, , They are delay penalty and load balancing penalty respectively.

[0128] Preferably, step S4 uses the multi-agent deep reinforcement learning MADDPG algorithm to solve the multi-agent MDP at the digital layer, implements the scheduling strategy formulation considering load balancing, and interacts with the physical layer in real time to complete the strategy issuance, realizing cloud-edge-end collaborative communication computing, as follows:

[0129] Step S4.1, Initialization: In the multi-agent deep reinforcement learning algorithm MADDPG, each terminal corresponds to an agent, and each agent has an actor network and a critic network. Initialize the actor network and critic network of each agent and the corresponding target network, and allocate an experience pool for each agent to store interaction experience;

[0130] Step S4.2, Scheduling Strategy Generation: At the beginning of each scheduling cycle, a scheduling strategy is generated based on the state space. Each agent uses its actor network to select its actions, including task split ratio, power allocation coefficient, and associated unloading method.

[0131] Step S4.3, Task Execution and Experience Storage: The digital layer sends the scheduling strategy to the physical layer, which provides real-time feedback based on the actual situation. The agent integrates and stores the experience into the experience pool.

[0132] Step S4.4, strategy optimization: The agent randomly samples multiple sets of experience data from the experience pool, updates the critic network by calculating the minimum loss function, and then updates the actor network by maximizing the critic network's evaluation of the current scheduling strategy;

[0133] Step S4.5, soft update: perform soft update on the target network at regular intervals, gradually approaching the current network parameters through the soft update coefficients to ensure the stability of the policy update process;

[0134] Step S4.6: After the strategy is formulated, it is sent to the physical layer for implementation to complete the overall scheduling work.

[0135] The present invention has the following technical effects: (1) A NOMA heterogeneous cloud-edge collaborative communication computing model based on digital twins is proposed, and edge computing and cloud computing are collaboratively applied to industrial Internet task processing, enriching the offloading mode, alleviating the computing pressure of the terminal, and combining NOMA to achieve large-scale access and high spectrum utilization communication interconnection, improving throughput and frequency utilization, using digital twin technology to perform real-time monitoring and simulation of physical entities, constructing corresponding digital twins, completing policy formulation at the digital layer and interacting with the physical layer to implement real-time dynamic policy formulation.

[0136] (2) Taking time delay and energy consumption into consideration, a problem model of resource allocation and task offloading is constructed. The QoS requirements of heterogeneous terminals and the security issues of collaborative edge computing are fully considered. Different offloading strategies are formulated for delay-sensitive and resource-saving tasks. Key security authentication is used for edge server migration. The deviation between digital twins and physical entities is considered in the construction of the problem model to improve the security and accuracy of the problem model and meet the QoS requirements of heterogeneous terminals.

[0137] (3) The improved multi-agent deep reinforcement learning algorithm MADDPG is used to complete resource allocation and task offloading strategy formulation, fully considering the load balancing problem of edge servers. Each terminal acts as a separate agent and shares global state information through the digital twin network. Each agent can independently formulate offloading strategies, and finally interact to form a global scheduling strategy applied to the physical layer, thereby reducing the system's time delay and energy consumption overall on the basis of load balancing. BRIEF DESCRIPTION OF THE DRAWINGS

[0138] Figure 1 Schematic diagram of the cloud-edge-device collaborative communication computing architecture of the present invention;

[0139] Figure 2 A flowchart of the authentication model;

[0140] Figure 3 This is the framework diagram of the MADDPG algorithm;

[0141] Figure 4 A comparison of the average training reward values ​​for edge server offloading only, cloud server offloading only, and without considering the digital twin bias;

[0142] Figure 5 This is a comparison chart of the average training reward values ​​of the present invention, DDPG, D3QN, and MADQN under the same environmental state parameters;

[0143] Figure 6 This is a comparison chart of the load balancing of edge servers under different strategies under the same environmental state parameters;

[0144] Figure 7 This is a comparison chart of the QoS requirements of each terminal under different strategy formulation under the same environmental state parameters. DETAILED DESCRIPTION

[0145] In order to make the contents of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments in conjunction with the accompanying drawings.

[0146] A NOMA-assisted cloud-edge communication computing method based on digital twins, with the following specific steps:

[0147] Step S1: Build an overall cloud-edge-end collaborative communication computing architecture and establish the physical layer for industrial scenarios, which includes three layers: cloud, edge, and end. The cloud layer is a cloud server with powerful computing capabilities, the edge layer includes edge servers with different computing capabilities and load capacities, and the terminal layer includes terminals (User Equipment, UE) in different industrial scenarios. Through real-time monitoring of the actual situation of physical entities, global information is accessed, and corresponding digital twins are built based on their digital mappings. The physical and digital layers interact with each other in real time.

[0148] Specifically: The physical layer includes terminals, edge servers and a cloud server , terminals and edge servers use sets and Indicates that the digital twin is constructed as , the terminal generates computing tasks in real time, The task parameters generated on the terminal are expressed as , Indicates the amount of generated task data, Indicates the number of CPU cycles required for the task to perform the calculation. Indicates the maximum computational delay that the task can accept.

[0149] Step S2: Taking into account time delay and energy consumption, a problem model for resource allocation and task offloading is constructed. The terminal QoS requirements and the security issues of collaborative edge computing are fully considered. Different offloading strategies are formulated for delay-sensitive and resource-saving tasks. Key security authentication is used for edge server migration. The deviation between the digital twin and the physical entity is also considered in the problem model construction.

[0150] The terminal generates computing tasks in real time, including data size, CPU computing requirements and task deadlines. Different task types have their corresponding time and delay sensitivity.

[0151] Task offloading implements partial ratio offloading, splitting the task into two unrelated parts. The tasks split according to the task split ratio are offloaded to the edge server or cloud server for collaborative computing. The local computing part of the task is defined as ,in, Represents a local computing task The amount of data, Represents a local computing task The number of CPU cycles required to offload tasks to edge servers or cloud servers is defined as:

[0152]

[0153]

[0154] The uninstall part is defined as:

[0155]

[0156]

[0157] in, Indicates the task split ratio, , Indicates the amount of data unloaded by the task. Indicates the number of CPU cycles required for the offloaded computation;

[0158] Each edge server covers a fixed number of terminals. The terminal first offloads part of the task data to its corresponding local edge server through NOMA communication transmission, and then chooses to calculate the task on the local edge server, or use the local server as a relay node to further offload the task to a third-party edge server or cloud server for calculation. The device association of the final task offloading adopts binary association, that is, the task can only be calculated on one edge server or cloud server in the end. The task offloading association coefficient is expressed as Indicates that the binary variable It is used to indicate the migration of edge servers and offloading to cloud servers. The specific expression is as follows:

[0159]

[0160] A problem model for resource allocation and task offloading was constructed, including authentication model, communication model, and computation model. Different offloading methods correspond to different time delays and energy consumption, as follows:

[0161] (1) Authentication model

[0162] If you want to offload computing tasks to a third-party edge server for collaborative computing, bilateral identity authentication is required, using digital certificates and unique signatures for authentication.

[0163] Figure 2 The steps of the authentication process are given. First, the terminal and the third-party edge server send a certificate request to the trusted authority (TA). TA is mainly responsible for distributing certificates, including expiration time, user ID and user public key. The unique certificate is signed and encrypted by TA using its private key. The certificate format is ,in and TA's private key and user's The public key of After obtaining the certificate, the terminal and the third-party edge server send certificates to each other, and then the terminal uses its own private key to Message and random number Sign / encrypt. After receiving the message, the third-party edge server uses the terminal public key to decrypt the message. After decryption, the third-party edge server encrypts the message with its own private key and sends it. and random numbers As a response to the terminal, the terminal uses the public key of the third-party edge server to decrypt the response, and finally the terminal uses its own private key to decrypt the After signing / decrypting and sending, the third-party edge server decrypts the message with the terminal public key to complete the mutual authentication between the third-party edge server and the terminal, thereby safely realizing the collaborative computing of the third-party edge server.

[0164] (2) Communication model

[0165] The terminals under the coverage of the same edge server are grouped into the same NOMA cluster. The terminals in the same cluster are multiplexed on the same sub-channel for task data offloading transmission. Assume that The terminal in Within the coverage area of ​​edge servers, the NOMA cluster is represented as , the terminals in the cluster are arranged according to the channel gain Sort ascending , the terminal signal is , then the signal received by the edge base station is

[0166]

[0167] in, For the The transmission power of a NOMA cluster to the edge server is calculated by multiplying the power allocation coefficient by the total transmission power. is the Gaussian white noise at the receiver,

[0168] The receiving end uses interference cancellation technology to receive information sequentially. It first decodes the terminal with good channel quality, subtracts it from the superimposed signal after decoding, and then decodes the user with poor channel quality. Therefore, the interference comes from the user with smaller channel gain, then the first The signal-to-noise ratio of a terminal is:

[0169]

[0170] in, is the noise power of additive white Gaussian noise,

[0171] Rule No. The terminal sends the The communication rate of a local edge server is calculated according to the Shannon formula:

[0172]

[0173] Rule No. The terminal sends the Communication data transmission delay of local edge servers Calculated as:

[0174]

[0175] in, Indicates the amount of data offloaded by the task;

[0176] If the task is migrated from the local edge server to a third-party edge server to offload the computation, the data transmission delay required for the migration needs to be considered. , edge servers communicate using optical fiber links:

[0177] in, Indicates the transmission rate of the optical fiber link;

[0178] The terminal offloads the task to the third-party edge server for calculation, and needs to obtain re-authentication between the terminal and the third-party edge server. Therefore, the total migration delay also needs to calculate the time consumed by obtaining authentication. , then the total communication delay for the terminal to offload the task to the edge server for calculation is:

[0179]

[0180]

[0181] Communication transmission energy consumption is expressed as:

[0182]

[0183] in, Indicates the energy consumption required to obtain certification, Energy consumption per unit time of transmission from the local edge server to the third-party edge server;

[0184] 2) Unloading from UE to cloud server:

[0185] The UE offloads part of the task to the cloud server. The calculation includes two transmission stages: UE to local edge server and local edge server to cloud server. The edge server to cloud server transmission also uses optical fiber links. The data transmission delay from edge server to cloud server is:

[0186]

[0187] in, Indicates the communication rate between the edge server and the cloud server;

[0188] No. The total data transmission delay of a UE offloading tasks to the cloud server is:

[0189]

[0190] Energy consumption of transferring tasks from edge servers to cloud servers Calculated as:

[0191]

[0192] in, Indicates the power transmitted from the edge server to the cloud server optical fiber link;

[0193] (3) Computational model

[0194] 1) Local computing:

[0195] When building a computational model, it is necessary to consider the data deviation between the digital twin and the physical entity. Indicates the CPU frequency of local computing in the twin network. The deviation value between the twin and physical entity computing resources is , then the local task calculation time is:

[0196]

[0197] The energy consumption of local computing is calculated as:

[0198]

[0199] in, Indicates the The effective switching capacitance of the chip for local computing of each terminal;

[0200] 2) Edge computing:

[0201] Similarly, the local edge server calculates the offload task time and task computing energy consumption and , migrate the third-party edge server to calculate the offloading task time and energy consumption and Calculated through the corresponding edge server parameter data,

[0202] The task computing offloading time and energy consumption of the local edge server are:

[0203]

[0204]

[0205] in, represents the energy consumption of the local edge server processing one CPU cycle, Indicates the CPU frequency of local edge server calculation in the digital layer, Deviation between digital layer and physical layer computing resources;

[0206] The time and energy consumption of migrating the third-party edge server to calculate the offloading task are:

[0207]

[0208]

[0209] in, Indicates the CPU frequency of the third-party edge server calculation in the digital layer. Calculate the resource deviation between the digital layer and the physical layer. represents the energy consumption of a CPU cycle processed by the third-party edge server;

[0210] Therefore, the total time delay of edge computing is:

[0211]

[0212] The total energy consumption of edge computing is:

[0213]

[0214] 3) Cloud computing:

[0215] The cloud server calculates the offloading task time as:

[0216]

[0217] in, Indicates the CPU frequency of the cloud server calculation in the digital layer. The deviation value of the computing resources of the digital layer and the physical layer is ;

[0218] The computing energy consumption of the cloud server is:

[0219]

[0220] in, Indicates the energy consumption of a cloud server processing one CPU cycle.

[0221] In summary, the total time delay of the terminal layer , the total time delay of the edge layer , the total time delay of the cloud layer ,as follows:

[0222]

[0223]

[0224]

[0225] Considering that computing tasks and data transmission and communication are performed in parallel in the cloud-edge-end collaboration method, the total time delay is , as well as The maximum time delay of the three is:

[0226]

[0227] The total energy consumption is:

[0228]

[0229] in, is the energy consumption for data transmission, is the energy consumption of task execution,

[0230]

[0231]

[0232] (4) Problem Model

[0233] The weighted sum of total time delay and total energy consumption is defined as the total cost of overall cloud-edge collaborative task offloading:

[0234]

[0235] In the formula and They are respectively the delay weight factor and the energy consumption weight factor. Taking into account the different QoS requirements of terminal heterogeneity for delay and energy consumption, their values ​​are determined by the terminal's sensitivity to delay and energy consumption, that is, its maximum tolerance for delay and energy consumption. and . Then the problem model is:

[0236]

[0237]

[0238]

[0239]

[0240]

[0241]

[0242]

[0243]

[0244] For the The maximum amount of computing that an edge server can load; the overall problem model is the weighted sum of delay and energy consumption, constraint (1) is the maximum delay constraint, constraint (2) represents the task splitting ratio constraint, constraint (3) is the device binary association, constraint (4) represents the single offloading association, constraint (5) represents the edge load constraint, constraints (6) and (7) represent the power allocation coefficient constraint, and the task splitting ratio, offloading method, and the allocation of NOMA cluster signal transmission power for communication between UE and edge server are comprehensively realized, thereby realizing the task offloading of cloud-edge collaborative computing.

[0245] In step S3, after proposing the problem model, the resource allocation and task offloading model is converted into a multi-agent MDP that considers load balancing. The MADDPG algorithm, which is enabled by digital twins, is trained at the digital layer. Each terminal learns the optimization strategy as a separate agent, and ultimately the whole system collaborates to achieve the minimum system cost. The MDP of an agent is represented by a tuple express, Describe the state of the environment and deploy global information shared through digital twins. represents the set of action spaces of all agents, is the reward value set of the individual agent, representing the benefit of performing actions in the current state, is the probability of transitioning to the next state. The following describes in detail the settings of the agent state space, action space, reward function, and penalty value.

[0246] 1) State space:

[0247] In the multi-agent deep reinforcement learning solution, the state space is the observation value of all agents on the environment. The observation values ​​include the amount of data and computation required for the terminal to generate the task. , based on the set of weighted coefficients of each terminal's sensitivity to latency and energy consumption, the computing power of the terminal, edge server, and cloud server, the deviation between the digital twin and the actual physical entity, the remaining allocable transmission power, and the associated terminal status returned from the edge server and cloud server digital twin.

[0248] 2 Action Space:

[0249] The action space is the set of all agent action spaces. The action space of each agent includes the task split ratio, power allocation coefficient, and associated offloading method:

[0250]

[0251] in, γ m (t) ∈[0,1] , λ k (t) ∈[0,1] , ; Representing an agent In time The task split ratio, express subchannels at time The power distribution coefficient, A binary variable representing edge server migration and offloading to cloud servers within time t;

[0252] 3 Reward function:

[0253] Reward Function is the weighted sum of delay and energy consumption Negative numbers and penalty values and Adding together, , They are delay penalty and load balancing penalty respectively. The calculated delay under the allocation strategy is compared with the maximum allowed delay of the task. If the delay exceeds the maximum limit, a cumulative penalty will be required. The load balancing penalty is calculated by multiplying the variance of the ratio of the offloaded computation amount to the tolerance computation amount of each edge server by the proportional coefficient.

[0254] In the MADDPG algorithm, each terminal corresponds to an agent, and each agent has an actor network. and a critic network , , For the respective weights of the two networks, the Actor network inputs a given environment space and outputs the corresponding action selection. To achieve the exploration of the action space, The strategy combines Gaussian noise exploration, the exploration rate decreases over time, the task split ratio and power allocation coefficient in the action space are continuous action space, and the subsequent unloading correlation coefficient is discrete action space, which makes use of the continuous action space. Discretization is performed. The Critic network evaluates the value of the current action based on the state and action of the intelligent agent, that is, predicts Value, used to guide the Actor's strategy learning.

[0255] Use the experience pool to store the agent's interaction experience, and randomly sample from it for training storage and sampling experience. Experience includes , i.e., action, reward, state, and next state. The experience pool automatically deletes old data when its capacity reaches its maximum to ensure that the stored experience remains updated. Each time, a batch of experience is randomly sampled from the experience pool for batch learning, and the weight values ​​are updated using policy gradients. The goal of the critic network is to minimize the error between the actual Q value and the target Q value, which is defined by the following formula:

[0256]

[0257] is the reward value, is the discount factor, controlling the importance of future rewards, is the Q-value of the target critic network for the next state and corresponding action. The goal of the actor network is to update the policy by maximizing the Q-value evaluated by the critic network. The parameters of the target actor and critic networks are synchronized with those of the main network through soft updates.

[0258] The specific algorithm flow is as follows:

[0259] Initialize actors for each agent and critics network And the corresponding target network , ; Initialize the experience pool for each agent; set the initial exploration parameters ; For each round of training: get the initial observation state value ; For each time slot for each agent: agent Observation environment to obtain observation values ; Agent Select Action ; Sampling based on probability distribution; by and Get reward value and the next state ; Update exploration probability ; Calculate load balancing penalty , update the reward value ; If there is space in the experience pool, store the experience ; If there is no space in the experience pool, store Replace the earliest experience;

[0260] The strategy optimization process is as follows: for each agent, batch sampling experience from the experience pool; minimizing the loss function to update the critic network ; Update the actor network by maximizing the estimated value ;Soft update target actor / critic network: , .

[0261] The digital twin of each terminal acts as a separate intelligent agent, sharing global information for resource allocation and task offloading decisions. The overall system cost depends not only on the current state of the system environment, but also on the collaborative actions taken by each intelligent agent. After the strategy is formulated, it is sent to the physical layer for further implementation to complete the overall scheduling work. The specific implementation steps are: for each time slot of each round of training, the intelligent agent Select action value ; Sampling is performed based on probability distribution; commands are issued to the physical layer; the digital layer obtains real-time environmental feedback from the physical layer; through real-time interaction between the digital layer and the physical layer, the resource allocation and task offloading strategy of the overall cloud-edge collaborative communication computing system is completed.

[0262] The present invention uses Python 3.11.5, Pytorch 2.1.2, and cuda 12.1 to build and run the test, considering 5 edge servers and 20 terminals. Figure 4 To compare the average training reward values ​​when only edge server offloading (Edge Only), only cloud server offloading (Cloud Only), and without considering digital twin deviation (No DT), it can be seen that under the same environmental conditions, the present invention can significantly reduce system costs. Figure 5This is a comparison of the average training reward values ​​of the present invention and other algorithms including DDPG, D3QN, and MADQN under the same environmental state parameters. The present invention outperforms the comparison schemes in terms of convergence speed and final convergence reward value, which shows that the present method performs well in resource allocation and task offloading strategy formulation. Figure 6 It represents the load balancing of edge servers under different strategies under the same environmental state parameters. The value is the variance of the ratio of the computing amount of each edge server to its maximum allowable computing amount. The smaller the value, the more balanced the load between edge servers and the better the effect. Figure 7 It represents the QoS requirements of each terminal under different strategies under the same environmental state parameters. The value is the ratio of the delay energy consumption ratio of each terminal to its delay energy consumption tolerance ratio. The larger the value, the more it conforms to the terminal's delay energy consumption sensitivity and better meets the terminal's QoS requirements. Figure 6 and Figure 7 The load balancing and QoS requirements of the scheduling strategies obtained by training various algorithms in the convergence state are compared, and the performance of the present invention is better than other solutions.

[0263] The detailed description of the embodiments of the present invention provided above is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

Claims

1. A NOMA-assisted cloud-edge communication computing method based on digital twins, characterized in that: The method comprises the following steps: Step S1: Build a cloud-edge-end collaborative communication computing architecture. S1.1, establish the physical layer in industrial scenarios, including three layers: cloud, edge and terminal. The cloud layer is a cloud server with powerful computing capabilities, the edge layer includes edge servers with different computing capabilities and load capacities, and the terminal layer includes terminals in different industrial scenarios. S1.2, through real-time monitoring of physical entities, a corresponding digital twin is constructed based on the data mapping of the physical entity, i.e., the digital layer, the physical layer, and the digital layer interact with each other in real time; Step S2: NOMA communication is introduced into the cloud-edge collaborative communication architecture, that is, NOMA communication is adopted between the terminal and the edge server. Each edge server covers a predetermined number of terminals. The terminals under the coverage of the same edge server are grouped into the same NOMA cluster. The terminals in the same cluster are multiplexed on the same sub-channel for task data transmission offloading, and the receiving end uses SIC interference cancellation technology for signal reception. According to the constructed cloud-edge collaborative communication computing architecture, considering the heterogeneity of terminals and the load balancing of edge servers, the computing tasks generated by the terminals in real time are partially offloaded, and the local edge server is selected as the relay node, which is further transmitted to the third-party edge server for collaborative computing. At the same time, considering the security issues of edge computing, the optimization objectives and corresponding constraints of resource allocation and task offloading decisions are constructed. The optimization goal of resource allocation and task offloading decision is to minimize the total cost of cloud-edge collaborative task offloading. , , for The total delay in the optimization goal of resource allocation and task offloading decision for the set of terminals is and total energy expenditure The time consumed by authentication generated by offloading to the third-party edge server is integrated and energy consumption required for certification , using a delay weight factor for terminal heterogeneity and energy consumption weight factor Dynamically adjust the delay weight factor to meet the different QoS requirements of terminal heterogeneity for delay and energy consumption. and energy consumption weight factor Determined based on the terminal's sensitivity to latency and energy consumption; the power allocation coefficient is introduced into its constraints , , for A collection of edge servers; Step S3, transform the optimization objective and the corresponding constraints into a multi-agent MDP, where Consider the maximum computational delay that the task generated by the mth terminal can accept , the action space is the set of all agent action spaces, and the action space of each agent includes the task split ratio, power allocation coefficient , associated unloading method, while considering the load balancing problem, introducing delay penalty in the reward function , load balancing penalty , delay penalty The calculated delay under the allocation strategy is compared with the maximum delay allowed by the task. If the maximum delay limit is exceeded, the penalty is accumulated, and the load balancing penalty is It is obtained by calculating the variance of the ratio of the offloaded computation amount of each edge server to the tolerance computation amount and multiplying it by the proportionality coefficient; In step S4, the multi-agent deep reinforcement learning MADDPG algorithm is used at the digital layer to solve the multi-agent MDP, implement the scheduling strategy formulation that takes load balancing into consideration, and interact with the physical layer in real time to complete the strategy issuance, realizing cloud-edge-end collaborative communication computing.

2. The digital twin-based NOMA-assisted cloud-edge communication computing method according to claim 1 is characterized in that: In step S1, the physical layer includes terminals, edge servers and a cloud server , terminals and edge servers use sets and Indicates that the digital layer is constructed as .

3. The NOMA-assisted cloud-edge-end communication computing method based on digital twins according to claim 1 is characterized in that In step S2, the heterogeneity of terminals and the load balancing of edge servers are considered, as follows: The terminal generates computing tasks in real time. The task parameters generated on the terminal are expressed as , Indicates the amount of generated task data, Indicates the number of CPU cycles required for the task to perform the calculation. Indicates the maximum computational time delay that the task can accept; Task offloading implements partial ratio offloading, splitting the task into two unrelated parts, and offloading the tasks divided according to the task split ratio to the edge server or cloud server for collaborative computing. The local computing task is defined as , ； ； in, Represents a local computing task The amount of data, Represents a local computing task The number of CPU cycles required, The tasks offloaded to edge servers or cloud servers are defined as: ； ； in, Indicates the task split ratio, , Indicates the amount of data unloaded by the task. Indicates the number of CPU cycles required for the offloaded computation; Each edge server covers a fixed number of terminals. The terminal first offloads part of the task offload data to its corresponding local edge server through NOMA communication transmission, and then chooses to calculate the task on the local edge server, or use the local server as a relay node to further offload the task to a third-party edge server or cloud server for calculation. The task can only be calculated on one edge server or cloud server in the end. The task offloading correlation coefficient is expressed as Indicates that the binary variable It is used to indicate the migration of edge servers and offloading to cloud servers. The specific expression is as follows: 。 4. The digital twin-based NOMA-assisted cloud-edge communication computing method according to claim 1 is characterized in that: In step S2, the optimization objectives and corresponding constraints of the resource allocation and task offloading decision are as follows: ； ； ； ； ； ； ； ； in, The total cost of offloading cloud-edge collaborative tasks, is the total time delay, is the total energy consumption, and are the delay weight factor and energy consumption weight factor respectively, and , The maximum computational delay that can be accepted for the task generated for the mth terminal, is the task split ratio, Binary variables for edge server migration and offloading to cloud servers, Indicates the number of CPU cycles required to offload the computation. For the The maximum amount of computing power that an edge server can handle, is the power allocation coefficient, for A collection of terminals, for A collection of edge servers; is the maximum delay constraint, represents the task split ratio constraint, Binary association for the device, Represents a single uninstall association, represents the edge load constraint, and Represents the power allocation coefficient constraint.

5. The digital twin-based NOMA-assisted cloud-edge-end communication computing method according to claim 4 is characterized in that: In step S2, the optimization objectives and corresponding constraints for resource allocation and task offloading decisions are constructed. The specific steps are as follows: S2.1, build an authentication model to offload computing tasks to a third-party edge server for collaborative computing using digital certificates and unique signatures for bilateral authentication; S2.2, Building a Communication Model 1) Terminal to edge server offloading The terminals under the coverage of the same edge server are grouped into the same NOMA cluster. The terminals of the same NOMA cluster are multiplexed on the same sub-channel for task data offloading transmission. Assume that The terminal in Within the coverage area of ​​the edge server, the terminals in the NOMA cluster are distributed according to the channel gain. Sort ascending , the terminal sends the signal , then the signal received by the edge base station is: ； in, For the The transmission power of a NOMA cluster to the edge server is calculated by multiplying the power allocation coefficient by the total transmission power. is the Gaussian white noise at the receiver; The receiving end uses interference cancellation technology to receive information sequentially. It first decodes the terminal with good channel quality, subtracts it from the superimposed signal after decoding, and then decodes the terminal with poor channel quality. The signal-to-noise ratio of a terminal is: ； in, is the noise power of additive white Gaussian noise, No. The terminal sends the The communication rate of the local edge server is: ； No. The terminal sends the Communication data transmission delay of local edge servers for: ； in, Indicates the amount of data offloaded by the task; If the task is migrated from the local edge server to a third-party edge server to offload the computation, the data transmission delay required for the migration needs to be considered. , edge servers communicate using optical fiber links: ； in, Indicates the transmission rate of the optical fiber link; The terminal offloads the task to the third-party edge server for computing. This requires identity authentication between the terminal and the third-party edge server. Therefore, the total migration delay is: ; in, Indicates the time consumed to obtain authentication; The total communication delay when the terminal offloads the task to the edge server for calculation is: ； Communication transmission energy consumption is expressed as: ； in, Indicates the energy consumption required to obtain certification, Energy consumption per unit time of transmission from the local edge server to the third-party edge server; 2) Uninstall from terminal to cloud server: The terminal offloads part of the task to the cloud server. The calculation includes two stages of transmission: from the terminal to the local edge server and from the local edge server to the cloud server. The total data transmission delay when a terminal offloads tasks to the cloud server for: ； The transmission from edge server to cloud server also uses optical fiber link transmission. The data transmission delay from edge server to cloud server is : ； in, Indicates the communication rate between the edge server and the cloud server; Energy consumption of transferring tasks from edge servers to cloud servers : ； in, Indicates the power transmitted from the edge server to the cloud server optical fiber link; S2.3, building a computational model 1) Local computing When building a computational model, it is necessary to consider the data deviation between the digital layer and the physical layer. Indicates the CPU frequency of local computing in the digital layer. The deviation between the computing resources of the digital layer and the physical layer is , then the local task calculation time is: ； The energy consumption of local computing is calculated as: ； in, Indicates the The effective switching capacitance of the chip for local computing of each terminal; Represents a local computing task The number of CPU cycles required; 2) Edge computing The task computation offloading time and energy consumption of the local edge server are: ； ； in, represents the energy consumption of the local edge server processing one CPU cycle, Indicates the CPU frequency of local edge server calculation in the digital layer, Deviation between digital layer and physical layer computing resources; The time and energy consumption of migrating the third-party edge server to calculate the offloading task are: ； ； in, Indicates the CPU frequency calculated by the third-party edge server in the digital layer. Calculate the resource deviation between the digital layer and the physical layer. represents the energy consumption of a CPU cycle processed by the third-party edge server; Therefore, the total time delay of edge computing is: ； The total energy consumption of edge computing is: ； 3) Cloud computing The cloud server calculates the offloading task time as: ； in, Indicates the CPU frequency of the cloud server calculation in the digital layer. The deviation value of the computing resources of the digital layer and the physical layer is ; The computing energy consumption of the cloud server is: ； in, It represents the energy consumption of a cloud server processing one CPU cycle; S2.4, considering that computing tasks and data transmission and communication are performed in parallel in the cloud-edge-device collaborative method, the total time delay is: ； in, is the total time delay of the terminal layer, is the total time delay of the edge layer, is the total time delay of the cloud layer; ； ； ； The total energy consumption is: ； in, is the energy consumption for data transmission, is the energy consumption of task execution, ； 。 6. The digital twin-based NOMA-assisted cloud-edge communication computing method according to claim 5 is characterized in that: Step S2.1 builds the authentication model, as follows: Step S2.1.1: The terminal and the third-party edge server send a certificate request to the trusted authority; Step S2.1.2: The trusted authority generates and issues a certificate, which includes the expiration date, user ID, and user public key. Step S2.1.3: After obtaining the certificate, the terminal and the third-party edge server exchange certificates; Step S2.1.4: The terminal signs the authentication information and random number and sends it to the third-party edge server; Step S2.1.5: The third-party edge server receives and verifies the message and responds. Step S2.1.6: After receiving the reply, the terminal verifies and responds, completing the authentication.

7. The digital twin-based NOMA-assisted cloud-edge communication computing method according to claim 1 is characterized in that: Step S3 is specifically as follows: In step S3.1, the digital twin of each terminal acts as a separate intelligent entity, and ultimately the entire entity collaborates to achieve the minimum system cost. The MDP of an agent is represented by a tuple express, Describe the state of the environment and deploy global information shared through digital twins. represents the set of action spaces of all agents, is the reward value set of the individual agent, representing the benefit of performing actions in the current state, is the probability of transitioning to the next state; Step S3.2, state space :The state space is the observation value of all agents on the environment The observation value includes the amount of data generated by the terminal task and tasks , a set of weighted coefficients of latency and energy consumption generated based on each terminal’s sensitivity to them, the computing power of the terminal, edge server, and cloud server, the deviation between the digital twin and the actual physical entity, the distribution of transmission power, and the associated terminal status returned from the edge server and cloud server digital twins; Step S3.3, action space :The action space is the set of all agent action spaces. The action space of each agent includes the task split ratio, power allocation coefficient, and associated unloading method: ； in, , , ; Representing an agent In time The task split ratio, Indicates the subchannels at time The power distribution coefficient, Indicates time Binary variants of internal edge server migration and offloading to cloud servers; Step S3.4, reward function : ； in, is the total time delay, is the total energy consumption, 、 are respectively the delay weight factor and the energy consumption weight factor, , They are delay penalty and load balancing penalty respectively.

8. The NOMA-assisted cloud-edge-end communication computing method based on digital twins according to claim 1 is characterized in that In step S4, the multi-agent deep reinforcement learning algorithm MADDPG is used at the digital layer to solve the multi-agent MDP, implement scheduling strategies that take load balancing into account, and interact with the physical layer in real time to complete strategy issuance, thus achieving cloud-edge-device collaborative communication and computing. The details are as follows: Step S4.1, Initialization: In the multi-agent deep reinforcement learning algorithm MADDPG, each terminal corresponds to an agent, and each agent has an actor network and a critic network. Initialize the actor network and critic network of each agent and the corresponding target network, and allocate an experience pool for each agent to store interaction experience; Step S4.2, Scheduling Strategy Generation: At the beginning of each scheduling cycle, a scheduling strategy is generated based on the state space. Each agent uses its actor network to select its actions, including task split ratio, power allocation coefficient, and associated unloading method. Step S4.3, Task Execution and Experience Storage: The digital layer sends the scheduling strategy to the physical layer, which provides real-time feedback based on the actual situation. The agent integrates and stores the experience into the experience pool. Step S4.4, strategy optimization: The agent randomly samples multiple sets of experience data from the experience pool, updates the critic network by calculating the minimum loss function, and then updates the actor network by maximizing the critic network's evaluation of the current scheduling strategy; Step S4.5, soft update: soft update the target network at regular intervals, gradually approximating the current network parameters through the soft update coefficients; Step S4.6: After the strategy is formulated, it is sent to the physical layer for implementation to complete the overall scheduling work.

Images