Hybrid non-orthogonal / orthogonal multiple access satellite virtualization intelligent scheduling method

By introducing hybrid NOMA/OMA virtualization intelligent scheduling method in satellite communications, combining artificial intelligence technology to optimize satellite transmission power distribution and user switching, the scarcity of satellite communication resources and low resource utilization under traditional orthogonal multiple access methods is solved. The problem of this has achieved efficient resource utilization and communication rate improvement.

CN116170052BActive Publication Date: 2024-11-26THE 54TH RESEARCH INSTITUTE OF CHINA ELECTRONICS TECHNOLOGY GROUP CORPORATION
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202211568877.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-12-08
Publication Date
2024-11-26
Estimated Expiration
2042-12-08

AI Technical Summary

Technical Problem

Scarce resources, strong dynamics, and large link delays in satellite communications lead to imperfect task scheduling. In addition, the traditional orthogonal multiple access method has low resource utilization in massive user scenarios, which cannot meet the high spectrum efficiency requirements of satellite Internet of Things. .

Method used

Using a virtualized intelligent scheduling method of hybrid non-orthogonal multiple access (NOMA) and orthogonal multiple access (OMA), combined with artificial intelligence technology, optimize satellite transmission power distribution, user handover and subcarrier allocation through dual-deep Q networks , adapt to network conditions and use deep reinforcement learning technology to solve complex optimization problems.

Benefits of technology

It improves the utilization rate of satellite communication resources, increases capacity, improves the reliability of network transmission and user fairness, is almost unaffected by information loss, adapts to different information acquisition situations, and optimizes the communication rate.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN116170052B_ABST
    Figure CN116170052B_ABST
Patent Text Reader

Abstract

The present invention discloses a hybrid non-orthogonal / orthogonal multiple access satellite virtualization intelligent scheduling method, which relates to the field of satellite communications. The method takes into account the communication between multiple satellites and multiple users with time delay, and jointly optimizes power allocation, NOMA / OMA mode switching, subcarrier selection, and user pairing by using a DDQN deep reinforcement learning algorithm to improve the sum of the total communication rate during satellite access. The present invention can adapt and adjust the optimization scheme to improve the communication rate under different virtualized network information acquisition conditions such as centralized, hybrid centralized / distributed, and distributed.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of satellite communication technology, and in particular to a hybrid non-orthogonal multiple access (NOMA) / orthogonal multiple access (OMA) satellite virtualization intelligent scheduling method in satellite communication. Background Art

[0002] Due to the scarcity of mobile satellite network resources, strong network dynamics, and long link delays, satellite communications may encounter task failures due to imperfect scheduling during task scheduling. Virtualized scheduling of mobile satellite platform communication resources refers to the dynamic adjustment of allocated network resources in a mobile satellite network environment, which can make full use of idle resources and alleviate problems such as task scheduling failures caused by satellite communication resource shortages. However, resource conflicts mapped to different tasks are easily caused during virtualized scheduling. Therefore, a mobile satellite communication virtual resource scheduling algorithm is proposed to address this problem.

[0003] In addition, with the rapid development of satellite Internet of Things, in order to meet the access requirements of large-scale connection and high spectrum efficiency of satellite communication networks, it is urgent to design a more effective multiple access mechanism to meet the performance requirements of scheduling tasks. In traditional satellite communication networks, available resources are usually divided into orthogonal resource blocks, and each satellite user accessing the network occupies one of the resource blocks for data transmission, thereby avoiding interference between users and ensuring the communication quality of users. In traditional satellite communication networks, available resources are usually divided into orthogonal resource blocks, and each satellite user accessing the network occupies one of the resource blocks for data transmission, thereby avoiding interference between users and ensuring the communication quality of users. When this orthogonal multiple access (OMA) method is adopted, the maximum number of users that can be accommodated in the satellite network is equal to the number of orthogonal resources. Under the constraints of satellite payload resources, the satellite network is limited in the scenario of massive user access. Therefore, with the rapid development of satellite Internet of Things, in order to meet the access requirements of large-scale connection and high spectrum efficiency of satellite Internet of Things, it is urgent to introduce a new user multiple access mechanism to improve its resource utilization.

[0004] Non-orthogonal Multiple Access (NOMA) technology can send multiple signals at the same frequency at the same time through power domain multiplexing and successive interference cancellation (SIC) technology. It has higher resource utilization and user fairness than OMA technology and has received great attention from academia and industry. Under the same network resource conditions, NOMA can greatly increase the capacity of wireless communication systems and improve the utilization of network resources. It has received widespread attention in 5G communication networks. In addition, the hybrid use of NOMA / OMA technology can adapt to changes in network conditions and further improve the reliability of satellite communication transmission. However, the application of the NOMA / OMA hybrid transmission mode in the virtualized scheduling of satellite communication networks lacks systematic analysis and verification, as well as further integration with existing technologies such as power allocation. Summary of the invention

[0005] Aiming at the virtualized scheduling problem, the present invention proposes a hybrid NOMA\OMA virtualized intelligent scheduling method in satellite communications. The method adopts artificial intelligence technology to analyze the environmental parameters based on outdated channel information and optimize the scheduling scheme.

[0006] The technical solution adopted by the present invention is:

[0007] A hybrid non-orthogonal / orthogonal multiple access satellite virtualization intelligent scheduling method comprises the following steps:

[0008] (1) The satellite communication state at time t is defined as s(t), which includes the channel information of the satellite-user transmission link in the communication environment, the satellite position, the ground user position, and the satellite transmission power;

[0009] (2) The artificial intelligence agent Agent selects action a(t) in a random exploration manner at the beginning. The action refers to the joint scheduling of satellite transmit power allocation, user switching, hybrid non-orthogonal / orthogonal multiple access switching and subcarrier allocation required by the satellite virtualization scheduling mechanism;

[0010] (3) The agent uses the actions in step (2) to interact with the environment and obtains a preset reward value r(t); the reward value is determined by the user's actual communication rate;

[0011] (4) Repeat steps (1)-(3) within time T and collect corresponding training data {s(t), a(t), r(t), s(t+1)}; then, use the dual-depth Q network to update the Q value, which is the evaluation value of the corresponding action of each state at each moment;

[0012] (5) Repeat steps (1)-(4) until convergence and obtain a trained agent;

[0013] (6) In the centralized, distributed, and hybrid centralized / distributed cases, iterative training is performed through steps (1)-(5) to obtain the corresponding multiple agents to perform intelligent scheduling of satellite virtualization.

[0014] Furthermore, in step (1), the channel information in the state s(t) is outdated channel information taking into account the delay, and the description of the outdated channel information is as follows:

[0015]

[0016] Among them, ρ AB is the Jakes autocorrelation model parameter, ω AB is a circularly symmetric complex Gaussian random variable, h AB is the actual channel parameter; the virtualized intelligent scheduling method obtains outdated channel information in step (1), and establishes the relationship between the outdated channel information and the actual communication rate in the training of step (4).

[0017] Furthermore, in step (4), the dual-depth Q network has two neural networks Q A and Q B , where network Q A The Q value update in is calculated as follows:

[0018] Q A (s(t), a(t)) = Q A (s(t), a(t))+δ(r (s(t),a(t)) + Q B (s(t+1),argmax a {Q A (s(t+1), a(t))})-Q A (s(t), a(t)))

[0019] Network Q B The Q value update in is calculated as follows:

[0020] Q B (s(t), a(t)) = Q B (s(t), a(t))+δ(r s(t),a(t)) + Q A (s(t+1),argmax a {Q B (s(t+1),a(t))})-Q B (s(t), a(t)))

[0021] Among them, Q A (s(t),a(t)) and Q B (s(t), a(t)) represent the network Q A and Q B The Q value of the corresponding state-action combination (s(t), a(t)); δ is the learning rate of the dual-depth Q network, which affects the learning speed of this network; r (s(t),a(t)) is the reward value corresponding to the state-action combination (s(t), a(t)); is the discount factor, which affects the weights of the newly obtained reward and the original strategy Q value in the new strategy; argmax a {·} means to select the action a that can obtain the maximum Q value in {·};

[0022] The dual-depth Q network first builds on the current network Q A Find the action corresponding to the maximum Q value, and then use the obtained action to obtain the network Q B The corresponding Q value in the next state; then update the network Q by updating the formula A Q value in B In Q A Keep it fixed during training and periodically change it by Q A Update the entire Q B network;

[0023] The training loss function of the two neural networks in the dual-depth Q network is as follows:

[0024]

[0025] Where T is the number of training data used in a single training of the neural network;

[0026] After convergence, the dual-depth Q network directly uses the action corresponding to the maximum Q value as the scheduling decision for any satellite communication state.

[0027] Furthermore, in step (6), centralized means that multiple satellites are scheduled by a unified central control node; distributed means that each satellite is controlled individually and does not share information with each other; hybrid centralized\distributed means that some satellites are uniformly scheduled by the central control node and some satellites are controlled individually.

[0028] The beneficial effects of the present invention are:

[0029] 1. The present invention is the first to use a fusion system of hybrid NOMA\OMA and power allocation technologies in satellite communications, and adopts deep reinforcement learning technology based on artificial intelligence to solve highly complex optimization problems.

[0030] 2. The present invention takes into account different information acquisition situations (centralized, mixed centralized\distributed, distributed), and introduces the influence of delayed channel information. Through the analysis of artificial intelligence technology, the present invention is almost unaffected by information loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a system model diagram of satellite communication in the present invention.

[0032] Figure 2 It is an interaction diagram between the Agent and the satellite communication environment in the present invention.

[0033] Figure 3 Comparison of the performance of the centralized DDQN algorithm with no power allocation and no subcarrier allocation under different satellite altitudes.

[0034] Figure 4 Comparison of the performance of the DDQN algorithm under different information acquisition conditions (centralized, hybrid centralized\distributed, distributed) at different satellite altitudes. DETAILED DESCRIPTION

[0035] In order to make the principles, technical process schemes and optimization effects of the present invention clearer, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0036] A hybrid non-orthogonal / orthogonal multiple access satellite virtualization intelligent scheduling method comprises the following steps:

[0037] (1) The satellite communication state at time t is defined as s(t), which includes the channel information of the satellite-user transmission link in the communication environment, the satellite position, the ground user position, and the satellite transmission power;

[0038] (2) The artificial intelligence agent Agent selects action a(t) in a random exploration manner at the beginning. The action refers to the joint scheduling of satellite transmit power allocation, user switching, hybrid non-orthogonal / orthogonal multiple access switching and subcarrier allocation required by the satellite virtualization scheduling mechanism;

[0039] (3) The agent uses the actions in step (2) to interact with the environment and obtains a preset reward value r(t); the reward value is determined by the user's actual communication rate;

[0040] (4) Repeat steps (1)-(3) within time T and collect corresponding training data {s(t), a(t), r(t), s(t+1)}; then, use the dual-depth Q network to update the Q value, which is the evaluation value of the corresponding action of each state at each moment;

[0041] (5) Repeat steps (1)-(4) until convergence and obtain a trained agent;

[0042] (6) Considering the inter-satellite collaboration problem and communication overhead in actual communication, in three cases, centralized (all satellites have a unified central control node that can collect all satellite information, such as a ground control station), distributed (all satellites have no direct or indirect communication with each other and do not exchange any information with each other), and hybrid centralized / distributed (some satellites have central control nodes, and other satellites do not exchange information with other satellites as in the distributed case), the corresponding multiple agents are obtained through iterative training of steps (1)-(5) to perform satellite virtualization intelligent scheduling.

[0043] In step (1), the channel information in the state s(t) is outdated channel information taking into account the delay. The description of the outdated channel information is as follows:

[0044]

[0045] Among them, ρ AB is the Jakes autocorrelation model parameter, ω AB is a circularly symmetric complex Gaussian random variable, h AB is the actual channel parameter; the virtualized intelligent scheduling method obtains outdated channel information in step (1), and establishes the relationship between the outdated channel information and the actual communication rate in the training of step (4).

[0046] In step (4), the dual-depth Q network has two neural networks Q A and Q B , where network Q A The Q value update in is calculated as follows:

[0047] Q A (s(t), a(t)) = Q A (s(t), a(t))+δ(r (s(t),a(t)) + Q B (s(t+1),argmax a {Q A (s(t+1),a(t))})-Q A (s(t), a(t)))

[0048] Network Q B The Q value update in is calculated as follows:

[0049] Q B (s(t), a(t)) = Q B (s(t), a(t))+δ(r (s(t),a(t)) + Q A (s(t+1), argmaxa {Q B (s(t+1), a(t))})-Q B (s(t), a(t)))

[0050] Among them, Q A (s(t),a(t)) and Q B (s(t), a(t)) represent the network Q A and Q B The Q value of the corresponding state-action combination (s(t), a(t)); δ is the learning rate of the dual-depth Q network, which affects the learning speed of this network; r (s(t),a(t)) is the reward value corresponding to the state-action combination (s(t), a(t)); is the discount factor, which affects the weights of the newly obtained reward and the original strategy Q value in the new strategy; argmax a {·} refers to selecting the action a that can obtain the maximum Q value in {·};

[0051] The dual-depth Q network first builds on the current network Q A Find the action corresponding to the maximum Q value, and then use the obtained action to obtain the network Q B The corresponding Q value in the next state; then update the network Q by updating the formula A Q value in B In Q A Keep it fixed during training and periodically change it by Q A Update the entire Q B network;

[0052] The training loss function of the two neural networks in the dual-depth Q network is as follows:

[0053]

[0054] Where T is the number of training data used in a single training of the neural network;

[0055] After convergence, the dual-depth Q network directly uses the action corresponding to the maximum Q value as the scheduling decision for any satellite communication state.

[0056] In satellite virtualization technology, this method solves the joint intelligent scheduling problem including satellite transmit power allocation, user switching, hybrid non-orthogonal\orthogonal multiple access switching and subcarrier allocation by training multiple centralized, distributed and hybrid centralized / distributed agents, and takes into account the problem of outdated channel information caused by delay.

[0057] The following is a specific example:

[0058] like Figure 1As shown in the figure, it is considered that multiple low Earth orbit (LEO) satellites simultaneously serve multiple users in a ground area. Each LEO satellite has the functions of hybrid NOMA / OMA adaptive transmission mode, subcarrier allocation, and power allocation. At each transmission moment, the ground user will switch to access the appropriate satellite. At the same time, given that the energy of the satellite is limited, the upper threshold of the energy consumption of the satellite when serving an area is considered in the power allocation. The goal is to maximize the sum of the communication capacity of multiple satellites serving all users in the same ground area by optimizing the hybrid NOMA / OMA adaptive transmission mode, subcarrier allocation, power allocation, and user-satellite access switching, and develop and optimize the proposed virtualized scheduling scheme that maximizes the transmission performance of satellite-to-ground communication. At the same time, considering the problem of inter-satellite collaboration in actual communications, this method considers optimization schemes under different communication environments such as centralized, distributed, and hybrid centralized / distributed. In addition, considering the delay problem in actual communications, this method introduces outdated channel information.

[0059] Since the above scheduling problem is a very complex non-convex mixed integer programming problem, this method uses deep reinforcement learning technology to maximize the sum of the communication capacity of users in the service area, and at the same time learns the impact and connection of changes in the communication environment, so that the proposed optimization scheme can adapt to three different situations: centralized, hybrid centralized\distributed and distributed, and develops a hybrid NOMA\OMA communication virtualization scheduling technology for multi-LEO collaboration.

[0060] In order to implement the above method, a training method based on artificial intelligence technology is adopted, and the specific steps are as follows:

[0061] (1) The satellite communication state at a certain time t is defined as s(t). The state includes the channel information of the satellite-user transmission link in the communication environment, the satellite position, the ground user position, the satellite transmission power and other environmental parameters necessary for satellite communication;

[0062] (2) The artificial intelligence agent Agent selects action a(t) in a random exploration manner at the beginning. The action refers to the joint scheduling of satellite transmit power allocation, user switching, hybrid NOMA\OMA switching and subcarrier allocation required by the satellite virtualization scheduling mechanism;

[0063] (3) The agent uses the action in step (2) to interact with the environment and obtains a preset reward value r(t). The reward value is determined by the user's actual communication rate. The interaction between the agent and the communication environment is as follows: Figure 2 As shown;

[0064] (4) Repeat steps (1)-(3) within time T and collect the corresponding training data {s(t), a(t), r(t), s(t+1)}. Then, use the (Deep Reinforcement Learning with Double Q-learning) DDQN algorithm to update the Q value. The Q value is the evaluation value of the corresponding action at each moment;

[0065] (5) Repeat steps (1)-(4) until the algorithm converges.

[0066] In step (1), the channel information in the state is outdated channel information taking into account the delay. The description of the outdated channel information is as follows:

[0067]

[0068] Among them, ρ AB is the Jakes autocorrelation model parameter, ω AB is a circularly symmetric complex Gaussian random variable, h AB is the actual channel parameter. The scheduling method used obtains outdated channel information in step (1), and establishes the relationship between the outdated channel information and the actual communication rate in the training of step (4).

[0069] In addition, in step (4), the DDQN algorithm has two neural networks Q A and Q B , where Q A The Q value update in the network is calculated as follows:

[0070] Q A (s(t), a(t)) = Q A (s(t), a(t))+δ(r (s(t),a(t)) + Q B (s(t+1), argmax a {Q A (s(t+1), a(t))})-Q A (s(t), a(t)))

[0071] Among them, δ is the learning rate of DDQN, which affects the learning speed of this network. is the discount factor, which affects the weights of the newly obtained reward and the original strategy Q value in the new strategy. A First find the action corresponding to the maximum Q value in the network, and then use the obtained action to obtain the Q value in the network. B Then update Q by updating the formula A The Q value in the network.

[0072] Network Q B The update formula is as follows:

[0073] Q B (s(t), a(t)) = Q B (s(t), a(t))+δ(r (s(t),a(t)) + Q A (s(t+1), argmax a {Q B (s(t+1), a(t))})-Q B (s(t), a(t)))

[0074] Considering the need to avoid the network being too aggressive in its exploration strategy, Q B In Q A Keep it fixed during training and periodically change it by Q A Update the entire Q B Network. The training loss function for the two neural networks in DDQN is as follows:

[0075]

[0076] DDQN repeats steps (1)-(4) until the algorithm converges. After convergence, the DDQN algorithm will directly use the action corresponding to the maximum Q value as the scheduling decision for any satellite communication state.

[0077] For the satellite communication network in this method, the environment refers to the entire satellite communication network environment, and the state refers to the communication necessary data obtained from the environment, including the channel information of the satellite-user transmission link, satellite position, ground user position, and satellite transmission power. The action refers to the joint scheduling of satellite transmission power allocation, user switching, hybrid NOMA\OMA switching, and subcarrier allocation required by the satellite virtualization scheduling mechanism. The reward depends on the actual communication rate obtained by the user after accessing the satellite communication network. Therefore, the greater the total rate obtained by the user, the greater the reward.

[0078] In this method, the DDQN algorithm is used to implement the development of satellite virtualized network scheduling technology. For the satellite communication environment, the state at a certain time t is defined as s(t). The agent selects an action a(t) in a random exploration manner at the beginning. After executing the action a(t), the environment changes the state s(t) to the state s(t+1) at the next moment, and obtains the corresponding reward r(t) based on the feedback of the user rate. Two networks Q are designed in the agent. A and Q B, by using the reward r(t) to update the evaluation value (Q value) of the corresponding s(t)-a(t) state-action combination, and use this to judge the performance of the combination.

[0079] Therefore, in the DDQN algorithm, a large number of data sets such as {s(t), a(t), r(t), s(t+1)} can be generated through random exploration, and then the two networks Q can be iteratively updated by updating the Q value. A and Q B . Considering that the number of combinations of states and actions in satellite communication networks is very large, it is almost impossible to build a corresponding table to record Q values. Therefore, a deep neural network is used to replace the table to record Q values. The deep neural network can input states and output actions. The mapping relationship between states and actions in the neural network is the Q value. The specific algorithm flow is as follows:

[0080] 1. Initialize the deep neural network Q A and Q B All parameters of (i.e., initialize Q values).

[0081] 2. Start the loop:

[0082] 2.1 For the state s(t) at time t, use A network to predict / randomly explore to get action a(t).

[0083] 2.2 By executing action a(t), we get the feedback reward r(t) from the environment and the state s(t+1) at the next moment.

[0084] 2.3 Generate training data {s(t), a(t), r(t), s(t+1)}

[0085] 2.4 After generating T training data, use the existing data set through Q A The update formula of Q A

[0086] 2.5 Q per training N times A After that, Q A The Q value is copied to Q B .

[0087] 3. Terminate the loop after reaching the preset number of iterations or convergence target

[0088] The test of this method is as follows Figure 3-Figure 4 The test results show the corresponding performance of the proposed algorithm with no power allocation, no subcarrier allocation and no power and no subcarrier allocation at different satellite altitudes, which is convenient for comparing the results. In addition, this method provides the freedom to choose centralized, hybrid centralized / distributed and distributed, and gives the corresponding rate performance comparison.

[0089] From the results of the centralized case, Figure 3 The proposed DDQN-based algorithm in has the best performance and significantly improves the total communication rate compared to the case of no power allocation (average power allocation). The effect of subcarrier selection is not significant because for long-distance satellite communications, the impact of small-scale fading is much lower than that of large-scale fading. Therefore, compared with power allocation, the gain of subcarrier selection is not obvious. Figure 4 From the results in , the centralized performance is the best, followed by the hybrid centralized distributed performance, and the distributed performance is the lowest. This is in line with the logic of scenario design, that is, global optimization must be stronger than the sum of local optimizations.

[0090] It can be seen that the present invention can guarantee the communication rate for a communication system between multiple low-orbit satellites and multiple users with time delay channel information consideration. At the same time, low-complexity neural networks are beneficial to saving computing resources during real-time computing.

[0091] In summary, the present invention takes into account the communication between multiple satellites and multiple users with time delay, and improves the sum of the total communication rate during satellite access by adopting the DDQN deep reinforcement learning algorithm to jointly optimize power allocation, NOMA / OMA mode switching, subcarrier selection, and user pairing. The present invention can adapt and adjust the optimization scheme to improve the communication rate under different virtualized network information acquisition conditions such as centralized, hybrid centralized / distributed, and distributed.

[0092] The embodiment described above is only a specific embodiment of the present invention, not all embodiments. Based on the embodiment of the present invention, other embodiments obtained by those skilled in the art without creative work are all within the scope of protection of the present invention.

Claims

1. A hybrid non-orthogonal / orthogonal multiple access satellite virtualization intelligent scheduling method, characterized in that: The following steps are involved: (1) The satellite communication state at time t is defined as s(t), which includes the channel information of the satellite-user transmission link in the communication environment, the satellite position, the ground user position, and the satellite transmission power; (2) The artificial intelligence agent Agent selects action a(t) in a random exploration manner at the beginning. The action refers to the joint scheduling of satellite transmit power allocation, user switching, hybrid non-orthogonal / orthogonal multiple access switching and subcarrier allocation required by the satellite virtualization scheduling mechanism; (3) The agent uses the actions in step (2) to interact with the environment and obtains a preset reward value r(t); the reward value is determined by the user's actual communication rate; (4) Repeat steps (1)-(3) within time T, collect corresponding training data {s(t), a(t), r(t), s(t+1)}; then, use the dual-depth Q network to update the Q value, which is the evaluation value of the corresponding action of each state at each moment; the dual-depth Q network has two neural networks Q A and Q B , where network Q A The Q value update in is calculated as follows: Q A (s(t),a(t))=Q A (s(t),a(t))+δ(r (s(t),a(t)) + Q B (s(t+1),argmax a {Q A (s(t+1),a(t))})-Q A (s(t),a(t))) Network Q B The Q value update in is calculated as follows: Q B (s(t),a(t))=Q B (s(t),a(t))+δ(r (s(t),a(t)) + Q A (s(t+1),argmax a {Q B (s(t+1),a(t))})-Q B (s(t),a(t))) Among them, Q A (s(t),a(t)) and Q B (s(t), a(t)) represent the network Q A and Q B The Q value of the corresponding state-action combination (s(t), a(t)); δ is the learning rate of the dual-depth Q network, which affects the learning speed of this network; r (s(t),a(t)) is the reward value corresponding to the state-action combination (s(t), a(t)); is the discount factor, which affects the weights of the newly obtained reward and the original strategy Q value in the new strategy; argmax a {·} refers to selecting the action a(t) that can obtain the maximum Q value in {·}; The dual-depth Q network first builds on the current network Q A Find the action corresponding to the maximum Q value, and then use the obtained action to obtain the network Q B The corresponding Q value in the next state; then update the network Q by updating the formula A Q value in B In Q A Keep it fixed during training and periodically change it by Q A Update the entire Q B network; The training loss function of the two neural networks in the dual-depth Q network is as follows: Where T is the number of training data used in a single training of the neural network; After convergence, the dual-depth Q network directly uses the action corresponding to the maximum Q value as the scheduling decision for any satellite communication state; (5) Repeat steps (1)-(4) until convergence and obtain a trained agent; (6) In the centralized, distributed, and hybrid centralized / distributed cases, iterative training is performed through steps (1)-(5) to obtain the corresponding multiple agents to perform intelligent scheduling of satellite virtualization.

2. The hybrid non-orthogonal / orthogonal multiple access satellite virtualization intelligent scheduling method according to claim 1 is characterized in that: In step (1), the channel information in the state s(t) is outdated channel information taking into account the delay. The description of the outdated channel information is as follows: Among them, ρ AB is the Jakes autocorrelation model parameter, ω AB is a circularly symmetric complex Gaussian random variable, h AB is the actual channel parameter; the virtualized intelligent scheduling method obtains outdated channel information in step (1), and establishes the relationship between the outdated channel information and the actual communication rate in the training of step (4).

3. The hybrid non-orthogonal / orthogonal multiple access satellite virtualization intelligent scheduling method according to claim 1 is characterized in that: In step (6), centralized means that multiple satellites are scheduled by a unified central control node; distributed means that each satellite is controlled individually and does not share information with each other; hybrid centralized / distributed means that some satellites are uniformly scheduled by the central control node and some satellites are controlled individually.