A method for allocating satellite-ground collaborative network slice resources based on reinforcement learning

Abstract

The application relates to a kind of star-ground collaborative network slice resource allocation methods based on reinforcement learning, comprising: by ground liaison station, low-orbit satellite network is connected with local network data center, and according to service request, the dedicated slice of matching service type is created;The process that ground liaison station node carries out resource allocation to each service request is regarded as Markov decision process, and the process model of single ground liaison station multi-slice resource allocation is established;For the resource request of each dedicated slice, the system total utility function considering the overall spectrum resource utilization rate of ground liaison station and user demand satisfaction is constructed;The calculation result of system total utility function is used as the reward value of single ground liaison station multi-slice resource allocation process model;A deep reinforcement learning model is constructed, and the deep reinforcement learning model is trained with the goal of maximizing reward value;The resource allocation strategy under the current state information is predicted by the trained deep reinforcement learning model to carry out network slice resource allocation.

Description

Technical Field

[0001] This invention relates to a method for allocating sliced ​​resources in a satellite-ground cooperative network based on reinforcement learning, belonging to the field of wireless communication technology. Background Technology

[0002] Network slicing technology, as one of the key technologies of 5G, enables flexible deployment and management of network resources, building multiple communication sub-networks with different needs on the basis of an integrated physical network. The control and orchestration of network slicing mostly adopts intelligent algorithms, focusing on the allocation methods of terrestrial network resources and equipment. Terrestrial communication networks have abundant resources and high throughput, but their service coverage is limited and they are susceptible to natural disasters. In contrast, the rapidly developing low-Earth orbit satellite constellations can cover wide areas through satellite networks. Satellite communication networks have broad coverage, but transmission latency increases with satellite altitude. Combining the advantages of various communication methods, space-ground integrated networks can adapt to the new demands and rapid growth of the communication field. Industry innovation is striving to break through traditional limitations, laying a solid foundation for the next generation of communication networks.

[0003] Most current network slicing technologies are based on terrestrial communication networks. In areas where the terrestrial basic communication infrastructure is underdeveloped, it is still impossible to provide users in the area with ultra-high data transmission rates, low latency, and high reliability communication services.

[0004] To address the current challenges, invention patent CN 116582855 B discloses a 5G-TSN converged network slice management method and system based on deep reinforcement learning. The specific solution is as follows: "With the goal of optimizing the time-frequency resource utilization efficiency of the radio-side base station, a preset deep reinforcement learning algorithm is used to perform admission control and slice resource allocation for various types of slice requests based on the number of available time-frequency resources on the base station side. Using the technical solution of this invention, resource management of multiple types of slices can be performed while ensuring service quality requirements, thereby maximizing the efficiency of base station radio resources." This existing technology can maximize the efficiency of base station radio resources while ensuring service quality requirements. However, the existing technology has the problem of not being able to effectively utilize satellite-ground converged resources to achieve a better balance between resource allocation that is neither excessive nor insufficient; at the same time, the hardware deployment cost is too high. Summary of the Invention

[0005] To address the problems existing in the prior art, this invention proposes a satellite-ground collaborative network slicing resource allocation method based on reinforcement learning. It focuses on the core of the ground system access network for satellite communication. The access network provides carrying capacity for the accessed services and realizes transparent transmission of services. The ground gateway station connects the local network data center with the low-Earth orbit satellite network to realize information exchange between the wireless access network and the core network. The user terminal interacts with the communication satellite through the ground gateway station. Typically, the data traffic transmitted by the user to the ground gateway station is relatively small, while the backhaul downlink data volume is relatively large.

[0006] The technical solution of the present invention is as follows:

[0007] This invention provides a method for allocating sliced ​​resources in a satellite-ground cooperative network based on reinforcement learning, comprising the following steps:

[0008] The low-Earth orbit satellite network is connected to the local network data center through a ground gateway station, and the user terminal interacts with the low-Earth orbit satellite through the ground gateway station;

[0009] Get the business requests submitted by the user, obtain the resource requirements of different types of business based on the business requests, and create exclusive slices that match the business type;

[0010] The process of resource allocation for each service request by the ground gateway node is regarded as a Markov decision process, and a multi-slice resource allocation process model of a single ground gateway is established.

[0011] For resource requests from each dedicated slice, a total system utility function is constructed that considers the overall spectrum resource utilization rate of ground gateway stations and user demand satisfaction; the calculation result of the total system utility function is used as the reward value of the multi-slice resource allocation process model of a single ground gateway station.

[0012] A deep reinforcement learning model is constructed, taking the state information of the single ground gateway station multi-slice resource allocation process model as input, the predicted resource allocation strategy as output, and training the deep reinforcement learning model with the goal of maximizing the reward value.

[0013] The resource allocation strategy is predicted based on the current state information by a trained deep reinforcement learning model, and then network slice resource allocation is performed based on the resource allocation strategy.

[0014] Preferably, the process of establishing a multi-slice resource allocation model for a single ground gateway station specifically includes:

[0015] Using Markov processes<S,A,P,R> This represents the admission decision process for slice requests; where S is the state information, A is the action space, P is the state transition probability, and R is the reward function.

[0016] The state information includes the priority, number of data packet requests, and number of users on a slice within a single time period; the action space is the set of actions A = {a1, a2, a3} performed by the agent on the environment within the current period, where a n The bandwidth allocated to a single slice within this period satisfies the constraint a1 + a2 + a3 = W, where W is the available spectrum resource of the ground gateway station; the state transition probability is the probability of transitioning to the next time state after taking an action; the specific construction process of the reward function is as follows:

[0017] Within a given time period, the available spectrum resources of a ground gateway station are W, and the slice set is S. EP n network slices s n ∈S EP The ground gateway station calculates the data packet volume of each slice based on the user request data packet size. n And priority to allocate bandwidth w for each slice n The data transmission rate v on the slice n As shown below:

[0018] v n =w n log2(1+SNR n )

[0019] Among them, SNR n The signal-to-noise ratio between the user and the ground gateway station;

[0020] The overall spectrum resource utilization rate (RU) of the ground gateway station is:

[0021]

[0022] Set any slice s n The maximum delay for the final transmission of a single data packet d is l. maxn The minimum data transfer rate required is r minn ; within any time period, any slice s n The total latency for delivering a single data packet to the user is t. n The data packet transmission is considered successful if the following conditions are met:

[0023] t n ≤l maxn

[0024] v n ≥r minn

[0025] Any slice s n The number of data packets successfully transmitted in China is The specific formula is as follows:

[0026]

[0027] In this system, a successful transmission of a single data packet is recorded as d=1, and a failed transmission is recorded as d=0; the ground gateway station receives data from any slice s. n The total set of data packets transmitted in the middle is

[0028] Any slice s n Sat user satisfaction n The ratio of the number of data packets successfully transmitted to the user to the total number of data packets transmitted:

[0029]

[0030] The satisfaction levels of each slice are weighted and summed, with each slice having a weight of α. n The overall user satisfaction with the system is as follows:

[0031]

[0032] The total utility function of the system is as follows:

[0033] E = βRU + Sat

[0034] Wherein, β is a weighting coefficient for spectrum resource utilization, and is set according to priority;

[0035] The total utility function of the system is used as the reward function R.

[0036] Preferably, the process of training a deep reinforcement learning model with the state information of a single ground gateway station multi-slice resource allocation process model as input, the predicted resource allocation strategy as output, and the objective of maximizing the reward value specifically includes:

[0037] The deep reinforcement learning model employs the Advantage Actor-Critic algorithm, which includes an actor network and a critic network; wherein the actor network and the critic network respectively include a policy network and a value function network.

[0038] Initialize the actor and critic networks and their parameters. Within each slice configuration time period T, set the time slots from i to T and initialize an empty buffer of length T. In the i-th time slot, obtain the network slice request status s. i The request status s i The state information includes the request priority, number of data packets, and number of users; the state information is placed in a buffer, and the information in the buffer will constitute the state space within one period T;

[0039] The current state space is used as input; the policy network calculates the probability distribution of actions, selects actions to execute based on the probability distribution, and determines whether to admit slice requests within the current configured time period. After admission, ground gateway resources that maximize efficiency are allocated to slices of different priorities, with low-latency slices having the highest priority, followed by high-speed slices, and large-connection network slices having the lowest priority. Remaining available resources within different time periods are allocated to them; the value function network calculates the state value based on the current state, and the difference between the value obtained by the executed action and the expected action value, plus the reward function value from environmental feedback, to obtain the advantage value of the selected action; the advantage function updates the network state based on the advantage value.

[0040] Preferably, the network state update specifically involves:

[0041] When an action is executed once, the first item is deleted from the buffer, and the new observation state is added to the tail to obtain the state space for the next cycle. The loss values ​​of the policy network and the value function network in the actor network and critic network are calculated respectively, and the network parameters are updated according to the gradient descent method.

[0042] The loss function of the policy network in the actor network is shown below:

[0043] Loss a =-δlogπ(a|s)=-(r+γ) a V(s n+1 )-V(s))logπ(a|s)

[0044] Where 'a' represents the action taken by the actor network in the current state; 'r' represents the reward function value, V(s) n+1 γ is the estimated expected return for the next state, V(s) is the estimated expected return for the current state, and γ is the expected return for the next state. a is the discount factor, reflecting the impact of future state value on the effect of current behavior, with a value between 0 and 1; logπ(a|s) is the log probability of taking action a in state s;

[0045] The update method for the policy network parameter θ in the actor network is as follows:

[0046]

[0047] in, θ represents the updated policy network parameters; α represents the current policy network parameters; and α represents the learning rate, which determines the step size for parameter updates. Let θ be the gradient of the network parameters θ with respect to the current policy; A(s,a) is the advantage function, representing the value advantage of taking action a relative to the average level in state s;

[0048] The value function network loss represents the difference between the predicted Q-value and the actual Q-value output by the critic network; this difference is measured using the mean squared error loss, and the specific formula is shown below:

[0049]

[0050] Where, q π (s n ,a n ;w) represents the Q-value of the critic network, parameterized with w, for predicting the state and action under the current policy network; r t γ represents the reward given by the environment for action a within a time allocation period; c The discount factor for predicting Q-values ​​represents the impact of future predictions on the present. The Q-value for predicting the next state and action in the updated policy network; This indicates the updated network parameters;

[0051] The gradient descent parameter update method of the critic network:

[0052]

[0053] Where k is the learning rate, which determines the step size for updating this parameter; This represents the gradient of the loss function with respect to the parameters;

[0054] The state value function is Among them, s n For network slicing; a n is the bandwidth allocated to a single slice within a single time period; w is the trainable parameter of the state value function;

[0055] By continuously optimizing the target value function, we can improve the satisfaction of target users and the utilization rate of spectrum resources, thereby maximizing the reward value.

[0056] On the other hand, the present invention also provides a satellite-ground cooperative network slice resource allocation system based on reinforcement learning, comprising:

[0057] The satellite-ground coordination module connects the low-Earth orbit satellite network with the local network data center through a ground gateway station, and the user terminal interacts with the low-Earth orbit satellite through the ground gateway station;

[0058] The slicing module obtains business requests from the user, acquires the resource requirements of different types of business based on the business requests, and creates exclusive slices that match the business type.

[0059] A resource allocation model module is constructed, which takes the process of allocating resources to each service request by the ground gateway node as a Markov decision process and establishes a resource allocation process model for a single ground gateway station with multiple slices.

[0060] The system total utility function module constructs a system total utility function that considers the overall spectrum resource utilization rate of ground gateway stations and user demand satisfaction for the resource requests of each dedicated slice; the calculation result of the system total utility function is used as the reward value of the single ground gateway station multi-slice resource allocation process model;

[0061] The model module was optimized, and a deep reinforcement learning model was constructed. The state information of the single ground gateway station multi-slice resource allocation process model was used as input, the predicted resource allocation strategy was used as output, and the deep reinforcement learning model was trained with the goal of maximizing the reward value.

[0062] The resource allocation module uses a trained deep reinforcement learning model to predict the resource allocation strategy under the current state information, and then allocates network slice resources based on the resource allocation strategy.

[0063] Preferably, the process of establishing a multi-slice resource allocation model for a single ground gateway station specifically includes:

[0064] Using Markov processes<S,A,P,R> This represents the admission decision process for slice requests; where S is the state information, A is the action space, P is the state transition probability, and R is the reward function.

[0065] The state information includes the priority, number of data packet requests, and number of users on a slice within a single time period; the action space is the set of actions A = {a1, a2, a3} performed by the agent on the environment within the current period, where a n The bandwidth allocated to a single slice within this period satisfies the constraint a1 + a2 + a3 = W, where W is the available spectrum resource of the ground gateway station; the state transition probability is the probability of transitioning to the next time state after taking an action; the specific construction process of the reward function is as follows:

[0066] Within a given time period, the available spectrum resources of a ground gateway station are W, and the slice set is S. EP n network slices s n ∈S EP The ground gateway station calculates the data packet volume of each slice based on the user request data packet size. n And priority to allocate bandwidth w for each slice n The data transmission rate v on the slice n As shown below:

[0067] v n =w nlog2(1+SNR n )

[0068] Among them, SNR n The signal-to-noise ratio between the user and the ground gateway station;

[0069] The overall spectrum resource utilization rate (RU) of the ground gateway station is:

[0070]

[0071] Set any slice s n The maximum delay for the final transmission of a single data packet d is l. maxn The minimum data transfer rate required is r minn ; within any time period, any slice s n The total latency for delivering a single data packet to the user is t. n The data packet transmission is considered successful if the following conditions are met:

[0072] t n ≤l maxn

[0073] v n ≥r minn

[0074] Any slice s n The number of data packets successfully transmitted in China is The specific formula is as follows:

[0075]

[0076] In this system, a successful transmission of a single data packet is recorded as d=1, and a failed transmission is recorded as d=0; the ground gateway station receives data from any slice s. n The total set of data packets transmitted in the middle is

[0077] Any slice s n Sat user satisfaction n The ratio of the number of data packets successfully transmitted to the user to the total number of data packets transmitted:

[0078]

[0079] The satisfaction levels of each slice are weighted and summed, with each slice having a weight of α. n The overall user satisfaction with the system is as follows:

[0080]

[0081] The total utility function of the system is as follows:

[0082] E = βRU + Sat

[0083] Wherein, β is a weighting coefficient for spectrum resource utilization, and is set according to priority;

[0084] The total utility function of the system is used as the reward function R.

[0085] Preferably, the process of training a deep reinforcement learning model with the state information of a single ground gateway station multi-slice resource allocation process model as input, the predicted resource allocation strategy as output, and the objective of maximizing the reward value specifically includes:

[0086] The deep reinforcement learning model employs the Advantage Actor-Critic algorithm, which includes an actor network and a critic network; wherein the actor network and the critic network respectively include a policy network and a value function network.

[0087] Initialize the actor and critic networks and their parameters. Within each slice configuration time period T, set the time slots from i to T and initialize an empty buffer of length T. In the i-th time slot, obtain the network slice request status s. i The request status s i The state information includes the request priority, number of data packets, and number of users; the state information is placed in a buffer, and the information in the buffer will constitute the state space within one period T;

[0088] The current state space is used as input; the policy network calculates the probability distribution of actions, selects actions to execute based on the probability distribution, and determines whether to admit slice requests within the current configured time period. After admission, ground gateway resources that maximize efficiency are allocated to slices of different priorities, with low-latency slices having the highest priority, followed by high-speed slices, and large-connection network slices having the lowest priority. Remaining available resources within different time periods are allocated to them; the value function network calculates the state value based on the current state, and the difference between the value obtained by the executed action and the expected action value, plus the reward function value from environmental feedback, to obtain the advantage value of the selected action; the advantage function updates the network state based on the advantage value.

[0089] Preferably, the network state update specifically involves:

[0090] When an action is executed once, the first item is deleted from the buffer, and the new observation state is added to the tail to obtain the state space for the next cycle. The loss values ​​of the policy network and the value function network in the actor network and critic network are calculated respectively, and the network parameters are updated according to the gradient descent method.

[0091] The loss function of the policy network in the actor network is shown below:

[0092] Loss a =-δlogπ(a|s)=-(r+γ) a V(s n+1 )-V(s))logπ(a|s)

[0093] Where 'a' represents the action taken by the actor network in the current state; 'r' represents the reward function value, V(s) n+1 γ is the estimated expected return for the next state, V(s) is the estimated expected return for the current state, and γ is the expected return for the next state. a is the discount factor, reflecting the impact of future state value on the effect of current behavior, with a value between 0 and 1; logπ(a|s) is the log probability of taking action a in state s;

[0094] The update method for the policy network parameter θ in the actor network is as follows:

[0095]

[0096] in, θ represents the updated policy network parameters; α represents the current policy network parameters; and α represents the learning rate, which determines the step size for parameter updates. Let θ be the gradient of the network parameters θ with respect to the current policy; A(s,a) is the advantage function, representing the value advantage of taking action a relative to the average level in state s;

[0097] The value function network loss represents the difference between the predicted Q-value and the actual Q-value output by the critic network; this difference is measured using the mean squared error loss, and the specific formula is shown below:

[0098]

[0099] Where, q π (s n ,a n ;w) represents the Q-value of the critic network, parameterized with w, for predicting the state and action under the current policy network; r t γ represents the reward given by the environment for action a within a time allocation period; c The discount factor for predicting Q-values ​​represents the impact of future predictions on the present. The Q-value for predicting the next state and action in the updated policy network; This indicates the updated network parameters;

[0100] The gradient descent parameter update method of the critic network:

[0101]

[0102] Where k is the learning rate, which determines the step size for updating this parameter; This represents the gradient of the loss function with respect to the parameters;

[0103] The state value function is Among them, s n For network slicing; a n is the bandwidth allocated to a single slice within a single time period; w is the trainable parameter of the state value function;

[0104] By continuously optimizing the target value function, we can improve the satisfaction of target users and the utilization rate of spectrum resources, thereby maximizing the reward value.

[0105] In another aspect, the present invention also provides an electronic device having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements a reinforcement learning-based satellite-ground cooperative network slice resource allocation method as described in any embodiment of the present invention.

[0106] In another aspect, the present invention also provides a computer-readable medium for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement a reinforcement learning-based satellite-ground cooperative network slice resource allocation method as described in any embodiment of the present invention.

[0107] The present invention has the following beneficial effects:

[0108] 1. This invention uses reinforcement learning algorithms to interact directly with the environment and learn through feedback information. Combined with network slicing technology, it enables the slice manager to make slice allocation behavior more intelligent and more in line with application scenarios.

[0109] 2. Intelligent network slicing can effectively utilize satellite-ground integrated resources, better achieving resource allocation that is neither excessive nor insufficient.

[0110] 3. Communication satisfaction is defined as the weighted sum of resource utilization and user satisfaction. The experimental results show that the satisfaction of each slice is over 90%.

[0111] 4. Adopt network resource virtualization technology to make full use of existing communication infrastructure resources and reduce hardware deployment costs. Attached Figure Description

[0112] Figure 1 This is a schematic diagram of the process of the present invention;

[0113] Figure 2 This is a schematic diagram of the ground gateway node algorithm flow of the present invention;

[0114] Figure 3 This invention provides a network application scenario, illustrating network resource slicing for ground gateway nodes in the access network.

[0115] Figure 4 This is a diagram showing the convergence effect of the model after parameter changes in this invention;

[0116] Figure 5 This is a diagram illustrating the resource allocation effect of the system of the present invention under conditions of relatively limited bandwidth resources;

[0117] Figure 6 This is a diagram showing the effect of changing the learning rate of the algorithm in the system of this invention. Detailed Implementation

[0118] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0119] It should be understood that the step numbers used in the text are for ease of description only and are not intended to limit the order in which the steps are performed.

[0120] It should be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0121] The terms “comprising” and “including” indicate the presence of the described feature, whole, step, operation, element and / or component, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or collections thereof.

[0122] The term “and / or” refers to any combination of one or more of the associated listed items, as well as all possible combinations, and includes these combinations.

[0123] Example 1:

[0124] This embodiment provides a method for allocating sliced ​​resources in a satellite-ground collaborative network based on reinforcement learning. The specific steps include:

[0125] S1. The low-Earth orbit satellite network is connected to the local network data center through the ground gateway station, and the user terminal interacts with the low-Earth orbit satellite through the ground gateway station;

[0126] S2. Obtain the business requests submitted by the user, and based on the resource requirements of different types of businesses, create exclusive slices that match the business type.

[0127] S3. The process of resource allocation for each service request by the ground gateway node is regarded as a Markov decision process, and a multi-slice resource allocation process model of a single ground gateway is established.

[0128] S4. For the resource requests of each dedicated slice, construct a system total utility function that considers the overall spectrum resource utilization rate of the ground gateway station and the user demand satisfaction; use the calculation result of the system total utility function as the reward value of the single ground gateway station multi-slice resource allocation process model.

[0129] S5. Construct a deep reinforcement learning model, using the state information of the single ground gateway station multi-slice resource allocation process model as input, the predicted resource allocation strategy as output, and train the deep reinforcement learning model with the goal of maximizing the reward value.

[0130] S6. Predict the resource allocation strategy under the current state information using the trained deep reinforcement learning model, and allocate network slice resources based on the resource allocation strategy.

[0131] In a preferred embodiment of this example, in steps S1-S4, the ground system access network core of satellite communication provides carrying capacity for the accessed services and realizes transparent transmission of services. The gateway station connects the local network data center with the low-orbit satellite network to realize information exchange between the wireless access network and the core network. The user terminal interacts with the communication satellite through the gateway station. Usually, the data traffic transmitted by the user to the gateway station is small, while the backhaul downlink data volume is large.

[0132] Using Markov processes<S,A,P,R> This represents the admission decision process for slice requests; where S is the state information, A is the action space, P is the state transition probability, and R is the reward function.

[0133] The state information includes the priority, number of data packet requests, and number of users on a slice within a single time period; the action space is the set of actions A = {a1, a2, a3} performed by the agent on the environment within the current period, where a n The bandwidth allocated to a single slice within this period satisfies the constraint a1 + a2 + a3 = W, where W is the available spectrum resource of the ground gateway station; the state transition probability is the probability of transitioning to the next time state after taking an action; the specific construction process of the reward function is as follows:

[0134] Within a given time period, the available spectrum resources of a ground gateway station are W, and the slice set is S. EP n network slices sn ∈S EP The ground gateway station calculates the data packet volume of each slice based on the user request data packet size. n And priority to allocate bandwidth w for each slice n High-priority services will be allocated priority. According to Shannon's theorem, if the bandwidth allocated to each slice has been fully utilized, the data transmission rate on that slice is expressed by the formula; the data transmission rate v on the slice... n As shown below:

[0135] v n =w n log2(1+SNR n )

[0136] Among them, SNR n The signal-to-noise ratio between the user and the ground gateway station;

[0137] The overall spectrum resource utilization rate (RU) of the ground gateway station is:

[0138]

[0139] Set any slice s n The maximum delay for the final transmission of a single data packet d is l. maxn The minimum data transfer rate required is r minn ; within any time period, any slice s n The total latency for delivering a single data packet to the user is t. n The data packet transmission is considered successful if the following conditions are met:

[0140] t n ≤l maxn

[0141] v n ≥r minn

[0142] Any slice s n The number of data packets successfully transmitted in China is The specific formula is as follows:

[0143]

[0144] In this system, a successful transmission of a single data packet is recorded as d=1, and a failed transmission is recorded as d=0; the ground gateway station receives data from any slice s. n The total set of data packets transmitted in the middle is

[0145] Any slice s n Sat user satisfaction nThe ratio of the number of data packets successfully transmitted to the user to the total number of data packets transmitted:

[0146]

[0147] The satisfaction levels of each slice are weighted and summed, with each slice having a weight of α. n The overall user satisfaction with the system is as follows:

[0148]

[0149] The total utility function of the system is as follows:

[0150] E = βRU + Sat

[0151] Wherein, β is the weighting coefficient of spectrum resource utilization, and is set according to priority; since user demand is variable, the state of the environment, i.e., the number of user data packet requests, will change in each resource allocation cycle. The gateway station can reasonably allocate bandwidth resources to each slice in each cycle according to the changes in the network environment state, and improve the overall spectrum resource utilization while meeting user demand satisfaction. That is, the optimization goal is to maximize the total system utility function E. This function is also used as a reward function to train the reinforcement learning process, so that it gradually selects the action that maximizes the reward function.

[0152] In a preferred embodiment of this example, step S5, which involves using the state information of a single ground gateway station multi-slice resource allocation process model as input, the predicted resource allocation strategy as output, and training the deep reinforcement learning model with the goal of maximizing the reward value, specifically includes:

[0153] The deep reinforcement learning model employs the Advantage Actor-Critic algorithm, which includes an actor network and a critic network; wherein the actor network and the critic network respectively include a policy network and a value function network.

[0154] First, define the state value function V. π (s) represents the weighted sum of the possible action values, also known as the base value:

[0155] The unique feature of the A2C network is that it uses an advantage function to calculate the gradient, which represents the advantage gained by taking an action in the current state.

[0156] A(s,a)=Q(s,a)-V(s)=E[r+γV(s) n+1 )]-V(s)

[0157] ≈r+γV(s n+1)-V(s)=δ

[0158] In an Actor network, π(a) can be used. n |s n To approximate π(a) with θ) n |s n ), where θ are the trainable parameters of the actor network. Similarly, q can be used in value networks. π (s n ,a n ;w) to approximate Q π (s n ,a n ), where w are the trainable parameters of the value network. Therefore, the value function can be approximated as:

[0159] Initialize the actor and critic networks and their parameters. Within each slice configuration time period T, set the time slots from i to T and initialize an empty buffer of length T. In the i-th time slot, obtain the network slice request status s. i The request status s i The state information includes the request priority, number of data packets, and number of users; the state variables are changing in each time slot, and the state information is put into a buffer. The information in the buffer will constitute the state space within one period T.

[0160] The current state space is used as input; the policy network calculates the probability distribution of actions and selects actions to execute based on the probability distribution. It then determines whether to admit slice requests within the current configured time period. After admission, it allocates ground gateway resources that maximize efficiency to slices of different priorities, with low-latency slices having the highest priority, followed by high-speed slices, and large-connection network slices having the lowest priority. Remaining available resources within different time periods are allocated to each slice. The value function network calculates the state value based on the current state, obtaining the advantage value of the selected action by adding the difference between the value obtained from the executed action and the expected action value, along with the reward function value from the environment. The advantage function updates the network state based on the advantage value, determining the extent of the advantage gained by the selected action by adding the reward function value from the environment. This allows the agent to continuously learn the scheme with the highest advantage value, thereby gradually interacting with the environment to achieve optimal network slice resource allocation.

[0161] Preferably, the network state update specifically involves:

[0162] When an action is executed once, the first item is deleted from the buffer, and the new observation state is added to the tail to obtain the state space for the next cycle. The loss values ​​of the policy network and the value function network in the actor network and critic network are calculated respectively, and the network parameters are updated according to the gradient descent method.

[0163] The loss function of the policy network in the actor network is shown below:

[0164] Loss a =-δlogπ(a|s)=-(r+γ) a V(s n+1 )-V(s))logπ(a|s)

[0165] Where 'a' represents the action taken by the actor network in the current state, which measures the quality of taking a certain action in the current state, increasing the probability of good actions and decreasing the probability of bad actions; 'r' represents the reward function value, V(s) n+1 γ is the estimated expected return for the next state, V(s) is the estimated expected return for the current state, and γ is the expected return for the next state. a is the discount factor, reflecting the impact of future state value on the effect of current behavior, with a value between 0 and 1; logπ(a|s) is the log probability of taking action a in state s;

[0166] The update method for the policy network parameter θ in the actor network is as follows:

[0167]

[0168] in, θ represents the updated policy network parameters; α represents the current policy network parameters; and α represents the learning rate, which determines the step size for parameter updates. Let θ be the gradient of the network parameters θ with respect to the current policy; A(s,a) is the advantage function, representing the value advantage of taking action a relative to the average level in state s;

[0169] The value function network loss represents the difference between the predicted Q-value and the actual Q-value output by the critic network; this difference is measured using the mean squared error loss, and the specific formula is shown below:

[0170]

[0171] Where, q π (s n ,a n ;w) represents the Q-value of the critic network, parameterized with w, for predicting the state and action under the current policy network; r t γ represents the reward given by the environment for action a within a time allocation period;c The discount factor for predicting Q-values ​​represents the impact of future predictions on the present. The Q-value for predicting the next state and action in the updated policy network; This indicates the updated network parameters;

[0172] The gradient descent parameter update method of the critic network:

[0173]

[0174] Where k is the learning rate, which determines the step size for updating this parameter; This represents the gradient of the loss function with respect to the parameters;

[0175] The state value function is Among them, s n For network slicing; a n is the bandwidth allocated to a single slice within a single time period; w is the trainable parameter of the state value function;

[0176] This paper employs the A2C algorithm from reinforcement learning to address the intelligent network slicing resource allocation problem in power application scenarios. The goal is to optimize user satisfaction and spectrum resource utilization, thereby maximizing the overall system utility function. During continuous training, the reward function gradually converges to its optimum. A slice list is created, recording the converged state and action spaces, thus establishing a queryable network slice resource allocation platform for power network application slice requests. To effectively match the environment, the overall system utility function is modeled as a weighted function of spectrum resource utilization and slice satisfaction. Spectrum resource utilization reflects the rational allocation of resources by network slices, while slice satisfaction reflects whether user communication needs meet standards such as low latency, high connectivity, and high speed.

[0177] The network slice resource allocation process starts from the number and priority of data packet requests on a slice within a single cycle. More resources are expected to be allocated to slices with higher data packet transmission rate requirements and larger data packet quantities; this is defined as state information. To fully utilize the communication advantages of low-Earth orbit satellite networks, agents deployed on gateway stations can make policy selections based on the differences in service requirements of each slice, i.e., based on state information. They select actions to take in the current cycle, i.e., the bandwidth allocated to a single slice. If the bandwidth granularity is too large, the flexibility of resource allocation will decrease, and it will not be able to fully adapt to diverse data packet requests. If the granularity is set too small, it will increase the allocation burden of the system, add complexity to the choice of action space, and affect the decision-making speed, but it can more accurately adapt to changes in the state of the environment.

[0178] By continuously optimizing the target value function, we can improve the satisfaction of target users and the utilization rate of spectrum resources, thereby maximizing the reward value.

[0179] As one implementation method, such as Figure 4 As shown, the total bandwidth of the ground gateway station is set to 10MHz, the bandwidth fineness is set to 0.2, the policy network learning rate is 0.001, the value network learning rate is 0.01, and the number of iterations is 10000. The entire process of the system's self-training is given here, and only the convergence effect of the model after parameter changes is observed.

[0180] Initialize the actor and critic networks; in each slice configuration cycle:

[0181] Initialize the executor and commenter parameters; initialize an empty buffer of length T; execute in each time period from i to T:

[0182] 1. Randomly select an action 'a' from the action space and execute it;

[0183] 2. In the i-th cycle, the agent obtains the observed p;

[0184] 3. Place p at the end of the buffer;

[0185] Repeat after completion:

[0186] 1. Form a state space st for each observation in the buffer;

[0187] 2. The state processing network requires st as input and calculates st';

[0188] 3. Decision networks require the next state space as input and calculate the probability distribution of actions;

[0189] 4. Value networks require the next state space as input to compute the state value V;

[0190] 5. The agent selects and executes an action from the probability distribution of actions;

[0191] 6. The agent receives a reward and gains new observations;

[0192] 7. Delete the first value in the buffer and append the new observation to the end;

[0193] 8. Connect the new buffers to form st+1;

[0194] 9. The state processing network requires st+1 as input and calculates st+1';

[0195] 10. The value network requires st+1' as input and computes V';

[0196] 11. Calculate the Q-value function;

[0197] 12. Calculate the loss function and update the AC network parameters;

[0198] 13. Enter the next time cycle.

[0199] As one implementation method, such as Figure 5 As shown, the gateway station bandwidth was set to 5MHz, and the effect of resource allocation in the system under relatively limited bandwidth resources was observed. The bandwidth fineness, policy network learning rate, value network learning rate, discount factor, and number of iterations remained unchanged.

[0200] As one implementation method, such as Figure 6 As shown, the gateway bandwidth was set to 10MHz, and the learning rates of the executor and commentator networks of the A2C algorithm were changed to 0.001 and 0.01, respectively. The scenario environment conditions, discount factor and number of iterations remained unchanged, and the effect of the system after the algorithm learning rate was changed was observed.

[0201] Example 2:

[0202] This embodiment provides a satellite-ground collaborative network slice resource allocation system based on reinforcement learning, including:

[0203] The satellite-ground coordination module connects the low-Earth orbit satellite network with the local network data center through a ground gateway station, and the user terminal interacts with the low-Earth orbit satellite through the ground gateway station; this module is used to implement the function of step S1 in the above implementation, and the details will not be elaborated further.

[0204] The slicing module obtains the business requests submitted by the user, and based on the resource requirements of different types of business, creates exclusive slices that match the business type. This module is used to implement the function of step S2 in implementation one above, and will not be described in detail here.

[0205] A resource allocation model module is constructed, which treats the process of resource allocation for each service request by the ground gateway node as a Markov decision process, and establishes a resource allocation process model for a single ground gateway station with multiple slices. This module is used to implement the function of step S3 in the above implementation, and the details will not be elaborated here.

[0206] The system total utility function module constructs a system total utility function that considers the overall spectrum resource utilization rate of the ground gateway station and user demand satisfaction for the resource requests of each dedicated slice; the calculation result of the system total utility function is used as the reward value of the single ground gateway station multi-slice resource allocation process model; this module is used to implement the function of step S4 in the above implementation one, and the details will not be elaborated further.

[0207] The model optimization module constructs a deep reinforcement learning model, using the state information of the single ground gateway station multi-slice resource allocation process model as input and the predicted resource allocation strategy as output, and trains the deep reinforcement learning model with the goal of maximizing the reward value; this module is used to implement the function of step S5 in the above implementation one, and the details will not be elaborated further.

[0208] The resource allocation module predicts the resource allocation strategy under the current state information through a trained deep reinforcement learning model, and performs network slice resource allocation based on the resource allocation strategy; this module is used to implement the function of step S6 in the above implementation one, and will not be described in detail here.

[0209] Example 3:

[0210] This embodiment provides an electronic device that stores a computer program. When the computer program is executed by a processor, it implements a reinforcement learning-based satellite-ground cooperative network slice resource allocation method as described in any embodiment of the present invention.

[0211] Example 4:

[0212] This embodiment provides a computer-readable medium for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement a reinforcement learning-based satellite-ground cooperative network slice resource allocation method as described in any embodiment of the present invention.

[0213] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, A and B simultaneously, or B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0214] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments disclosed herein can be implemented using electronic hardware, computer software, or a combination of electronic hardware and software. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0215] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0216] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0217] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for allocating sliced ​​resources in a satellite-ground cooperative network based on reinforcement learning, characterized in that, Includes the following steps: The low-Earth orbit satellite network is connected to the local network data center through a ground gateway station, and the user terminal interacts with the low-Earth orbit satellite through the ground gateway station; Get the business requests submitted by the user, obtain the resource requirements of different types of business based on the business requests, and create exclusive slices that match the business type; The process of resource allocation for each service request by the ground gateway node is regarded as a Markov decision process, and a multi-slice resource allocation process model of a single ground gateway is established. For resource requests from each dedicated slice, a total system utility function is constructed that considers the overall spectrum resource utilization of ground gateway stations and user satisfaction. The calculation result of the total system utility function is used as the reward value of the single ground gateway station multi-slice resource allocation process model; A deep reinforcement learning model is constructed, taking the state information of the single ground gateway station multi-slice resource allocation process model as input, the predicted resource allocation strategy as output, and training the deep reinforcement learning model with the goal of maximizing the reward value. The resource allocation strategy is predicted based on the current state information by a trained deep reinforcement learning model, and then network slice resource allocation is performed based on the resource allocation strategy. The deep reinforcement learning model employs the Advantage Actor-Critic algorithm, comprising an actor network and a critic network. The actor network and critic network each include a policy network and a value function network, respectively. The value function network calculates the state value based on the current state, and the difference between the value obtained from the executed action and the expected action value, plus the reward function value from environmental feedback, yields the advantage value for the selected action. The advantage function network updates its state based on this advantage value. The network status update specifically refers to: When an action is executed once, the first item is deleted from the buffer, and the new observation state is added to the tail to obtain the state space for the next cycle. The loss values ​​of the policy network and the value function network in the actor network and critic network are calculated respectively, and the network parameters are updated according to the gradient descent method. The loss function of the policy network in the actor network is shown below: in, The action taken by the actor network in the current state; For the reward function value, The estimated return for the next state. The estimated return for the current state. This is a discount factor that reflects the impact of future state value on the effect of current behavior, and its value ranges from 0 to 1. Let be the logarithmic probability of taking action a in state s; Policy network parameters in actor network The update method is as follows: in, For the updated policy network parameters; These are the network parameters for the current policy. The learning rate is used to determine the step size for parameter updates; To adjust the network parameters of the current policy The gradient; Let be the advantage function, representing the value advantage of taking action a relative to the average level in state s; The value function network loss represents the difference between the predicted Q-value and the actual Q-value output by the critic network; this difference is measured using the mean squared error loss, and the specific formula is shown below: in, For the current strategy Next, use The parameterized critic network predicts the Q-values ​​of states and actions; The reward given by the environment for action a within a time allocation period; The discount factor for predicting Q-values ​​represents the impact of future predictions on the present. The Q-value for predicting the next state and action in the updated policy network; This indicates the updated network parameters; The gradient descent parameter update method of the critic network: Where k is the learning rate, which determines the step size for updating this parameter; This represents the gradient of the loss function with respect to the parameters; The state value function is ;in, Network slicing; The bandwidth allocated to a single slice within a single time period; These are the trainable parameters of the state-value function; By continuously optimizing the target value function, we can improve the satisfaction of target users and the utilization rate of spectrum resources, thereby maximizing the reward value.

2. The method for allocating satellite-ground cooperative network slice resources based on reinforcement learning according to claim 1, characterized in that, The process of establishing a multi-slice resource allocation model for a single ground gateway station specifically includes: Using Markov processes<S,A,P,R> This represents the admission decision process for slice requests; where S is the state information, A is the action space, P is the state transition probability, and R is the reward function. The state information includes the priority, number of data packet requests, and number of users on the slice within a single time period; the action space is the set of actions performed by the agent on the environment within the current period. ,in The bandwidth allocated to a single slice within this period, while satisfying the constraints. , The available spectrum resources of the ground gateway station; the state transition probability is the probability of transitioning to the next time state after taking an action; the specific construction process of the reward function is as follows: Within a given time period, the available spectrum resources of a ground gateway station are The slice set is n network slices The ground gateway station determines the amount of data packets requested by the user for each slice. And priority to allocate bandwidth to each slice Data transmission rate on the slice As shown below: in, The signal-to-noise ratio between the user and the ground gateway station; Overall spectrum resource utilization rate of ground gateway stations for: Set any slice The last single data packet The maximum delay for the final transmission to arrive is The minimum data transfer rate required ; within any time period, any slice The total latency for delivering a single data packet to the user is The data packet transmission is considered successful if the following conditions are met: Any slice The number of data packets successfully transmitted in China is The specific formula is as follows: Among them, a single data packet transmission success record If the transmission fails, record 0; Ground gateway stations from any slice The total set of data packets transmitted in the middle is ; Any slice User satisfaction The ratio of the number of data packets successfully transmitted to the user to the total number of data packets transmitted: The satisfaction levels of each slice are weighted and summed, with the weight of each slice being... The overall user satisfaction with the system is as follows: The total utility function of the system is as follows: in, This is a weighting coefficient for spectrum resource utilization, set according to priority. The total utility function of the system is used as the reward function R.

3. The method for allocating satellite-ground cooperative network slice resources based on reinforcement learning according to claim 1, characterized in that, The process of training a deep reinforcement learning model with the state information of a single ground gateway station multi-slice resource allocation process model as input, the predicted resource allocation strategy as output, and the objective of maximizing the reward value specifically includes: Initialize the actor and critic networks and their parameters. Within each slice configuration time period T, set the time slots from i to T and initialize an empty buffer of length T. In the i-th time slot, obtain the network slice request status. The request status The state information includes the request priority, number of data packets, and number of users; the state information is placed in a buffer, and the information in the buffer will constitute the state space within one period T; The current state space is used as input; the policy network calculates the probability distribution of actions, selects the action to be executed based on the probability distribution, determines whether to admit slice requests in the current configured time period, and after admission, allocates ground gateway resources that maximize the benefits to slices of different priorities, with low-latency slices having the highest priority, high-speed slices second, and large-connection network slices having the lowest priority, and allocates the remaining available resources in different time periods to them.

4. A satellite-ground collaborative network slice resource allocation system based on reinforcement learning, characterized in that, include: The satellite-ground coordination module connects the low-Earth orbit satellite network with the local network data center through a ground gateway station, and the user terminal interacts with the low-Earth orbit satellite through the ground gateway station; The slicing module obtains business requests from the user, acquires the resource requirements of different types of business based on the business requests, and creates exclusive slices that match the business type. A resource allocation model module is constructed, which takes the process of allocating resources to each service request by the ground gateway node as a Markov decision process and establishes a resource allocation process model for a single ground gateway station with multiple slices. The system total utility function module constructs a system total utility function that considers the overall spectrum resource utilization rate of ground gateway stations and user demand satisfaction for the resource requests of each dedicated slice; The calculation result of the total system utility function is used as the reward value of the single ground gateway station multi-slice resource allocation process model; The model module was optimized, and a deep reinforcement learning model was constructed. The state information of the single ground gateway station multi-slice resource allocation process model was used as input, the predicted resource allocation strategy was used as output, and the deep reinforcement learning model was trained with the goal of maximizing the reward value. The resource allocation module predicts the resource allocation strategy under the current state information through a trained deep reinforcement learning model, and performs network slice resource allocation based on the resource allocation strategy. The deep reinforcement learning model employs the Advantage Actor-Critic algorithm, comprising an actor network and a critic network. The actor network and critic network each include a policy network and a value function network, respectively. The value function network calculates the state value based on the current state, and the difference between the value obtained from the executed action and the expected action value, plus the reward function value from environmental feedback, yields the advantage value for the selected action. The advantage function network updates its state based on this advantage value. The network status update specifically refers to: When an action is executed once, the first item is deleted from the buffer, and the new observation state is added to the tail to obtain the state space for the next cycle. The loss values ​​of the policy network and the value function network in the actor network and critic network are calculated respectively, and the network parameters are updated according to the gradient descent method. The loss function of the policy network in the actor network is shown below: in, The action taken by the actor network in the current state; For the reward function value, The estimated return for the next state. The estimated return for the current state. This is a discount factor that reflects the impact of future state value on the effect of current behavior, and its value ranges from 0 to 1. Let be the logarithmic probability of taking action a in state s; Policy network parameters in actor network The update method is as follows: in, For the updated policy network parameters; These are the network parameters for the current policy. The learning rate is used to determine the step size for parameter updates; To adjust the network parameters of the current policy The gradient; Let be the advantage function, representing the value advantage of taking action a relative to the average level in state s; The value function network loss represents the difference between the predicted Q-value and the actual Q-value output by the critic network; this difference is measured using the mean squared error loss, and the specific formula is shown below: in, For the current strategy Next, use The parameterized critic network predicts the Q-values ​​of states and actions; The reward given by the environment for action a within a time allocation period; The discount factor for predicting Q-values ​​represents the impact of future predictions on the present. The Q-value for predicting the next state and action in the updated policy network; This indicates the updated network parameters; The gradient descent parameter update method of the critic network: Where k is the learning rate, which determines the step size for updating this parameter; This represents the gradient of the loss function with respect to the parameters; The state value function is ;in, Network slicing; The bandwidth allocated to a single slice within a single time period; These are the trainable parameters of the state-value function; By continuously optimizing the target value function, we can improve the satisfaction of target users and the utilization rate of spectrum resources, thereby maximizing the reward value.

5. A satellite-ground cooperative network slice resource allocation system based on reinforcement learning according to claim 4, characterized in that, The process of establishing a multi-slice resource allocation model for a single ground gateway station specifically includes: Using Markov processes<S,A,P,R> This represents the admission decision process for slice requests; where S is the state information, A is the action space, P is the state transition probability, and R is the reward function. The state information includes the priority, number of data packet requests, and number of users on the slice within a single time period; the action space is the set of actions performed by the agent on the environment within the current period. ,in The bandwidth allocated to a single slice within this period, while satisfying the constraints. , The available spectrum resources of the ground gateway station; the state transition probability is the probability of transitioning to the next time state after taking an action; the specific construction process of the reward function is as follows: Within a given time period, the available spectrum resources of a ground gateway station are The slice set is n network slices The ground gateway station determines the amount of data packets requested by the user for each slice. And priority to allocate bandwidth to each slice Data transmission rate on the slice As shown below: in, The signal-to-noise ratio between the user and the ground gateway station; Overall spectrum resource utilization rate of ground gateway stations for: Set any slice The last single data packet The maximum delay for the final transmission to arrive is The minimum data transfer rate required ; within any time period, any slice The total latency for delivering a single data packet to the user is The data packet transmission is considered successful if the following conditions are met: Any slice The number of data packets successfully transmitted in China is The specific formula is as follows: Among them, a single data packet transmission success record If the transmission fails, record 0; Ground gateway stations from any slice The total set of data packets transmitted in the middle is ; Any slice User satisfaction The ratio of the number of data packets successfully transmitted to the user to the total number of data packets transmitted: The satisfaction levels of each slice are weighted and summed, with the weight of each slice being... The overall user satisfaction with the system is as follows: The total utility function of the system is as follows: in, This is a weighting coefficient for spectrum resource utilization, set according to priority. The total utility function of the system is used as the reward function R.

6. A satellite-ground cooperative network slice resource allocation system based on reinforcement learning according to claim 4, characterized in that, The process of training a deep reinforcement learning model with the state information of a single ground gateway station multi-slice resource allocation process model as input, the predicted resource allocation strategy as output, and the objective of maximizing the reward value specifically includes: Initialize the actor and critic networks and their parameters. Within each slice configuration time period T, set the time slots from i to T and initialize an empty buffer of length T. In the i-th time slot, obtain the network slice request status. The request status The state information includes the request priority, number of data packets, and number of users; the state information is placed in a buffer, and the information in the buffer will constitute the state space within one period T; The current state space is used as input; the policy network calculates the probability distribution of actions, selects the action to be executed based on the probability distribution, determines whether to admit slice requests in the current configured time period, and after admission, allocates ground gateway resources that maximize the benefits to slices of different priorities, with low-latency slices having the highest priority, high-speed slices second, and large-connection network slices having the lowest priority, and allocates the remaining available resources in different time periods to them.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements a reinforcement learning-based satellite-ground collaborative network slice resource allocation method as described in any one of claims 1 to 3.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements a reinforcement learning-based satellite-ground cooperative network slice resource allocation method as described in any one of claims 1 to 3.

Images