Resource allocation method, device, equipment and medium

Abstract

The present application relates to the field of satellite communications, and specifically to a resource allocation method, apparatus, device, and medium for achieving efficient utilization of satellite internet wireless communication resources. The method is applied to a core network element of a communication system, the communication system also including a terminal, a satellite, and a cloud computing center. The method comprises: obtaining status information of multiple first terminals through requests from multiple first terminals; the status information includes task queues received by the multiple first terminals and transmission channels selectable by the multiple first terminals; inputting the status information into a trained resource allocation model to obtain a resource allocation strategy; the trained resource allocation model is obtained by training the resource allocation model with minimizing the total energy consumption of the communication system as the optimization goal; and sending signaling containing the resource allocation strategy to the multiple first terminals via the satellite, so that the multiple first terminals transmit their respective received task queues to the cloud computing center according to their corresponding transmission channels and transmission powers.

Description

Technical Field

[0001] The present application relates to the field of satellite communications, and specifically to a resource allocation method, apparatus, device, and medium. Background Art

[0002] In recent years, cloud-network convergence has become a major development trend in the cloud computing field. This convergence involves integrating communication network technologies into cloud computing and cloud computing into communication networks. However, current terrestrial communication networks, as part of this convergence, still lack coverage for over 80% of land areas and over 95% of ocean areas, hindering the extension of cloud computing services to these regions. Therefore, existing cloud-network convergence cannot achieve ubiquitous connectivity and global coverage. Satellite internet, as a new strategic space infrastructure enabling global, all-encompassing, all-time, and end-to-end information network interconnection, combined with the concept of cloud-network convergence, can bring cloud computing services to every corner of the world.

[0003] Current resource allocation for cloud-network convergence primarily focuses on allocating and scheduling computing and storage resources (e.g., CPU, storage space, etc.) within cloud computing centers. Satellite internet wireless communication resources are extremely limited, and achieving efficient utilization of these resources is a key issue. Summary of the Invention

[0004] The embodiments of the present application provide a resource allocation method, apparatus, device, and medium for solving the problem of how to achieve efficient utilization of satellite Internet wireless communication resources.

[0005] In a first aspect, the present application provides a resource allocation method, which is applied to a core network element of a communication system, wherein the communication system further includes a terminal, a satellite, and a cloud computing center. The method includes:

[0006] Acquiring status information of the plurality of first terminals through requests of the plurality of first terminals; wherein the status information includes task queues received by the plurality of first terminals and transmission channels selectable by the plurality of first terminals;

[0007] Inputting the state information into a trained resource allocation model to obtain a resource allocation strategy; wherein the trained resource allocation model is obtained by training the resource allocation model with minimizing the total energy consumption of the communication system as an optimization goal, and the resource allocation strategy is used to indicate the transmission channels and transmission powers corresponding to the multiple first terminals;

[0008] The signaling including the resource allocation strategy is sent to the multiple first terminals via the satellite, so that the multiple first terminals send their respective received task queues to the cloud computing center according to the resource allocation strategy.

[0009] In one embodiment of the present invention, the total energy consumption of the communication system includes processing energy consumption of the multiple first terminals, transmission energy consumption between the multiple first terminals and the satellite, and processing energy consumption of the cloud computing center.

[0010] In another embodiment of the present invention, before inputting the state information into the trained resource allocation model to obtain the resource allocation strategy, the method further includes:

[0011] A plurality of training data are obtained by randomly sampling from pre-stored experience replay data; wherein the experience replay data includes a plurality of data pieces in consecutive time slots, each data piece includes current time slot state information, a target action, a reward, and next time slot state information, the reward being negatively correlated with the total energy consumption of the communication system when a plurality of second terminals execute the target action under the current time slot state information, and the target action includes the plurality of second terminals selecting corresponding transmission channels and corresponding transmission powers;

[0012] The resource allocation model is trained based on the multiple training data to obtain the trained resource allocation model.

[0013] In another embodiment of the present invention, the resource allocation model includes a first network and a second network with identical structures; training the resource allocation model based on the training data to obtain the trained resource allocation model includes:

[0014] Inputting the current time slot state information and the target action of the first training data into the first network to obtain a first result; wherein the first result represents the predicted value of the plurality of second terminals performing the target action of the first training data under the current time slot state information of the first training data;

[0015] Inputting the next time slot state information of the first training data into the second network to obtain a second result, wherein the second result represents a maximum value among multiple values ​​of the multiple second terminals performing multiple actions under the next time slot state information of the first training data, the multiple actions being actions in a pre-established action space;

[0016] updating the parameters of the first network based on an error between the first result and a third result; wherein the third result represents actual value of the plurality of second terminals executing the target action of the first training data under the current time slot state information of the first training data, and the third result is positively correlated with the reward of the first training data and the second result;

[0017] When the number of iterations of the first network is an integer multiple of a preset number, the parameters of the first network are assigned to the second network until the number of assignments of the second network reaches a preset threshold, and the second network is used as the trained resource allocation model.

[0018] In another embodiment of the present invention, before randomly sampling from pre-stored experience playback data to obtain a plurality of training data, the method further includes:

[0019] Determining target actions corresponding to multiple time slot state information from the multiple actions according to a preset strategy;

[0020] executing target actions corresponding to the multiple time slot state information respectively under the multiple time slot state information, obtaining next time slot state information corresponding to the multiple time slot state information respectively, and determining rewards corresponding to the multiple time slot state information respectively based on the total energy consumption of the communication system;

[0021] Save the multiple time slot state information, the target actions corresponding to the multiple time slot state information, the rewards corresponding to the multiple time slot state information, and the next time slot state information corresponding to the multiple time slot state information to obtain the experience replay data.

[0022] In a second aspect, the present application provides a resource allocation device, which is provided in a core network element of a communication system for satellite internet, wherein the communication system further includes a terminal, a satellite, and a cloud computing center, and the device includes:

[0023] An information acquisition module, configured to acquire status information of the plurality of first terminals through requests of the plurality of first terminals; wherein the status information includes task queues received by the plurality of first terminals and transmission channels selectable by the plurality of first terminals;

[0024] a policy decision module, configured to input the state information into a trained resource allocation model to obtain a resource allocation policy; wherein the trained resource allocation model is trained with minimizing the total energy consumption of the communication system as an optimization goal, and the resource allocation policy is used to indicate the transmission channels and transmission powers corresponding to the multiple first terminals;

[0025] The sending module is used to send the signaling containing the resource allocation strategy to the multiple terminals via the satellite, so that the multiple first terminals send their respective received task queues to the cloud computing center according to the resource allocation strategy.

[0026] In one embodiment of the present invention, the total energy consumption of the communication system includes processing energy consumption of the multiple first terminals, transmission energy consumption between the multiple first terminals and the satellite, and processing energy consumption of the cloud computing center.

[0027] In another embodiment of the present invention, the apparatus further comprises a model training module, wherein the model training module is configured to:

[0028] Before inputting the state information into a trained resource allocation model to obtain a resource allocation strategy, a plurality of training data are randomly sampled from pre-stored experience replay data; wherein the experience replay data includes a plurality of data pieces in consecutive time slots, each data piece includes current time slot state information, a target action, a reward, and next time slot state information, the reward being negatively correlated with the total energy consumption of the communication system when multiple second terminals perform the target action under the current time slot state information, and the target action includes each second terminal selecting a corresponding transmission channel and a corresponding transmission power;

[0029] The resource allocation model is trained based on the multiple training data to obtain the trained resource allocation model.

[0030] In another embodiment of the present invention, the resource allocation model includes a first network and a second network with the same structure; and the model training module is specifically configured to:

[0031] Inputting the current time slot state information and the target action of the first training data into the first network to obtain a first result; wherein the first result represents the predicted value of the plurality of second terminals performing the target action of the first training data under the current time slot state information of the first training data;

[0032] Inputting the next time slot state information of the first training data into the second network to obtain a second result, wherein the second result represents a maximum value among multiple values ​​of the multiple second terminals performing multiple actions under the next time slot state information of the first training data, the multiple actions being actions in a pre-established action space;

[0033] updating the parameters of the first network based on an error between the first result and a third result; wherein the third result represents actual value of the plurality of second terminals executing the target action of the first training data under the current time slot state information of the first training data, and the third result is positively correlated with the reward of the first training data and the second result;

[0034] When the number of iterations of the first network is an integer multiple of a preset number, the parameters of the first network are assigned to the second network until the number of assignments of the second network reaches a preset threshold, and the second network is used as the trained resource allocation model.

[0035] In another embodiment of the present invention, the model training module is further used to:

[0036] Before randomly sampling a plurality of training data from pre-stored experience playback data, determining target actions corresponding to a plurality of time slot state information from the plurality of actions according to a preset strategy;

[0037] executing target actions corresponding to the multiple time slot state information respectively under the multiple time slot state information, obtaining next time slot state information corresponding to the multiple time slot state information respectively, and determining rewards corresponding to the multiple time slot state information respectively based on the total energy consumption of the communication system;

[0038] Save the multiple time slot state information, the target actions corresponding to the multiple time slot state information, the rewards corresponding to the multiple time slot state information, and the next time slot state information corresponding to the multiple time slot state information to obtain the experience replay data.

[0039] In a third aspect, the present application provides an electronic device, comprising:

[0040] a memory for storing program instructions;

[0041] The processor is configured to call the program instructions stored in the memory and execute the method according to any one of the first aspects according to the obtained program instructions.

[0042] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes any one of the methods according to the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technology descriptions.

[0044] Figure 1 A schematic diagram of an application scenario of a resource allocation method provided in an embodiment of the present application;

[0045] Figure 2 A schematic diagram of a resource allocation method according to an embodiment of the present invention;

[0046] Figure 3 A structural diagram of a resource allocation device provided in an embodiment of the present application;

[0047] Figure 4 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] To make the purpose, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Among them, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0049] Moreover, in the description of the embodiments of the present application, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two.

[0050] In the following, the terms "first" and "second" are used for descriptive purposes only and should not be understood to imply or suggest relative importance or implicitly indicate the number of the technical features indicated. Therefore, features defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of this application, unless otherwise specified, "plurality" means two or more.

[0051] First, we briefly introduce the application scenarios to which the technical solutions of the embodiments of this application can be applied. It should be noted that the application scenarios described below are only used to illustrate the embodiments of this application and are not limiting. In the specific implementation process, the technical solutions provided by the embodiments of this application can be flexibly applied according to actual needs.

[0052] The resource allocation method provided in the embodiments of the present application can be applied to various communication systems, such as the Long Term Evolution (LTE) system, the Global System for Mobile Communication (GSM), the Universal Mobile Telecommunications System (UMTS), the Code Division Multiple Access (CDMA) system, the device-to-device (D2D) system, the vehicle-to-everything (V2X) system, and other communication systems between various ground terminals and satellites. The following describes the embodiments of the present application using a cloud-network converged communication system for satellite internet as an example.

[0053] Please refer to Figure 1, which is a schematic diagram of an application scenario of a resource allocation method provided in an embodiment of the present application, or can be understood as a structural diagram of a communication system, which includes a terminal 101, a satellite 102, a gateway station 103, a core network element 104, and a cloud computing center 105.

[0054] Orthogonal Frequency Division Multiple Access (OFDMA) is used between the terminal 101 and the satellite 102 to transmit data on multiple wireless channels. The terminal 101 and the satellite 102, as well as the satellite 102 and the gateway 103, are connected via wireless links. The gateway 103 and the cloud computing center 105 are connected via optical fiber. The gateway 103 and the cloud computing center 105 communicate with each other through the core network element 104.

[0055] Terminal 101 is a device that can provide voice and / or data connectivity to users. For example, terminal devices include handheld devices with wireless connection capabilities, vehicle-mounted devices, etc. Currently, terminal devices can be: mobile phones, tablets, laptops, PDAs, mobile internet devices (MIDs), wearable devices, virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, or wireless terminals in smart homes, etc. The computing tasks of terminal 101 can be processed locally or offloaded to the cloud computing center 105 for processing via a satellite link.

[0056] Cloud computing center 105 is a server located on the ground, providing high-performance computing services to various sectors based on high-performance computers. The server can be a standalone physical server, a server cluster composed of multiple physical servers, or a distributed system. It can also provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms, but is not limited to these.

[0057] It should be noted that Figure 1The example uses one terminal 101, one satellite 102, and one gateway 103. In practice, there is no limit to the number of terminals 101, satellites 102, and gateways 103. Multiple satellites 102 form a low-orbit satellite constellation, and adjacent satellites 102 communicate via intersatellite links. A single cloud computing center 105 exists in the system.

[0058] In order to achieve efficient utilization of satellite Internet wireless communication resources, an embodiment of the present application provides a resource allocation method, which can be executed by the core network element 104 in the communication system.

[0059] Specifically, when terminal 101 receives a task, it can send a request to satellite 102. Satellite 102 forwards the request to gateway 103, which in turn forwards the request to core network element 104. After receiving terminal 101's request, core network element 104 allocates a corresponding transmit channel and transmit power to terminal 101. Terminal 101 then sends the task to satellite 102 according to the allocated transmit channel and transmit power. Satellite 102 forwards the task to gateway 103, which in turn forwards the task to core network element 104, which then offloads the task to cloud computing center 105. The specific allocation of transmit channels and transmit power by core network element 104 will be described in detail below.

[0060] As described above, the application scenarios are combined with Figure 1 The application scenario shown is Figure 1 The resource allocation method executed by the core network element 104 in the example is introduced. Figure 2 , which is a flow chart of a resource allocation method provided in an embodiment of the present application.

[0061] S201: Acquire status information of multiple first terminals through requests of multiple first terminals.

[0062] When the first terminal receives the task queue, it can forward the request to the core network element through the satellite and the gateway, wherein the request includes the task queue received by the first terminal. Please refer to the content discussed above for the request forwarding process, which will not be repeated here. After receiving the requests from multiple first terminals, the core network element parses the requests from the multiple first terminals, obtains the task queues received by the multiple first terminals, and based on the requests from the multiple first terminals, queries the total transmission channels and the currently occupied transmission channels, thereby obtaining the transmission channels that can be selected by the multiple first terminals, and then obtains the status information of the multiple first terminals. The status information includes the task queues received by the multiple first terminals and the transmission channels that can be selected by the multiple first terminals.

[0063] S202: Input the state information into the trained resource allocation model to obtain a resource allocation strategy.

[0064] After the core network element obtains the status information of the multiple first terminals, it can input the status information of the multiple first terminals into a trained resource allocation model to obtain a resource allocation policy. The trained resource allocation model is trained with minimizing the total energy consumption of the communication system as an optimization objective, and the resource allocation policy is used to indicate the transmit channels and transmit powers corresponding to the multiple first terminals.

[0065] S203: Send a signaling including a resource allocation strategy to the plurality of first terminals via a satellite, so that the plurality of first terminals send their respective received task queues to the cloud computing center according to the resource allocation strategy.

[0066] After obtaining the resource allocation policy, the core network element sends a signaling message containing the resource allocation policy to the gateway. The gateway sends the signaling message to the satellite, which then sends the signaling message to multiple first terminals. After receiving the signaling message, the multiple first terminals parse the signaling message to obtain the resource allocation policy. The multiple first terminals then send their respective received task queues to the satellite according to the transmit channels and transmit powers specified in the resource allocation policy. The satellite then forwards the task queues sent by the multiple first terminals to the gateway. The gateway then forwards the task queues sent by the multiple terminals to the core network element. The core network element then forwards the task queues sent by the multiple first terminals to the cloud computing center.

[0067] In an embodiment of the present application, the allocation of wireless resources and computing resources is jointly considered, and a trained resource allocation model is obtained with the optimization goal of minimizing the energy consumption of the cloud-network fusion system. When requests from multiple first terminals are received, the status information of the multiple first terminals is input into the trained resource allocation model to obtain a resource allocation strategy. Each of the multiple first terminals can send its respective receiving tasks on the corresponding transmission channel according to the corresponding transmission power in the resource allocation strategy, thereby efficiently utilizing wireless communication resources while avoiding channel conflicts between the terminals.

[0068] In one embodiment of the present invention, the total energy consumption of the communication system includes the processing energy consumption of the multiple first terminals, the transmission energy consumption between the multiple first terminals and the satellite, and the processing energy consumption of the cloud computing center. The various energy consumptions are described below.

[0069] 1. Terminal processing energy consumption.

[0070]

[0071] in, is the local processing energy consumption of terminal n in time slot t, γ1 is a constant factor determined by the processing capability of terminal n. k0 is the computing resource consumed by processing the unit bit task, D n(t) is the amount of data processed locally by terminal n in time slot t, and its formula is as follows:

[0072]

[0073] in, is the local data processing time of terminal n, represents the computing capability of terminal n, and k0 is the computing resource consumed to process the unit bit task.

[0074] 2. Transmission energy consumption between the terminal and the satellite.

[0075]

[0076] in, represents the transmission energy consumption between terminal n and the satellite, is the time it takes for terminal n to upload data to the satellite, Indicates whether terminal n uploads data through channel k in time slot t. If Indicates that terminal n uploads data through channel k in time slot t. If It indicates that terminal n did not upload data through channel k in time slot t. is the transmit power of terminal n in channel k in time slot t, is the power consumption of terminal n in addition to the transmission power, a is a constant factor, and K is the total number of channels between the terminal and the satellite.

[0077] 3. Processing energy consumption of cloud computing centers.

[0078]

[0079] in, represents the processing energy consumption of the cloud computing center in time slot t, γ2 is a constant factor determined by the processing capacity of the cloud computing center, is the processing time of the data allocated by the cloud computing center to terminal n in time slot t, and its formula is as follows:

[0080]

[0081] Among them, k0 is the computing resources consumed by processing unit bit tasks, is the computing resource allocated by the cloud computing center to terminal n in time slot t, v n (t) represents the amount of data uploaded by terminal n to the cloud computing center in time slot t, and the formula is as follows:

[0082]

[0083] Where W is the channel bandwidth, K. Please refer to the previous discussion for the meaning of σ, which will not be repeated here. 2 is the Gaussian white noise power, is the channel gain between terminal n and satellite on channel k at time slot t.

[0084] The total energy consumption of the communication system in time slot t is:

[0085]

[0086] Wherein, N represents the total number of terminals. is the local processing energy consumption of terminal n in time slot t, is the transmission energy consumption between terminal n and the satellite in time slot t, is the processing energy consumption of the cloud computing center in time slot t. C(t)=[C1(t),C2(t),…,C N (t)] represents the channel allocation matrix, is the channel selection vector of terminal n, P(t)=[P1(t),P2(t),…,P N (t)] represents the transmit power matrix of N terminals, is the transmit power vector of terminal n in K channels.

[0087] Therefore, the resource allocation problem is transformed into a wireless channel resource and power allocation problem. In order to minimize the energy consumption of the communication system, the optimization goal of the model is: The constraints require that the terminal queue is stable, each terminal selects a single channel in a single time slot, and the terminal's transmit power does not exceed the power threshold.

[0088] In one embodiment of the present invention, core network elements can use traditional neural networks, such as BP networks, to build a resource allocation model. Considering the highly dynamic environment of satellite Internet, its resource allocation process is also dynamic. Therefore, in one embodiment of the present invention, core network elements can also use Deep Q Networks (DQNs) to build a resource allocation model, transforming the process of minimizing the total energy consumption of the communication system into a DQN reinforcement learning process. The three elements of the DQN algorithm are set as follows:

[0089] State space S: The state information at time slot t is s(t) = [H(t), Qe(t)], s(t)∈S. Where H(t) = [h1(t), h2(t), …, h N (t)] represents the channel matrix of all terminals in time slot t. represents the channel state vector of terminal n. The terminal executes the computing task locally in a first-in-first-out manner, Qe(t)={Qe n(t), n = 1, 2, ..., N} represents the task queue received by terminal n in time slot t, and its expression is:

[0090] Qe(t)=max{Qe(t-1)+B n (t)-v n (t)-D n (t),0}

[0091] Where Qe(t-1) represents the task queue received by terminal n in the previous time slot t-1. n (t), v n (t), D n The meaning of (t) is as described above and will not be repeated here. n (t) represents the amount of task data received by terminal n in time slot t, which follows Poisson distribution.

[0092] Action space A: a(t) = {a n (t)},a(t)∈A,a n (t)=[C n (t),P n (t)]. Where C n (t) is the transmission channel selected by terminal n in time slot t, P n (t) is the transmit power selected by terminal n in time slot t. Discretizing the continuous transmit power is beneficial to the convergence of the deep reinforcement learning algorithm DQN.

[0093] Reward function: R(t) = -E total (t) or Among them, E total (t) is the total energy consumption of the communication system in time slot t.

[0094] The process of training a resource allocation model using the DQN algorithm involves preset experience replay data, which includes multiple data items in consecutive time slots, each of which includes current time slot state information, target action, reward, and next time slot state information. For example, a piece of data is {s(t), a(t), R(t), s(t+1)}, where s(t) represents the current time slot state information, a(t) represents the target action, R(t) represents the reward, and s(t+1) represents the next time slot state information. It should be noted that the preset experience replay data can be historical data obtained from multiple second terminals, or historical data related to multiple second terminals obtained from core network elements, and this application does not limit this. The so-called multiple second terminals refer to terminals in the communication system used to assist in training the resource allocation model during the DQN algorithm training process. After the resource allocation model training is completed, the above-mentioned multiple second terminals can also serve as the first terminal in the embodiment of this application, accepting the control of the above-mentioned trained resource allocation model during resource allocation.

[0095] The reward is negatively correlated with the total energy consumption of the communication system when multiple second terminals perform the target action under the current time slot state information. For example, R(t) = -E total (t) or E total (t) represents the total energy consumption of the communication system when multiple second terminals perform the target action under the current time slot state information. The target action includes the multiple second terminals selecting corresponding transmit channels and corresponding transmit powers. For example, a(t) = [C(t), P(t)], where C(t) represents the channel allocation matrix of the multiple second terminals, and P(t) represents the transmit power matrix of the multiple second terminals. The specific formulas for C(t) and P(t) are discussed above and are not repeated here.

[0096] Taking into account the strong correlation between multiple pieces of experience replay data, which will lead to unstable network training, therefore, in one embodiment of the present invention, the core network element can randomly sample multiple pieces of training data from the pre-stored experience replay data, train the resource allocation model based on the multiple pieces of training data, and obtain a trained resource allocation model.

[0097] In the embodiment of the present application, random sampling is used to break the association between data, ensuring that the training data has sufficient diversity. In this way, samples from different time slots are randomly used when updating the network parameters at the current moment, allowing the network to learn meaningful weights.

[0098] In another embodiment of the present invention, the resource allocation model includes a first network and a second network with identical structures. The first network is also referred to as the target Q network, and the second network is also referred to as the primary Q network. After obtaining a plurality of training data through random sampling, the core network element trains the resource allocation model in the manner described in S1.1-S1.4.

[0099] S1.1. Input the current time slot state information and target action of the first training data into the first network to obtain a first result.

[0100] The first training data refers to any one of the multiple training data. In other words, the multiple training data are input into the first network for training. The first result represents the predicted value, i.e., the predicted Q value, of the multiple second terminals executing the target action of the first training data under the current time slot state information of the first training data.

[0101] S1.2. Input the next time slot state information of the first training data into the second network to obtain a second result.

[0102] The second result represents the maximum value of multiple values ​​of multiple actions performed by multiple second terminals under the next time slot state information of the first training data, where the multiple actions are actions in a pre-established action space A. The meaning of action space A is discussed above and is not repeated here.

[0103] S1.3. Update the parameters of the first network according to the error between the first result and the third result.

[0104] Specifically, the core network element first calculates a third result. The third result represents the actual value of multiple second terminals performing the target action of the first training data under the current time slot state information of the first training data, i.e., the target Q value. The third result is positively correlated with the reward of the first training data and the second result. The core network element then calculates the error between the first and third results, and uses this error to update the parameters of the first network using a gradient descent algorithm.

[0105] For example, the formula for the third result is as follows:

[0106] Target_Q_Value=R1+γmax(q1,q2,…,q n)

[0107] Among them, Target_Q_Value represents the third result, R1 represents the reward of the first training data, max(q1,q2,…,q n) Indicates the second result, q n) represents the Q value of executing the nth action, and γ represents the attenuation coefficient.

[0108] The error formula is as follows:

[0109] Loss=MSE(Predict_Q_value-Target_Q_Value)

[0110] Among them, Target_Q_Value represents the third result, and Predict_Q_value represents the first result.

[0111] S1.4. When the number of iterations of the first network is an integer multiple of a preset number, the parameters of the first network are assigned to the second network until the number of assignments of the second network reaches a preset threshold, and the second network is used as the trained resource allocation model.

[0112] In an embodiment of the present application, the resource allocation model includes a first network and a second network with the same structure. The parameters of the second network are kept unchanged for a period of time to ensure that the second result output by the second network remains stable in a short period of time, making the training process more stable. When the number of assignments of the second network reaches a preset threshold, the parameters learned by the first network are assigned to the second network, so that the second network also improves and can output a more accurate second result.

[0113] As described above, how to train the resource allocation model is introduced. In one embodiment of the present invention, the experience replay data is obtained in the following manner.

[0114] The core network element can determine target actions corresponding to multiple time slot state information from multiple actions according to a preset strategy, execute the target actions corresponding to the multiple time slot state information under the multiple time slot state information, obtain the next time slot state information corresponding to the multiple time slot state information, and determine the rewards corresponding to the multiple time slot state information based on the total energy consumption of the communication system. The multiple time slot state information, the target actions corresponding to the multiple time slot state information, the rewards corresponding to the multiple time slot state information, and the next time slot state information corresponding to the multiple time slot state information are then saved to obtain experience replay data.

[0115] The preset strategy can be to select the action that maximizes the Q value, or to randomly select an action. For example, the ε-greedy strategy sets a random probability p. If p is greater than ε, the action that maximizes the Q value is selected. If p is less than ε, an action is randomly selected from the action space A.

[0116] For example, at the initial moment, the state information of the initialized system is S(t), the target action is determined to be a(t) according to the preset strategy, the target action a(t) is executed under the state information S(t), and the next state information S(t+1) and reward reward are obtained. {S(t), a(t), reward, S(t+1)} is regarded as an experience data and stored in the experience replay space. And so on, the data is replayed according to the obtained experience.

[0117] In summary, existing cloud-network integration cannot achieve ubiquitous connectivity and global coverage. The embodiments of the present application combine satellite Internet with the concept of cloud-network integration to bring cloud computing capabilities to any location in the world. Existing resource allocation technologies mainly focus on the allocation of computing resources and storage resources. The embodiments of the present application address the core issue of limited wireless resources in satellite Internet and jointly consider the allocation of wireless resources and computing resources. Traditional resource allocation methods such as convex optimization and game theory are not intelligent and dynamic enough and are not suitable for highly dynamic satellite Internet environments. Therefore, the embodiments of the present application use a deep reinforcement learning algorithm to dynamically allocate wireless resources for satellite Internet. By informing the terminal of the correct channel and transmit power through signaling, it efficiently utilizes wireless resources while avoiding channel conflicts and optimizing system energy consumption.

[0118] Based on the same inventive concept, the present application also provides a resource allocation device, which is arranged in a core network element of a communication system, and the communication system also includes a terminal, a satellite and a cloud computing center, please refer to Figure 3 , the device comprises:

[0119] The information acquisition module 301 is configured to acquire status information of multiple first terminals through requests from the multiple first terminals; wherein the status information includes task queues received by the multiple first terminals and transmission channels selectable by the multiple first terminals;

[0120] A policy decision module 302 is configured to input the state information into a trained resource allocation model to obtain a resource allocation policy; wherein the trained resource allocation model is trained with minimizing the total energy consumption of the communication system as an optimization objective, and the resource allocation policy is configured to indicate transmit channels and transmit powers corresponding to the plurality of first terminals;

[0121] The sending module 303 is configured to send a signaling including a resource allocation strategy to the plurality of first terminals via a satellite, so that the plurality of first terminals send their respective received task queues to the cloud computing center according to the resource allocation strategy.

[0122] In one embodiment of the present invention, the total energy consumption of the communication system includes the processing energy consumption of the multiple first terminals, the transmission energy consumption between the multiple first terminals and the satellite, and the processing energy consumption of the cloud computing center.

[0123] In another embodiment of the present invention, the apparatus further includes a model training module 304, which is configured to:

[0124] Before inputting the state information into the trained resource allocation model to obtain the resource allocation strategy, a plurality of training data are randomly sampled from pre-stored experience replay data; wherein the experience replay data includes a plurality of data pieces in consecutive time slots, each data piece includes current time slot state information, a target action, a reward, and next time slot state information, wherein the reward is negatively correlated with the total energy consumption of the communication system when the plurality of second terminals execute the target action under the current time slot state information, and the target action includes the plurality of second terminals selecting corresponding transmission channels and corresponding transmission powers;

[0125] The resource allocation model is trained based on the plurality of training data to obtain a trained resource allocation model.

[0126] In another embodiment of the present invention, the resource allocation model includes a first network and a second network with the same structure; the model training module 304 is specifically configured to:

[0127] Inputting the current time slot state information and the target action of the first training data into the first network to obtain a first result; wherein the first result represents the predicted value of the plurality of second terminals executing the target action of the first training data under the current time slot state information of the first training data;

[0128] Inputting the next time slot state information of the first training data into the second network to obtain a second result; wherein the second result represents a maximum value among multiple values ​​of multiple second terminals performing multiple actions under the next time slot state information of the first training data, where the multiple actions are actions in a pre-established action space;

[0129] updating the parameters of the first network based on the error between the first result and the third result; wherein the third result represents the actual value of the plurality of second terminals executing the target action of the first training data under the current time slot state information of the first training data, and the third result is positively correlated with the reward of the first training data and the second result;

[0130] When the number of iterations of the first network is an integer multiple of a preset number, the parameters of the first network are assigned to the second network until the number of assignments of the second network reaches a preset threshold, and the second network is used as the trained resource allocation model.

[0131] In another embodiment of the present invention, the model training module 304 is further configured to:

[0132] Before randomly sampling multiple training data from pre-stored experience playback data, target actions corresponding to multiple time slot state information are determined from multiple actions according to a preset strategy;

[0133] Executing target actions corresponding to the multiple time slot state information under the multiple time slot state information, obtaining next time slot state information corresponding to the multiple time slot state information, and determining rewards corresponding to the multiple time slot state information based on the total energy consumption of the communication system;

[0134] Save multiple time slot state information, target actions corresponding to the multiple time slot state information, rewards corresponding to the multiple time slot state information, and next time slot state information corresponding to the multiple time slot state information to obtain experience replay data.

[0135] It should be noted that although several modules or submodules of the device are mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more modules described above can be embodied in one module. Conversely, the features and functions of one module described above can be further divided and embodied by multiple modules.

[0136] It should be noted that Figure 3 The middle device can also be used to implement the resource allocation method discussed above, which will not be described here in detail.

[0137] Based on the same inventive concept, an electronic device is also provided in the embodiment of the present application. The device is equivalent to the core network element discussed above. Please refer to Figure 4 , the device includes a processor 401 and a memory 402.

[0138] Memory 402, for storing program instructions;

[0139] The processor 401 is configured to call the program instructions stored in the memory 402 and execute the resource allocation method described above according to the obtained program instructions. The processor 401 may also implement Figure 3 The functions of each module in the device shown.

[0140] The embodiment of the present application does not limit the specific connection medium between the processor 401 and the memory 402. Figure 4 In the example, the processor 401 and the memory 402 are connected via a bus 400. Figure 4 The connections between other components are shown in bold. The bus 400 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, Figure 4 The diagram is represented by only one thick line, but this does not mean that there is only one bus or one type of bus. Alternatively, the processor 401 may also be referred to as a controller, without limitation to the name.

[0141] Among them, the processor 401 is the control center of the device, which can use various interfaces and lines to connect the various parts of the entire device, and monitor the device as a whole by running or executing instructions stored in the memory 402 and calling data stored in the memory 402, the various functions of the device and processing data.

[0142] In one design, processor 401 may include one or more processing units. Processor 401 may integrate an application processor and a modem processor. The application processor primarily processes the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 401. In some embodiments, processor 401 and memory 402 may be implemented on the same chip. In some embodiments, they may also be implemented on separate chips.

[0143] The processor 401 can be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and can implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the resource allocation method disclosed in the embodiments of the present application can be directly embodied as being executed by a hardware processor, or can be executed by a combination of hardware and software modules in the processor.

[0144] The memory 402 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 402 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, etc. The memory 402 is any other medium that can be used to carry or store a desired program code in the form of an instruction or data structure and can be accessed by a computer, but is not limited thereto. The memory 402 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0145] By designing and programming the processor 401, the code corresponding to the resource allocation method described in the above embodiment can be fixed into the chip, so that the chip can execute the code when running. Figure 3 How to design and program the processor 401 is a technique well known to those skilled in the art and will not be described in detail here.

[0146] Based on the same inventive concept, embodiments of the present application provide a computer-readable storage medium storing a computer program. The computer program includes program instructions. When executed by a computer, the program instructions cause the computer to perform any of the resource allocation methods discussed above. Because the principles underlying the problem solved by the computer-readable storage medium are similar to those of the resource allocation method, the implementation of the computer-readable storage medium can be referred to as the implementation of the method, and any repetitions will not be repeated.

[0147] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0148] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0149] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0150] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0151] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. These changes and modifications fall within the scope of the claims of the present application and their equivalents.

Claims

1. A resource allocation method, characterized in that: In a core network element of a communication system, the communication system further includes a terminal, a satellite, a gateway, and a cloud computing center, wherein the terminal and the satellite, and the satellite and the gateway are communicatively connected, and the gateway is communicatively connected to the cloud computing center via the core network element. The method includes: Acquiring status information of the plurality of first terminals through requests of the plurality of first terminals; wherein the status information includes: task queues received by the plurality of first terminals, which are unoccupied and used to provide selected transmission channels for the plurality of first terminals; Inputting the state information into a trained resource allocation model to obtain a resource allocation strategy; wherein the trained resource allocation model is obtained by training the resource allocation model with minimizing the total energy consumption of the communication system as an optimization goal, and the resource allocation strategy is used to indicate the transmission channels and transmission powers corresponding to the multiple first terminals; The signaling including the resource allocation strategy is sent to the multiple first terminals via the satellite, so that the multiple first terminals send their respective received task queues to the cloud computing center according to the resource allocation strategy.

2. The method according to claim 1, wherein The total energy consumption of the communication system includes processing energy consumption of the multiple first terminals, transmission energy consumption between the multiple first terminals and the satellite, and processing energy consumption of the cloud computing center.

3. The method according to claim 1 or 2, wherein: Before inputting the state information into the trained resource allocation model to obtain the resource allocation strategy, the method further includes: A plurality of training data are obtained by randomly sampling from pre-stored experience replay data; wherein the experience replay data includes a plurality of data pieces in consecutive time slots, each data piece includes current time slot state information, a target action, a reward, and next time slot state information, the reward being negatively correlated with the total energy consumption of the communication system when a plurality of second terminals execute the target action under the current time slot state information, and the target action includes the plurality of second terminals selecting corresponding transmission channels and corresponding transmission powers; The resource allocation model is trained based on the multiple training data to obtain the trained resource allocation model.

4. The method according to claim 3, wherein The resource allocation model includes a first network and a second network having the same structure; Training the resource allocation model based on the plurality of training data to obtain the trained resource allocation model includes: Inputting the current time slot state information and the target action of the first training data into the first network to obtain a first result; wherein the first result represents the predicted value of the plurality of second terminals performing the target action of the first training data under the current time slot state information of the first training data; Inputting the next time slot state information of the first training data into the second network to obtain a second result, wherein the second result represents a maximum value among multiple values ​​of the multiple second terminals performing multiple actions under the next time slot state information of the first training data, the multiple actions being actions in a pre-established action space; updating the parameters of the first network based on an error between the first result and a third result; wherein the third result represents actual value of the plurality of second terminals executing the target action of the first training data under the current time slot state information of the first training data, and the third result is positively correlated with the reward of the first training data and the second result; When the number of iterations of the first network is an integer multiple of a preset number, the parameters of the first network are assigned to the second network until the number of assignments of the second network reaches a preset threshold, and the second network is used as the trained resource allocation model.

5. The method according to claim 4, wherein Before randomly sampling from pre-stored experience playback data to obtain a plurality of training data, the method further includes: Determining target actions corresponding to multiple time slot state information from the multiple actions according to a preset strategy; executing target actions corresponding to the multiple time slot state information respectively under the multiple time slot state information, obtaining next time slot state information corresponding to the multiple time slot state information respectively, and determining rewards corresponding to the multiple time slot state information respectively based on the total energy consumption of the communication system; Save the multiple time slot state information, the target actions corresponding to the multiple time slot state information, the rewards corresponding to the multiple time slot state information, and the next time slot state information corresponding to the multiple time slot state information to obtain the experience replay data.

6. A resource allocation device, characterized in that: The device is arranged in a core network element of a communication system, the communication system further comprising a terminal, a satellite, a gateway station, and a cloud computing center, the terminal and the satellite are communicatively connected, and the satellite and the gateway station are communicatively connected to the cloud computing center via the core network element, and the device comprises: An information acquisition module, configured to acquire, through requests from the plurality of first terminals, status information of the plurality of first terminals; wherein the status information includes: task queues received by the plurality of first terminals, which are unoccupied and used to provide transmission channels selected by the plurality of first terminals; a policy decision module, configured to input the state information into a trained resource allocation model to obtain a resource allocation policy; wherein the trained resource allocation model is trained with minimizing the total energy consumption of the communication system as an optimization goal, and the resource allocation policy is used to indicate the transmission channels and transmission powers corresponding to the multiple first terminals; The sending module is used to send the signaling containing the resource allocation strategy to the multiple terminals via the satellite, so that the multiple first terminals send their respective received task queues to the cloud computing center according to the resource allocation strategy.

7. The device according to claim 6, characterized in that The total energy consumption of the communication system includes processing energy consumption of the multiple first terminals, transmission energy consumption between the multiple first terminals and the satellite, and processing energy consumption of the cloud computing center.

8. The device according to claim 6 or 7, characterized in that The device further includes a model training module, which is configured to: Before inputting the state information into a trained resource allocation model to obtain a resource allocation strategy, training data is obtained by randomly sampling multiple pieces of pre-stored experience replay data; wherein the multiple pieces of data are continuous time slot data, each piece of data includes current time slot state information, a target action, a reward, and next time slot state information, the reward is negatively correlated with the total energy consumption of the communication system when multiple second terminals perform the target action under the current time slot state information, and the target action includes the multiple second terminals selecting corresponding transmission channels and corresponding transmission powers; The resource allocation model is trained based on the training data to obtain the trained resource allocation model.

9. An electronic device, characterized in that: include: a memory for storing program instructions; A processor is configured to call the program instructions stored in the memory and execute the method according to any one of claims 1 to 5 according to the obtained program instructions.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a computer, the computer is caused to perform the method according to any one of claims 1 to 5.

Images