Task allocation model training method, inter-satellite task allocation method, system, device and medium

Abstract

The application relates to the technical field of data processing, and discloses a training method of a task allocation model, an inter-satellite task allocation method, a system, equipment and a medium. The method comprises the following steps: determining the action of a current time slot of a first satellite through an action network according to the state of the current time slot of the first satellite; determining the reward obtained by the first satellite for performing the action of the current time slot and the state of a next time slot according to the action of the current time slot of the first satellite; determining the income of the next time slot of the first satellite through a value network according to the state of the next time slot of the first satellite; after a preset number of time slots are traversed, obtaining the income of the current time slot according to the income of the next time slot of the first satellite and the reward of the current time slot; and training and optimizing the action network and the value network based on the income of the current time slot and the value prediction value of the current state of the value network, so as to obtain the optimized action network and the value network. The application improves the resource utilization rate of the satellite.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a training method for a task allocation model, a method for inter-satellite task allocation, a system, equipment, and a medium. Background Technology

[0002] Existing internet resource scheduling technologies mainly employ static resource allocation and rule-based dynamic scheduling methods. Static resource allocation methods pre-allocate fixed resources such as bandwidth and time slots to different services. Rule-based dynamic scheduling methods, on the other hand, allocate resources using preset rules (such as priority scheduling and round-robin scheduling).

[0003] Existing static task allocation and rule-based dynamic scheduling methods cannot intelligently orchestrate tasks and schedule resources according to the real-time business needs of the power grid, resulting in low resource utilization. In particular, when there is a sudden increase in business demand in the power grid, the problem of uneven allocation of satellite resources is likely to occur. Summary of the Invention

[0004] In view of this, this application provides a training method for a task allocation model, a method for task allocation between satellites, a system, equipment and medium, which improves the resource utilization rate of satellites.

[0005] This application discloses a training method for a task allocation model, which includes an action network and a value network, comprising: Based on the current time slot status of the first satellite, the action of the current time slot of the first satellite is determined through the action network, wherein the first satellite is any satellite in the satellite communication system; Based on the actions of the first satellite in the current time slot, determine the reward obtained by the first satellite in performing the actions in the current time slot and the state of the next time slot; Based on the status of the next time slot of the first satellite, the revenue of the next time slot of the first satellite is determined through the value network; After traversing a preset number of time slots, the revenue of the current time slot is obtained based on the revenue of the next time slot of the first satellite and the reward of the current time slot; Based on the current time slot's revenue and the value network's prediction of the current state's value, the action network and the value network are trained and optimized to obtain optimized action networks and value networks.

[0006] Furthermore, based on the actions of the first satellite in the current time slot, the reward obtained by the first satellite in performing the actions in the current time slot and the state of the next time slot are determined, including: Based on the total delay determined after the first satellite performs the action in the current time slot, a corresponding delay reward is determined. The total delay is the total time from when the task to be assigned begins to wait for transmission in the first satellite until the second satellite starts calculation. The delay reward is the reward obtained by the first satellite in the current time slot when it performs the selected action. The second satellite is any satellite in the satellite communication system other than the first satellite. Based on the energy consumption of the task waiting for transmission in the first satellite and the energy consumption of transmission to the second satellite, a corresponding energy reward is determined; the energy reward is the reward corresponding to the energy consumption of the task from the start of waiting for transmission in the first satellite to the point of reaching the second satellite. Based on the delay reward and the energy reward, determine the reward obtained by the first satellite for performing the action in the current time slot; After the first satellite performs the action in the current time slot, it updates its task queue to obtain the status of the next time slot.

[0007] Furthermore, based on the total delay determined after the first satellite performs its action in the current time slot, a corresponding delay reward is determined, including: Based on the current actions performed by the first satellite, determine the tasks to be sent to the second satellite. Based on the determined tasks to be assigned, the total number of tasks to be assigned is obtained, and multiple total delays corresponding to the total number are determined, wherein each task to be assigned corresponds to a total delay. Calculate the absolute values ​​of multiple total delays, and select the maximum value as the delay reward obtained by the first satellite for performing the action in the current time slot.

[0008] Further, determining the corresponding energy reward based on the energy consumption of the task awaiting transmission in the first satellite and the energy consumption of transmission to the second satellite includes: The energy consumption of all tasks awaiting assignment in the first satellite and waiting to be transmitted to the second satellite is accumulated, and the negative of the accumulated result is used as the energy reward obtained by the first satellite for performing the action in the current time slot.

[0009] Further, based on the latency reward and the energy reward, the reward obtained by the first satellite for performing the action in the current time slot is determined, including: The reward obtained by the first satellite in performing the action in the current time slot is determined based on the weighted sum of the delay reward and the energy reward.

[0010] Further, determining the total delay includes: Based on the length of the task to be assigned on the first satellite and the transmission rate between the first and second satellites, the transmission waiting delay of the task to be assigned on the first satellite is determined; the transmission waiting delay refers to the time that the task to be assigned waits on the first satellite to be sent to the second satellite. Based on the data size of the task to be assigned and the transmission rate between the first and second satellites, the transmission delay of the task to be assigned from the first satellite to the second satellite is determined. Based on the length of the task to be calculated on the second satellite and the calculation time required for the task to be assigned, the calculation waiting delay of the task to be assigned on the second satellite is determined; the calculation waiting delay refers to the queuing time for the task to be assigned while waiting for the second satellite to perform calculations on it. The total delay is determined based on the transmission waiting delay, the transmission delay, and the calculation waiting delay.

[0011] Further, determining the total delay based on the transmission waiting delay, the transmission delay, and the calculated waiting delay includes: The total delay is the sum of the transmission waiting delay and the calculation waiting delay.

[0012] Further, obtaining the energy consumption includes: Based on the length of the task to be assigned on the first satellite, the amount of data in the task to be assigned, the transmission rate between the first and second satellites, and the transmission power of the first satellite, the energy consumption of the task to be assigned from the time it starts waiting for transmission on the first satellite to the time it arrives at the second satellite is determined.

[0013] Furthermore, obtaining the transmission rate between the first satellite and the second satellite includes: The transmission rate between the first and second satellites is obtained based on the available bandwidth of the route between the first and second satellites, the signal transmission power of the first satellite, the gain of the transmitting antenna of the first satellite, the gain of the receiving antenna of the second satellite, the path loss between the first and second satellites, the noise bandwidth, and the quantum efficiency of the photodetector.

[0014] Furthermore, based on the state of the first satellite's next time slot, the revenue of the first satellite's next time slot is determined through a value network, including: Based on the state corresponding to the first satellite target time slot, the value prediction value corresponding to that state is determined through a value network; the target time slot is the sum of the next time slot and a preset number; The value network determines the revenue of the first satellite in the next time slot based on the rewards of the preset number of time slots and the value prediction value corresponding to the target time slot.

[0015] Furthermore, the value network determines the revenue of the first satellite in the next time slot based on the rewards of the preset number of time slots and the predicted value corresponding to the target time slot, including: The value network weights and sums the rewards for the preset number of time slots starting from the next time slot with the predicted value corresponding to the target time slot, and accumulates the weighted results to obtain the revenue of the first satellite in the next time slot.

[0016] Furthermore, the training and optimization of the action network and value network includes: Determine the loss function based on the revenue of the current time slot and the predicted value; By minimizing the loss function, the network parameters of the action network and the value network are updated, thus completing the training optimization of the action network and the value network.

[0017] This application also discloses a method for inter-satellite mission allocation, which includes: The first satellite loads the task allocation model, where the first satellite is any satellite in the satellite communication system; The current state of the first satellite is input into the task allocation model to obtain the target action; the state is the task queue of the first satellite, and the target action is used to determine the task to be allocated from the task queue. The first satellite transmits the task to be assigned to the second satellite, which is any satellite in the satellite communication system other than the first satellite.

[0018] Furthermore, the task queue includes all tasks to be processed by the first satellite; the target action is the action corresponding to the maximum benefit among the benefits corresponding to a preset number of time slots.

[0019] Furthermore, the tasks to be processed by the first satellite originate from power grid equipment; Based on the location of the power grid equipment and the motion trajectory of each first satellite in the satellite communication system, the first satellite closest to the power grid equipment is determined, and the power grid equipment sends its mission to the first satellite.

[0020] This application also discloses an inter-satellite mission allocation system, which includes: The model loading module is used to load the task allocation model for the first satellite, which is any satellite in the satellite communication system. The task allocation module is used to input the current state of the first satellite into the task allocation model to obtain the target action; the state is the task queue of the first satellite, and the target action is used to determine the task to be allocated from the task queue; The task sending module is used by the first satellite to send the task to be assigned to the second satellite, wherein the second satellite is any satellite in the satellite communication system other than the first satellite.

[0021] This application also discloses an electronic device including a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program in the memory to implement the method described above.

[0022] This application also discloses a computer-readable storage medium storing a computer program or instructions that, when executed on a computer, cause the computer to perform the methods described above.

[0023] Due to the adoption of the above technical solution, this application has the following advantages: This application improves the resource utilization rate of the satellite. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0025] Figure 1 This is a flowchart illustrating a training method for a task allocation model according to an embodiment of this application. Figure 2 This is a schematic diagram of the task allocation model training process in an embodiment of this application; Figure 3 This is a flowchart illustrating an inter-satellite task allocation method according to an embodiment of this application; Figure 4 This is a flowchart illustrating another method for inter-satellite task allocation according to an embodiment of this application; Figure 5 This is a schematic diagram illustrating an application scenario of an inter-satellite task allocation method according to an embodiment of this application; Figure 6 This is a block diagram of an inter-satellite task allocation system according to an embodiment of this application; Figure 7 This is a block diagram of an electronic device according to an embodiment of this application. Detailed Implementation

[0026] The present application will be further described in conjunction with the accompanying drawings and embodiments. The described embodiments are only some, not all, of the embodiments of the present application. All other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of the present application.

[0027] See Figure 1 This application provides an embodiment of a training method for a task allocation model, wherein the task allocation model includes an action network and a value network, and the method includes: Step 101: Based on the current time slot status of the first satellite, determine the action of the current time slot of the first satellite through the action network.

[0028] The first satellite is any one of the satellites in the satellite communication system, which includes multiple satellites.

[0029] In one embodiment of this application, the Actor network shown below is based on... Output Action ,in These are the current parameters of the Actor network. Assign a strategy output function to the task. This represents the current state of the satellite in time slot t. The current state of the satellite in the current time slot t The action chosen below.

[0030] Step 102: Based on the actions of the first satellite in the current time slot, determine the reward obtained by the first satellite in performing the actions in the current time slot and the status of the next time slot.

[0031] In one embodiment of this application, determining the reward obtained by the first satellite in performing the action in the current time slot and the state of the next time slot based on the action of the first satellite in the current time slot includes: Based on the total delay determined after the first satellite performs the action in the current time slot, the corresponding delay reward is determined. The total delay is the total time from when the task to be assigned begins to wait for transmission in the first satellite until the second satellite starts calculation. The delay reward is the reward obtained by the first satellite in the current time slot when it performs the selected action. The second satellite is any satellite in the satellite communication system other than the first satellite. The corresponding energy reward is determined based on the energy consumption of the task waiting for transmission in the first satellite and the energy consumption of transmission to the second satellite. The energy reward is the reward corresponding to the energy consumption of the task from the start of waiting for transmission in the first satellite until it reaches the second satellite. The reward obtained by the first satellite in performing the action in the current time slot is determined based on the time delay reward and the energy reward. After the first satellite performs its actions in the current time slot, it updates its task queue to obtain the status of the next time slot.

[0032] In one embodiment of this application, a corresponding delay reward is determined based on the total delay determined after the first satellite performs its action in the current time slot, including: Based on the current actions performed by the first satellite, determine the tasks to be sent to the second satellite. Based on the identified tasks to be assigned, obtain the total number of tasks to be assigned and determine multiple total delays corresponding to this total number, where each task to be assigned corresponds to a total delay. Calculate the absolute values ​​of multiple total delays, and select the maximum value as the delay reward obtained by the first satellite for performing the action in the current time slot.

[0033] Optionally, based on the current action performed by the first satellite, the task allocation model outputs an allocation score for each computational task in the task queue, sorts them from high to low according to the allocation scores, and selects the computational tasks at the top to determine the tasks to be sent to the second satellite.

[0034] In one embodiment of this application, the corresponding energy reward is determined based on the energy consumption of the task awaiting transmission in the first satellite and the energy consumption of transmission to the second satellite, including: The energy consumption of all tasks awaiting assignment in the first satellite waiting to be transmitted to the second satellite is accumulated, and the negative of the accumulated result is used as the energy reward obtained by the first satellite for performing the action in the current time slot.

[0035] In one embodiment of this application, the reward obtained by the first satellite for performing the action in the current time slot is determined based on the delay reward and the energy reward, including: The reward obtained by the first satellite for performing the action in the current time slot is determined by the weighted sum of the time delay reward and the energy reward.

[0036] In one possible implementation of this application, the resource allocation orchestration problem is formulated as a deep reinforcement learning process, which focuses on the optimal strategy for offloading grid services from satellite internet computation. Since the state of the entire system (the satellite internet resource allocation system, which consists of satellite communication systems and terrestrial grid equipment) in the current time slot is only affected by its state in the previous time slot, and the state of the previous time slot exhibits Markov properties, the state change of the entire system can be formulated as a Markov decision process, which can be represented using tuples. express. Represents the system's state set. Refers to the system's action set. Represents the probability of state transition. Let represent the reward function. Since the state transition probabilities and rewards cannot be predicted in advance in the environment where the unloading policy is computed, an asynchronous distributed reinforcement learning method is proposed to address this problem.

[0037] Specifically, each satellite in the entire satellite communication system will be in a time slot The state is represented as This includes the status information of each satellite in the satellite internet, that is, the task queue maintained by each satellite, representing the actions that each satellite can take in a time slot in the entire satellite communication system as... It describes how each satellite will manage its queue, that is, select some tasks to be assigned to another satellite in the satellite communication system.

[0038] Optionally, the reward function used in the deep reinforcement learning model (task allocation model) primarily focuses on the maximum latency for each satellite in the entire satellite communication system and the total transmission energy consumption resulting from task allocation. Because the goal is to maximize the utilization of resources in the entire satellite communication system and avoid some satellites being idle while others are overloaded, the focus is primarily on the task with the highest completion latency. Furthermore, it is necessary to control the data transmission energy cost incurred during task allocation. However, the total energy cost of task computation does not change due to resource allocation; only the computational component changes, so this embodiment does not consider it.

[0039] Ultimately, the designed reward function Includes latency rewards and energy rewards Regarding latency rewards, in order to ensure that the resources of the entire satellite communication system are fully utilized and that each power grid service is completed as quickly as possible, the latency reward depends on the service that is completed last.

[0040]

[0041] in, Indicates the satellite in the current time slot t satellite communication system (First Satellite) Select Action The execution reward and latency reward depend on the satellite. Tasks in its task queue From satellite The satellite is waiting to transmit data to the satellite communication system. (The second satellite) began calculating the mission. Total required latency The maximum value of i; the range of i is from 1 to M, where M is a positive integer, and M is the task value. The total number, i.e., the total number of tasks to be assigned. Actions Used to indicate the current state Next satellite A task to be assigned to a satellite in a satellite communication system.

[0042] Regarding energy rewards Its expression is:

[0043] The above formula shows that for satellites The energy consumption corresponding to the M tasks to be assigned is accumulated.

[0044] Therefore, the final system reward function can be represented by a weighted average, which is the current time slot t satellite. Select Action The corresponding total reward function is set as follows:

[0045] in, Indicates the satellite selection action in the current time slot t. Total reward for execution (the reward obtained by the satellite for performing the action in the current time slot). and These are the corresponding weighting coefficients.

[0046] In one embodiment of this application, determining the total delay includes: Based on the length of the task to be assigned on the first satellite and the transmission rate between the first and second satellites, the transmission waiting delay of the task to be assigned in the first satellite is determined; the transmission waiting delay refers to the time that the task to be assigned waits in the first satellite to be sent to the second satellite. Based on the data size of the task to be assigned and the transmission rate between the first and second satellites, the transmission delay of the task to be assigned from the first satellite to the second satellite is determined. The computation waiting delay of the task to be assigned on the second satellite is determined based on the length of the task to be computed on the second satellite and the computation time required for the task to be assigned. The computation waiting delay refers to the queuing time for the task to be assigned while waiting for the second satellite to compute it. The total delay is determined based on the transmission waiting delay, the transmission delay, and the calculation waiting delay.

[0047] In one embodiment of this application, the total delay is determined based on the transmission waiting delay, the transmission delay, and the calculated waiting delay, including: The total delay is the sum of the transmission waiting delay, the transmission delay, and the calculation waiting delay.

[0048] In one possible implementation of this application embodiment, each task generated by the power grid equipment can be described by two metrics: the scale of the data. and the CPU time required to complete the calculation Assuming a satellite In the queue, there are a total of There are several computational tasks to be processed, and the task queue it needs to maintain can be represented as follows: , To calculate the first The time required for each computational task.

[0049] When a satellite needs to offload tasks from its mission queue to other satellites in the satellite communication system, all locally computed tasks are removed, leaving only the tasks that need to be offloaded to other satellites for computation, thus forming a new transmission queue. , For satellite Calculate the first in the sending queue The time required for each computation task is ignored because the results of edge computing are generally smaller than the original data.

[0050] Task In satellite Waiting in the queue for transmission until satellite Start calculating the task Total required latency Includes the sum of queuing delay, transmission delay, and computation delay:

[0051] in, Indicates due to the task The delay required for queuing in the transmission queue (transmission wait delay). Indicates task From satellite Transmitted to satellite Transmission delay, Indicates task Send to satellite The delay caused by queuing for calculation (calculation waiting delay). Indicates satellite The length of the sending queue, Indicates task The data size, measured in bytes. Represents computational task The required time (which can be multiple clock cycles). Indicates time slot satellite and satellite The transmission rate between them Indicates satellite Calculate the queue length. Indicates satellite CPU frequency.

[0052] In one embodiment of this application, obtaining energy consumption includes: Based on the length of the task to be assigned on the first satellite, the amount of data in the task to be assigned, the transmission rate between the first and second satellites, and the transmission power of the first satellite, the energy consumption of the task to be assigned from the time it starts waiting for transmission on the first satellite to the time it arrives at the second satellite is determined.

[0053] Based on the above possible implementation methods, the queuing and transmission process (referring to the process of transferring tasks) can be calculated. During the transmission process, the satellite Energy consumption.

[0054]

[0055] in, Indicates satellite The satellite's launch power It is to receive tasks The satellite.

[0056] In one embodiment of this application, obtaining the transmission rate between the first satellite and the second satellite includes: The transmission rate between the first and second satellites is obtained based on the available bandwidth of the route between the first and second satellites, the signal transmission power of the first satellite, the gain of the transmitting antenna of the first satellite, the gain of the receiving antenna of the second satellite, the path loss between the first and second satellites, the noise bandwidth, and the quantum efficiency of the photodetector.

[0057] In one possible implementation of this application embodiment, since the satellite's speed relative to the Earth is tens of kilometers per second, this high-speed motion will cause a Doppler frequency shift of tens of kilohertz. It is assumed that the Doppler effect between the IoT device and the low Earth orbit satellite has been eliminated in advance through automatic frequency control of the receiving system, and that existing Doppler measurement and compensation techniques are relatively easy to implement. It is also assumed that the transmitter and receiver beams are aligned, so there will be no loss due to polarization mismatch in either uplink or downlink.

[0058] Based on the above possible implementation methods, communication between satellites uses laser links. and satellite Between time slots The transmission rate is:

[0059] in, This refers to the available bandwidth of inter-satellite routing. It is a satellite The signal transmission power, and They are satellites The gain of the transmitting and receiving antennas, It refers to the quantum efficiency of photodetectors. It is Planck's constant. It is the laser frequency. It is the path loss between satellites. It is the noise bandwidth.

[0060] Step 103: Determine the revenue of the next time slot of the first satellite through the value network based on the status of the next time slot of the first satellite.

[0061] In one embodiment of this application, the revenue of the next time slot of the first satellite is determined through a value network based on the state of the next time slot, including: Based on the state corresponding to the first satellite target time slot, the value prediction value corresponding to that state is determined through a value network; the target time slot is the sum of the next time slot and a preset number; The value network determines the revenue of the first satellite in the next time slot based on the rewards of a preset number of time slots and the predicted value corresponding to the target time slot.

[0062] In one embodiment of this application, the value network determines the revenue of the first satellite in the next time slot based on the rewards of a preset number of time slots and the predicted value corresponding to the target time slot, including: The value network calculates the reward for a predetermined number of time slots starting from the next time slot by weighting and summing the reward with the value prediction value corresponding to the target time slot, and then sums the weighted results to obtain the revenue of the first satellite in the next time slot.

[0063] Based on the above possible implementation methods, the task allocation strategy output function ,in These are the parameters of the Actor network; their function is based on the Actor network parameters. and the current time slot status Output action The value function is defined as follows: Weighted returns for (preset number) states (Current time slot) (profits)

[0064] in, Indicates the Critic network's... Status of the time slot (target time slot) The projected value, express Total reward for time slots These are the parameters of the Critic network. The value range is from 0 to 1.

[0065] Based on this, the weighted return for the next time slot can be obtained. (Next time slot) (profits)

[0066] Step 104: After traversing the preset number of time slots, obtain the revenue of the current time slot based on the revenue of the next time slot of the first satellite and the reward of the current time slot.

[0067] The specific implementation process of the embodiments of this application can be found in the following description. ,in, This indicates the benefit of the satellite's current time slot. This represents the reward for the current time slot. This indicates the benefits of the satellite in the next time slot.

[0068] Step 105: Based on the current time slot's revenue and the value network's prediction of the current state's value, train and optimize the action network and value network to obtain the optimized action network and value network.

[0069] In one embodiment of this application, training and optimizing the action network and the value network includes: Determine the loss function based on the revenue of the current time slot and the predicted value; By minimizing the loss function, the network parameters of the action network and the value network are updated, thus completing the training optimization of the action network and the value network.

[0070] For possible implementations of steps 101 to 105 mentioned above, please refer to the main implementation process of the resource allocation algorithm based on the Actor-Critic network below: Input: Number of system runs Total number of time slots Weight

[0071] Network parameters Local parameters

[0072] Initialize the iteration variables for each training round

[0073] while do While iterating through all satellites before doing initialization

[0074] initialization Get the current state

[0075] while do Actor network is based on Output Action

[0076] implement Receive rewards and the next state

[0077] Output status based on a local Critci network. Benefits

[0078] Save the above data and let

[0079] end while initialization

[0080] while do Get state Benefits

[0081] According to Critic Value forecast and Calculate the loss function

[0082] end while Update the parameters of the Actor and Critic networks based on the loss function.

[0083] end while

[0084] Based on the final Actor network, the output ranges from 1 to... Optimal action for each time slot

[0085] end while Output: Final network parameters, Actor network and Critic network From multiple time slots ( In the actions corresponding to each time slot, determine the weighted return. The largest target action, which is from 1 to Optimal action for each time slot This target action is used to determine the satellite. The transmission queue, which is controlled by the satellite The tasks to be assigned consist of...

[0086] The resource allocation algorithm based on the Actor-Critic network described above only illustrates the training process of the deep reinforcement learning model for any single satellite. Figure 2 This allows for the training of deep reinforcement learning models across all satellites in the satellite network.

[0087] See Figure 3 This application provides an embodiment of an inter-satellite task allocation method, which includes: Step 201: First satellite loading task allocation model.

[0088] In this system, the first satellite can be any satellite in the satellite communication system, which includes multiple satellites. The task allocation model is trained using the method described in the preceding embodiments.

[0089] Considering the resource limitations of satellites themselves and the uneven distribution of tasks among ground-based IoT devices, some satellites may struggle to process large volumes of received power grid services quickly. Therefore, in this embodiment, a pre-trained deep reinforcement learning model (task allocation model) is deployed on the satellite. This model intelligently allocates network resources based on the current state of the satellite network (satellite communication system), enabling load balancing among satellites, improving the real-time performance and resource utilization of the entire system, and optimizing the processing efficiency of power grid services.

[0090] Step 202: Input the current state of the first satellite into the task allocation model to obtain the target action.

[0091] Among them, the task queue of the first satellite is in a certain state, and the target action is used to determine the tasks to be assigned from the task queue.

[0092] Step 203: The first satellite sends the tasks to be assigned corresponding to the total number to the second satellite.

[0093] The second satellite is any satellite in the satellite communication system other than the first satellite.

[0094] In one embodiment of this application, the task queue includes all tasks to be processed by the first satellite; the target action is the action corresponding to the maximum benefit among the benefits corresponding to a preset number of time slots.

[0095] In one embodiment of this application, the task to be processed by the first satellite originates from power grid equipment.

[0096] In one embodiment of this application, it further includes: Based on the location of the power grid equipment and the movement trajectories of each first satellite in the satellite communication system, the first satellite closest to the power grid equipment is determined, and the power grid equipment sends its mission to that first satellite.

[0097] Compared with existing static task allocation and rule-based dynamic scheduling methods, this embodiment achieves intelligent resource orchestration and scheduling through a deep reinforcement learning model. It can dynamically adjust resource allocation according to real-time business needs, significantly improve resource utilization, accelerate scheduling decision response speed, and improve the overall system computing efficiency and network robustness.

[0098] See Figure 4 In practical applications, this can also include an operation monitoring and control phase. The ground core network receives operational data from the satellite, including operational data from power grid equipment, and adjusts the status of the satellite network and power grid equipment based on the monitoring results. In the event of an emergency, the operation logs can be used to promptly determine whether the power grid equipment and satellite equipment are operating normally and decide whether to take maintenance or replacement measures.

[0099] This application's embodiments, by deploying distributed deep reinforcement learning models on various satellites, ensure that the power grid maintains efficient resource utilization and low-latency communication even when there is a sudden increase in business demand, thereby enhancing the reliability of power grid operations. Compared with traditional solutions that offload computing tasks to the terrestrial core network, this application offloads the computing tasks of power grid equipment to satellites, reducing data transmission latency and meeting the power grid's demand for low-latency communication.

[0100] Specifically, in Figure 5 In the scenario depicted, space is divided into two layers: ground and near-Earth orbit. Devices in these two layers communicate via satellite-to-ground links, including uplink and downlink. Satellites in the satellite network communicate via inter-satellite links. Ground-based devices are mainly divided into two categories. The first category consists of IoT devices related to power grid operations, responsible for sensing the power grid's operational status, collecting power grid data, and receiving instructions to adjust the power grid's operating state. In traditional solutions, the second category comprises core network devices serving as the command and control center for the entire system. Their primary function is to receive data transmitted from the first category of devices, record daily data on the operation of various power grid components, perform calculations based on this data to make management decisions, and then transmit instructions back to the first category of devices.

[0101] Unlike traditional solutions, the first type of equipment, located relatively far from the ground core network, no longer transmits data to the cloud service center via terrestrial communication links. Instead, it directly uploads task data to the satellite network for edge computing via an air-to-ground communication link. In this embodiment, the satellite closest to the target power grid equipment in the satellite network takes over the computing tasks originally belonging to the ground cloud service center, significantly reducing communication latency.

[0102] It should be further noted that this embodiment focuses on the intelligent allocation of satellite network resources, with the default scenario being that task data is offloaded to the satellite. The task data offloading rules for the first type of device are only briefly described here. For example, the device selects the computing node to handle the service based on its distance from the ground base station. When the distance between the first type of device and the ground base station is less than a preset threshold, the data is transmitted to the cloud service center via the terrestrial fiber optic network. When the distance between the first type of device and the ground base station is greater than or equal to the preset threshold, or when the terrestrial network becomes unreachable due to unforeseen circumstances, the task data is offloaded to the satellite. Based on the above judgment rules, factors such as the quality of the satellite-to-ground link and satellite load can also be considered to more comprehensively determine the offloading targets for power grid services.

[0103] Generally, a satellite, acting as the nearest node determined by the power grid equipment, receives computing tasks uploaded by the corresponding power grid equipment. See also Figure 4 During the edge computing access phase of power grid services, power grid equipment selects the most suitable satellite based on the specific distance and offloads power grid services to that satellite via the uplink, in units of computing tasks. That is, the power grid equipment prioritizes uploading computing tasks to the satellite closest to it. Simultaneously, it also receives the computing results of previous services via the downlink.

[0104] Compared to traditional solutions that rely on terrestrial core network infrastructure for computational task offloading, this embodiment introduces a dynamic proximity offloading mechanism using satellites. Specifically, by assessing satellite trajectories and grid equipment locations in real time, and utilizing satellite-to-ground links and nearby task migration, end-to-end latency is significantly reduced. This ensures ultra-low latency response and reliability for grid services, making it particularly suitable for smart grid monitoring and fault diagnosis in remote areas.

[0105] For details, see Figure 4During the satellite network resource allocation phase, each satellite deploys a deep reinforcement learning model, which is trained and updated in real time based on the overall system's operational status. This deep reinforcement learning model can control each satellite undertaking edge computing tasks, enabling it to rationally allocate computing tasks to other satellites in the network, thereby improving the overall network resource utilization. In the resource allocation phase, the output results of the deep reinforcement learning model can be used for real-time resource scheduling to adapt to sudden demands from power grid operations. Compared to traditional static scheduling algorithms, this improves resource utilization, reduces scheduling decision time, and enhances overall computational efficiency and network robustness.

[0106] See Figure 6 This application also provides an inter-satellite mission allocation system, which includes: The model loading module is used to load the task allocation model for the first satellite, which is any satellite in the satellite communication system. The task allocation module is used to input the current state of the first satellite into the task allocation model to obtain the target action; the state is the task queue of the first satellite, and the target action is used to determine the task to be allocated from the task queue; The task sending module is used by the first satellite to send the task to be assigned to the second satellite, wherein the second satellite is any satellite in the satellite communication system other than the first satellite.

[0107] See Figure 7 This application also provides an electronic device including a memory and a processor. The memory stores a computer program, which, when executed by the processor, implements the methods described in the above embodiments. As an example, the electronic device may include multiple processors. A processor may refer to one or more devices, circuits, and / or computing units for processing data (e.g., computer programs). The processor can invoke the computer program stored in the memory to implement the methods described in the above embodiments. Figure 7 Taking an electronic device consisting of one processor and one memory as an example, the processor and memory are used to indicate a type of device or equipment, and the quantity of each type of device or equipment can be determined according to business needs.

[0108] This application also provides a computer-readable storage medium storing a computer program or instructions that, when executed on a computer, cause the computer to perform the methods described in the above-described method embodiments.

[0109] It should be noted that, in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0110] Those skilled in the art should clearly understand that, for the sake of convenience and brevity, the specific working processes of the inter-satellite task allocation system, electronic equipment, and computer-readable storage medium described in the above embodiments can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0111] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing the relevant hardware to implement them. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0112] The above are merely optional embodiments of this application, used only to illustrate the technical solution of this application and not to limit it. Any modifications, equivalent substitutions, improvements, etc., to the specific implementation of this application without departing from the spirit and scope of this application should be covered within the protection scope of this application.

Claims

1. A training method for a task allocation model, wherein the task allocation model includes an action network and a value network, characterized in that, include: Based on the current time slot status of the first satellite, the action of the current time slot of the first satellite is determined through the action network. The status is the task queue of the first satellite, and the action is that the first satellite selects a task to be assigned from the task queue and sends it to any satellite in the satellite communication system other than the first satellite. The first satellite is any satellite in the satellite communication system. Based on the actions of the first satellite in the current time slot, determine the reward obtained by the first satellite in performing the actions in the current time slot and the state of the next time slot; The reward includes a latency reward and an energy reward. The latency reward is the negative of the maximum total latency value of the tasks to be assigned in the task queue. The total latency is the total time from when the task to be assigned waits for transmission on the first satellite until the second satellite starts calculation. The energy reward is the negative of the cumulative energy consumption of the task to be assigned from when it waits for transmission on the first satellite until it reaches the second satellite. The second satellite is any satellite in the satellite communication system other than the first satellite. Based on the status of the next time slot of the first satellite, the revenue of the next time slot of the first satellite is determined through the value network; After traversing a preset number of time slots, the revenue of the current time slot is obtained based on the revenue of the next time slot of the first satellite and the reward of the current time slot; Based on the current time slot's revenue and the value network's prediction of the current state's value, the action network and the value network are trained and optimized to obtain optimized action networks and value networks.

2. The method according to claim 1, characterized in that, Based on the actions of the first satellite in the current time slot, determine the reward obtained by the first satellite in performing the actions in the current time slot and the state of the next time slot, including: The corresponding latency reward is determined based on the total latency determined after the first satellite performs its action in the current time slot; The corresponding energy reward is determined based on the energy consumption of the task waiting to be transmitted in the first satellite and the energy consumption of the task being transmitted to the second satellite. Based on the delay reward and the energy reward, determine the reward obtained by the first satellite for performing the action in the current time slot; After the first satellite performs the action in the current time slot, it updates its task queue to obtain the status of the next time slot.

3. The method according to claim 2, characterized in that, Based on the total delay determined after the first satellite performs its action in the current time slot, a corresponding delay reward is determined, including: Based on the current actions performed by the first satellite, determine the tasks to be sent to the second satellite. Based on the determined tasks to be assigned, the total number of tasks to be assigned is obtained, and multiple total delays corresponding to the total number are determined, wherein each task to be assigned corresponds to a total delay. Calculate the absolute values ​​of multiple total delays, and select the maximum value as the delay reward obtained by the first satellite for performing the action in the current time slot.

4. The method according to claim 2, characterized in that, The step of determining the corresponding energy reward based on the energy consumption of the task waiting for transmission in the first satellite and the energy consumption of transmission to the second satellite includes: The energy consumption of all tasks awaiting assignment in the first satellite and waiting to be transmitted to the second satellite is accumulated, and the negative of the accumulated result is used as the energy reward obtained by the first satellite for performing the action in the current time slot.

5. The method according to claim 2, characterized in that, Based on the latency reward and the energy reward, determine the reward obtained by the first satellite for performing the action in the current time slot, including: The reward obtained by the first satellite in performing the action in the current time slot is determined based on the weighted sum of the delay reward and the energy reward.

6. The method according to claim 2, characterized in that, Determining the total delay includes: The transmission waiting time of the task to be assigned in the first satellite is determined based on the length of the task to be assigned in the first satellite and the transmission rate between the first satellite and the second satellite. Based on the data size of the task to be assigned and the transmission rate between the first and second satellites, the transmission delay of the task to be assigned from the first satellite to the second satellite is determined. Based on the length of the task to be calculated on the second satellite and the calculation time required for the task to be assigned, the calculation waiting delay of the task to be assigned on the second satellite is determined. The total delay is determined based on the transmission waiting delay, the transmission delay, and the calculation waiting delay.

7. The method according to claim 6, characterized in that, The step of determining the total delay based on the transmission waiting delay, the transmission delay, and the calculated waiting delay includes: The total delay is the sum of the transmission waiting delay and the calculation waiting delay.

8. The method according to claim 2, characterized in that, Obtaining the energy consumption includes: Based on the length of the task to be assigned on the first satellite, the amount of data in the task to be assigned, the transmission rate between the first and second satellites, and the transmission power of the first satellite, the energy consumption of the task to be assigned from the time it starts waiting for transmission on the first satellite to the time it arrives at the second satellite is determined.

9. The method according to claim 6 or 8, characterized in that, Obtain the transmission rate between the first satellite and the second satellite, including: The transmission rate between the first and second satellites is obtained based on the available bandwidth of the route between the first and second satellites, the signal transmission power of the first satellite, the gain of the transmitting antenna of the first satellite, the gain of the receiving antenna of the second satellite, the path loss between the first and second satellites, the noise bandwidth, and the quantum efficiency of the photodetector.

10. The method according to claim 1, characterized in that, Based on the status of the first satellite's next time slot, the revenue of the first satellite's next time slot is determined through a value network, including: Based on the state corresponding to the first satellite target time slot, the value prediction value corresponding to that state is determined through a value network; the target time slot is the sum of the next time slot and a preset number; The value network determines the revenue of the first satellite in the next time slot based on the rewards of the preset number of time slots and the value prediction value corresponding to the target time slot.

11. The method according to claim 10, characterized in that, The value network determines the revenue of the first satellite in the next time slot based on the rewards of the preset number of time slots and the predicted value corresponding to the target time slot, including: The value network weights and sums the rewards for the preset number of time slots starting from the next time slot with the predicted value corresponding to the target time slot, and accumulates the weighted results to obtain the revenue of the first satellite in the next time slot.

12. The method according to claim 1, characterized in that, The training and optimization of the action network and value network includes: Determine the loss function based on the revenue of the current time slot and the predicted value; By minimizing the loss function, the network parameters of the action network and the value network are updated, thus completing the training optimization of the action network and the value network.

13. A method for assigning tasks among satellites, characterized in that, include: The first satellite loads the task allocation model, where the first satellite is any satellite in the satellite communication system; The task allocation model is obtained by the method described in any one of claims 1 to 12; The current state of the first satellite is input into the task allocation model to obtain the target action; the state is the task queue of the first satellite, and the target action is used to determine the task to be allocated from the task queue. The first satellite transmits the task to be assigned to the second satellite, which is any satellite in the satellite communication system other than the first satellite.

14. The method according to claim 13, characterized in that, The task queue includes all tasks to be processed by the first satellite; the target action is the action corresponding to the maximum benefit among the benefits corresponding to a preset number of time slots.

15. The method according to claim 13, characterized in that, The tasks to be processed by the first satellite originate from power grid equipment; Based on the location of the power grid equipment and the motion trajectory of each first satellite in the satellite communication system, the first satellite closest to the power grid equipment is determined, and the power grid equipment sends its mission to the first satellite.

16. A satellite mission allocation system, implementing the method of any one of claims 13 to 15, characterized in that, include: The model loading module is used to load the task allocation model for the first satellite, which is any satellite in the satellite communication system. The task allocation module is used to input the current state of the first satellite into the task allocation model to obtain the target action; the state is the task queue of the first satellite, and the target action is used to determine the task to be allocated from the task queue; The task sending module is used by the first satellite to send the task to be assigned to the second satellite, wherein the second satellite is any satellite in the satellite communication system other than the first satellite.

17. An electronic device, characterized in that, It includes a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program in the memory to implement the method of any one of claims 1 to 15.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program or instructions that, when executed on a computer, cause the computer to perform the method of any one of claims 1 to 15.

Images