Networked measurement and control data transmission task scheduling method for giant low-orbit satellite constellations

Abstract

The present invention discloses a networked measurement, control and data transmission task scheduling method for a giant low-orbit satellite constellation, which solves the problem of low completion rate of measurement, control and data transmission tasks in the prior art due to the inability to deploy stations globally. The method comprises: obtaining tasks to be planned; obtaining resource attributes, ground station status and low-orbit satellite status by using a task layer, a resource layer, a device status and a constellation status, and performing data conversion to obtain input data, inputting the input data into a trained multi-agent network to obtain an action corresponding to a current time slot, and determining an inbound low-orbit satellite and an associated ground station according to the action; obtaining a routing path between an inbound low-orbit satellite and a mission low-orbit satellite according to an inter-satellite shortest path routing table, and determining a scheduling sequence according to the routing path and the associated ground station; achieving consideration of the mutual influence between global tasks and resources and reducing the effectiveness of decision-making during execution, and at the same time improving the scheduling efficiency of measurement, control and data transmission tasks in a giant low-orbit satellite constellation scenario.

Description

Technical Field

[0001] The present invention relates to the field of low-orbit satellite communication technology, and in particular to a networked measurement, control and data transmission task scheduling method for a giant low-orbit satellite constellation. Background Art

[0002] Low-orbit satellite measurement, control and data transmission are important technical means to ensure that low-orbit satellites can perform their tasks normally and realize on-board data transmission. The existing measurement, control and data transmission system is mainly based on ground-based resources. The low-orbit satellite and the ground station must be visible to the ground before the measurement, control and data transmission information can be transmitted to the ground network. At the same time, a ground station has multiple visible low-orbit satellites, but a ground station can only establish a connection with one low-orbit satellite at the same time, which will inevitably cause conflicts in the use of equipment. With the development of giant low-orbit satellite constellations, the demand for measurement, control and data transmission has increased dramatically. Due to the inability to deploy stations globally, this conflict between low-orbit satellite constellations and ground equipment has increased dramatically. Relying solely on ground-based measurement and control has poor timeliness, low task completion rate, and serious resource conflicts, which seriously affects the efficient operation of giant low-orbit satellite constellations. The existing measurement, control and data transmission system architecture has shifted from ground-based control to networked measurement, control and data transmission technology that integrates the earth and the sky. Networked measurement, control and data transmission technology transmits measurement, control and data transmission information in the network to a unified platform, coordinates and dispatches low-orbit satellites and ground-based equipment, and ensures the efficient operation of giant low-orbit satellite constellations. How to use existing resources to design a reasonable measurement, control and data transmission task scheduling framework and related scheduling methods to reduce conflicts in equipment use and improve resource utilization and task completion rate is an important issue in the operation, maintenance and management of low-orbit giant constellations.

[0003] Existing TT&C mission planning schemes often ignore the research related to real-time acquisition of overseas low-orbit satellite data. Due to geographical or political reasons, it is impossible to deploy stations globally, and overseas low-orbit satellites cannot directly establish satellite-to-ground links with ground stations, nor can they achieve real-time monitoring of overseas low-orbit satellites and real-time data transmission. Intersatellite links build a bridge for transmitting TT&C information and low-orbit satellite data between overseas low-orbit satellites and domestic low-orbit satellites, making the TT&C operations that originally required entry into the country become remote operations that can be performed at any time, and also improving the utilization rate of equipment resources. However, the problem of how to use intersatellite links to realize the transmission of TT&C information and design corresponding TT&C mission scheduling methods has arisen.

[0004] Low-orbit satellites play an irreplaceable role in emergency rescue, fire monitoring and air transport expansion, and are an important means of information acquisition, transmission and processing. In recent years, large-scale low-orbit satellite constellations have been developing rapidly. For example, Starlink plans to launch 42,000 low-orbit satellites. Low-orbit satellite measurement and control is an important cornerstone for ensuring the normal operation of giant constellations, completing designated tasks, and responding quickly to sudden demands. The process of low-orbit satellites transmitting data to ground stations after performing tasks is called low-orbit satellite data transmission, which is an important technical means for ground networks to receive low-orbit satellite data. However, low-orbit satellite measurement and control and data transmission tasks need to be completed through the establishment of links between onboard antennas and receiving devices within a geometric time window, but the low-orbit satellite and the receiving device are not always visible. Due to the inability to deploy stations globally, the use of satellite-to-ground links to perform measurement and control data transmission tasks in giant constellation scenarios has resulted in many tasks being unable to be effectively processed, with low task completion rates and resource utilization, posing major challenges to the operation, maintenance and management of giant constellations.

[0005] Therefore, how to combine existing resources to design a measurement and control data transmission task adjustment method for giant constellation resource management is an urgent problem to be solved. Summary of the invention

[0006] The present invention provides a networked measurement, control and data transmission task scheduling method for giant low-orbit satellite constellations, thereby solving the problem of low completion rate of measurement, control and data transmission tasks in the prior art due to the inability to deploy stations globally. It takes into account the mutual influence between global tasks and resources and reduces the effectiveness of decision-making during execution, while improving the scheduling efficiency of measurement, control and data transmission tasks in the giant low-orbit satellite constellation scenario.

[0007] The present invention provides a networked measurement and control data transmission task scheduling method for a giant low-orbit satellite constellation, the method comprising:

[0008] Get the tasks to be planned;

[0009] The task layer is used to extract elements of the task to be planned to obtain task elements, and the resource layer, equipment status and constellation status are used to extract the status of the measurement and control center, the status of the low-orbit satellite constellation and the status of the ground station to obtain resource attributes, ground station status and low-orbit satellite status;

[0010] The task elements, the resource attributes, the ground station status and the low-orbit satellite status are respectively converted to obtain input data, and the input data is input into the trained multi-agent network to obtain the action corresponding to the current time slot, and the inbound low-orbit satellite and the associated ground station are determined according to the action;

[0011] The routing path between the inbound low-orbit satellite and the mission low-orbit satellite is obtained according to the shortest path routing table, and a scheduling sequence is determined according to the routing path and the associated ground station.

[0012] In a possible implementation, the extracting elements of the task to be planned by using the task layer to obtain the task elements, and extracting the status of the measurement and control center, the low-orbit satellite constellation and the ground station by using the resource layer, the equipment status and the constellation status to obtain the resource attributes, the ground station status and the low-orbit satellite status, including:

[0013] The task layer includes a plurality of tasks to be planned, and the task types of the tasks to be planned include: measurement and control tasks and data transmission tasks; the task elements include: task type, shortest execution time, task priority and task minimum elevation angle;

[0014] The resource layer includes visible time window resources and inter-satellite link resources, and the resource attributes include time window start time, time window end time, time window maximum elevation angle and inter-satellite route shortest hop count;

[0015] The device status includes a ground station status and a ground station type, wherein the ground station status includes: a current execution status of the ground station and a functional abstraction of the ground station;

[0016] The constellation status includes: a low-orbit satellite status and a link status. The low-orbit satellite status includes: a time interval between the last measurement and control mission performed by the low-orbit satellite and a current execution status of the low-orbit satellite.

[0017] In a possible implementation, the training process of the multi-agent network includes:

[0018] The multi-agent network includes: an online strategy network, an online value network, a target strategy network and a target value network;

[0019] The input of the online policy network is a local observation vector, the output is an action, and the gradient descent method is used to update the parameters of the online policy network through the action to obtain a trained online policy network;

[0020] The input of the online value network is an observation vector and an action vector, and the output is a one-dimensional output value. The gradient descent method is used to update the parameters of the online value network through the one-dimensional output value to obtain a trained online value network;

[0021] Using a soft update method to complete the update of the target value network and the target policy network corresponding to the online value network and the online policy network, and obtain a trained target value network and a target policy network;

[0022] Wherein, the local observation vector is a part of the state of each agent in the current time slot calculated by each agent according to available resources;

[0023] The available resources are obtained by screening the task elements, resource attributes, ground station status and low-orbit satellite status corresponding to each time slot according to the first constraint condition; the available resources are different in each round of training, and the available resources in each round of training are calculated according to the second constraint condition and the reward calculation formula for each intelligent agent, and the available resources are updated according to the reward;

[0024] The task elements are obtained by performing data processing on the tasks in the training set using the task layer;

[0025] The resource attributes, the ground station status and the low-orbit satellite status are obtained by extracting the status of the measurement and control center, the status of the low-orbit satellite constellation and the status of the ground station respectively using the resource layer, the equipment status and the constellation status;

[0026] The observation vector is obtained by concatenating the agent states of each agent in the experience replay pool;

[0027] The action vector is obtained by concatenating the output actions of each agent in the experience replay pool;

[0028] The data stored in the experience replay pool is obtained by the online strategy network after multiple rounds of training.

[0029] In a possible implementation, the reward calculation formula is expressed as:

[0030] ;

[0031] in, Representation Task In the visible time window whether it is scheduled for execution; represents the gain factor; Representation Task In the visible time window whether conflicts occur; represents the loss factor; Indicates the agent chooses to perform tasks The subsequent income; express Slotted Agent Rewards after performing actions; Representing an Agent exist Parameters of the time slot.

[0032] In a possible implementation, the local observation vector is expressed as:

[0033] ;

[0034] in, Indicates whether there is a task; Indicates mission low-orbit satellite; Indicates the task priority; Indicates ground station Status; Indicates ground station Whether to support tasks to be planned; Respectively represent the ground station Visible low-orbit satellite information; Indicates selection visible low-orbit satellites; express Slotted Agent The local observation vector of Representing an Agent exist Parameters of the time slot; Indicates ground station Status; Indicates ground station Whether to support tasks to be planned; Indicates that the ground station Visible low-orbit satellite information.

[0035] In a possible implementation, the action is a two-dimensional tuple, and the two-dimensional tuple is represented as:

[0036] ;

[0037] in, Indicates selection ground stations; Indicates that the ground station The visible Low Earth Orbit Satellites; express Slotted Agent Actions; Representing an Agent exist Parameters of the time slot.

[0038] In a possible implementation, determining an inbound low-orbit satellite and an associated ground station according to the action includes:

[0039] Mapping the action into a one-dimensional action space to obtain a mapping result;

[0040] The corresponding relationship between the action and the mapping result is expressed as:

[0041] ;

[0042] in, Expressed as a ceiling function; Representing an Agent exist The mapping result of time slots; express The total number of visible LEO satellites in the selected time slot; Indicates selection ground stations; Indicates that the ground station The visible Low Earth Orbit Satellites; express Slotted Agent action.

[0043] In a possible implementation, the objective function of the networked measurement and control data transmission task scheduling method is expressed as:

[0044] ;

[0045] in, Represents a set of tasks; represents the set of ground stations; Represents a set of visible time windows; represents the gain factor; Representation Task In the visible time window whether it is scheduled for execution; represents the loss factor; Representation Task In the visible time window whether conflicts occur; Indicates the agent chooses to perform tasks After the income.

[0046] One or more technical solutions provided in the present invention have at least the following technical effects or advantages:

[0047] The present invention adopts a networked measurement and control data transmission task scheduling method for giant low-orbit satellite constellations. First, the measurement and control data transmission task scheduling method is applicable to large-scale low-orbit satellite networks. Secondly, the measurement and control data transmission task scheduling process is expressed as a multi-agent deep reinforcement learning process, and a measurement and control data transmission task scheduling model suitable for giant LEO low-orbit satellite constellations is established by combining strategy networks and value networks. In addition, the "centralized training, distributed execution" characteristics of the MADDPG algorithm can take into account the influence between multiple ground stations and multiple tasks during task selection, and can improve the scheduling efficiency of measurement and control data transmission tasks in giant low-orbit satellite constellations, and ensure the timeliness of measurement and control data transmission task allocation;

[0048] The present invention describes the constraint satisfaction target model inherent in measurement and control data transmission task scheduling as a dynamic decision-making process that changes with time series through a multi-agent deep reinforcement learning framework. When the tasks in the environment change, existing models can be used for training, which greatly shortens the model training time and ensures the scalability of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flowchart of the steps of a networked measurement, control and data transmission task scheduling method for a giant low-orbit satellite constellation provided by an embodiment of the present invention;

[0050] Figure 2 The embodiment of the present invention provides a giant low-orbit satellite constellation measurement, control and data transmission function framework;

[0051] Figure 3 A schematic diagram of a giant low-orbit satellite constellation measurement, control, and data transmission scheduling scenario provided by an embodiment of the present invention;

[0052] Figure 4 A flowchart of a specific use example provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] The present invention provides a networked measurement and control data transmission task scheduling method for a giant low-orbit satellite constellation. Figure 1 As shown, the method includes the following steps S101 to S104.

[0055] S101, obtaining tasks to be planned;

[0056] S102, extracting elements of the task to be planned by using the task layer to obtain task elements, and extracting the status of the measurement and control center, the low-orbit satellite constellation and the ground station by using the resource layer, the equipment status and the constellation status to obtain resource attributes, the ground station status and the low-orbit satellite status;

[0057] Specifically, in step S102, the task layer is used to extract elements of the planned task to obtain task elements, and the resource layer, equipment status and constellation status are used to extract the status of the measurement and control center, the status of the low-orbit satellite constellation and the status of the ground station to obtain resource attributes, ground station status and low-orbit satellite status, including the following steps S1021 to S1023.

[0058] S1021, the task layer includes a plurality of tasks to be planned, and the task types of the tasks to be planned include: measurement and control tasks and data transmission tasks; the task elements include: task type, shortest execution time, task priority and task minimum elevation angle;

[0059] S1022, the resource layer includes visible time window resources and inter-satellite link resources, and the resource attributes include time window start time, time window end time, time window maximum elevation angle and inter-satellite route shortest hop count;

[0060] S1023, the device status includes the ground station status and the ground station type, and the ground station status includes: the current execution status of the ground station and the functional abstraction of the ground station;

[0061] S1024, the constellation status includes: low-orbit satellite status and link status. The low-orbit satellite status includes: the time interval between the last measurement and control mission performed by the low-orbit satellite and the current execution status of the low-orbit satellite.

[0062] S103, respectively converting the task elements, resource attributes, ground station status and low-orbit satellite status to obtain input data, and inputting the input data into the trained multi-agent network to obtain the action corresponding to the current time slot, and determining the inbound low-orbit satellite and the associated ground station according to the action;

[0063] Specifically, in step S103, the multi-agent network includes: an online strategy network, an online value network, a target strategy network and a target value network.

[0064] The training process of a multi-agent network includes:

[0065] The input of the online policy network is the local observation vector, and the output is the action. The gradient descent method is used to update the parameters of the online policy network through the action to obtain the trained online policy network.

[0066] Here, the local observation vector is a part of the state of each agent in the current time slot calculated by each agent based on the available resources; the available resources are obtained by screening the task elements, resource attributes, ground station status and low-orbit satellite status corresponding to each time slot according to the first constraint condition; the available resources in each round of training are different, and the available resources in each round of training are calculated based on the second constraint condition and the reward calculation formula for each agent. The available resources are updated according to the reward.

[0067] Here, the task elements are obtained by using the task layer to perform data processing on the tasks in the training set; the resource attributes, ground station status and low-orbit satellite status are obtained by using the resource layer, equipment status and constellation status to extract the status of the measurement and control center, the status of the low-orbit satellite constellation and the status of the ground station respectively.

[0068] The local observation vector is expressed as:

[0069] (1);

[0070] in, Indicates whether there is a task; Indicates mission low-orbit satellite; Indicates the task priority; Indicates ground station Status; Indicates ground station Whether to support tasks to be planned; Respectively represent the ground station Visible low-orbit satellite information; Indicates selection visible low-orbit satellites; express Slotted Agent The local observation vector of Representing an Agent exist Parameters of the time slot; Indicates ground station Status; Indicates ground station Whether to support tasks to be planned; Indicates that the ground station Visible low-orbit satellite information.

[0071] Here, the action is a two-dimensional tuple, which is represented by:

[0072] (2);

[0073] in, Indicates selection ground stations; Indicates that the ground station The visible Low Earth Orbit Satellites; express Slotted Agent Actions; Representing an Agent exist Parameters of the time slot.

[0074] The reward calculation formula is expressed as:

[0075] (3);

[0076] in, Representation Task In the visible time window whether it is scheduled for execution; represents the gain factor; Representation Task In the visible time window whether conflicts occur; represents the loss factor; Indicates the agent chooses to perform tasks The subsequent income; express Slotted Agent Rewards after performing actions; Representing an Agent exist Parameters of the time slot.

[0077] Specifically, in step S103, the inbound low-orbit satellite and the associated ground station are determined according to the action, including: mapping the action into a one-dimensional action space to obtain a mapping result.

[0078] The corresponding relationship between actions and mapping results is expressed as:

[0079] (4);

[0080] in, Expressed as a ceiling function; Representing an Agent exist The mapping result of time slots; express The total number of visible LEO satellites in the selected time slot; Indicates selection ground stations; Indicates that the ground station The visible Low Earth Orbit Satellites; express Slotted Agent action.

[0081] The input of the online value network is the observation vector and the action vector, and the output is a one-dimensional output value. The gradient descent method is used to update the parameters of the online value network through the one-dimensional output value to obtain the trained online value network.

[0082] Here, the observation vector is obtained by concatenating the agent states of each agent in the experience replay pool, and the action vector is obtained by concatenating the output actions of each agent in the experience replay pool. The data stored in the experience replay pool is obtained after multiple rounds of training of the online policy network.

[0083] The target value network and target policy network corresponding to the online value network and the online policy network are updated by using a soft update method to obtain the trained target value network and target policy network.

[0084] Exemplarily, the environment of the multi-agent network includes: a ground measurement and control center MCC, ground stations, and a giant LEO (Low Earth Orbit) satellite constellation.

[0085] The ground measurement, control and transmission center (MCC) is responsible for arranging instructions for the measurement, control and data transmission requirements of the LEO low-orbit satellite constellation.

[0086] The ground station is responsible for the injection of measurement and control instructions and the reception of digital data. According to its function, it can be divided into measurement and control ground station, digital transmission ground station and multi-function ground station. Among them, the measurement and control ground station and the digital transmission ground station can only perform the corresponding tasks respectively, and the multi-function ground station can perform both measurement and control tasks and digital transmission tasks.

[0087] The giant LEO low-orbit satellite constellation refers to low-orbit satellites with orbital altitudes between 200km-1200km. The LEO low-orbit satellites use a "one-satellite four-link" connection method to achieve the transmission of measurement and control commands or digital information. It is set that each low-orbit satellite can establish inter-satellite links with four low-orbit satellites at the same time, including one forward and backward link within the same orbit, and one direction-finding link on the left and right different orbital planes. In detail, the ground measurement and control center MCC arranges commands according to the type and characteristics of the mission, implements command injection through the ground stations of each type, and then reaches the corresponding low-orbit satellite through the inter-satellite link; at the same time, the data of the low-orbit satellite abroad is transmitted to the country through the inter-satellite link, and then the data is transmitted down through the satellite-to-ground link.

[0088] The giant LEO satellite constellation is described as: , Indicates the total number of low-orbit satellites. It is assumed that each low-orbit satellite can establish inter-satellite links with four low-orbit satellites at the same time, including one forward and one backward link within the same orbit, and one direction-finding link on the left and right different orbit planes. The use of inter-satellite links enables low-orbit satellites to complete the transmission of measurement, control and digital information through inter-satellite links without entering the country.

[0089] The input tasks of the multi-agent network are divided into two categories: measurement and control tasks and data transmission tasks. The tasks in the network are uniformly described as , Indicates the total number of tasks, ,in, Indicates the task number, which is the unique identifier to distinguish tasks. Indicates the number of the low-orbit satellite that performs the mission. Indicates the shortest task execution time. represents the minimum elevation angle requirement of the mission, Indicates the task priority. Indicates the task type. The task types are divided into two categories: "TTC" (measurement and control task) and "DDT" (data transmission task). Indicates the orbit type required by the mission. The orbit types are Raising, Lowering, and Null (no orbit requirement).

[0090] The ground station is described as , represents the total number of ground stations, where ,in, Indicates the ground station number, which is the unique identifier to distinguish different ground stations in the network. Indicates the type of ground station. In the current network, ground station types are divided into: TTC (Telemetry and Control Ground Station), DDT (Digital Transmission Ground Station) and TTC&DDT (Multi-function Ground Station).

[0091] The set of visible time windows is described as , Indicates the number of visible time windows, , ,in, Indicates the measurement and control equipment number. Indicates the low-orbit satellite number. Indicates the start time of the visible time window. Indicates the end time of the visible time window. represents the maximum elevation angle in this time window, Indicates the track type.

[0092] like Figure 2The figure shows the measurement, control and data transmission functional framework of a giant low-orbit satellite constellation. The task layer includes multiple tasks to be executed. The task types are measurement, control and data transmission tasks. These tasks drive the operation of the entire intelligent agent cluster. Task elements are extracted according to task characteristics and requirements. Task elements include task type, shortest execution time, task priority and minimum task elevation angle, etc.

[0093] The resource layer includes visible time window resources and inter-satellite link resources. The present invention adopts the method of visible time window plus inter-satellite link to perform measurement and control data transmission tasks, and extracts visible time window and inter-satellite link resource elements. The resource elements include the time window start time, the time window end time, the time window maximum elevation angle and the shortest number of inter-satellite routing hops.

[0094] The device status includes the ground station status and ground station type. The extracted related elements include the current execution status of the ground station and the functional abstraction of the ground station. The task to be executed must match the function of the ground station before it can be executed on the station.

[0095] The constellation status includes the low-orbit satellite status and link status. The extracted related factors include the time interval of the low-orbit satellite's previous measurement and control mission, the current execution status of the low-orbit satellite, etc. The execution of the low-orbit satellite measurement and control mission must meet a certain time interval.

[0096] For multi-agent networks, after the task layer, resource layer, equipment status and constellation status are extracted, the elements are reshaped into a readable format for the agent set. The key data is used as the state input of the agent. Each agent selects the optimal way to execute the task and the corresponding ground station according to the state.

[0097] Application scenarios of the present invention are as follows Figure 3 As shown, the ground measurement and operation control center quickly calculates the status of each intelligent agent in the current network environment based on the visible time window resource attributes, inter-satellite link resource attributes, task attributes, low-orbit satellite status and ground station status.

[0098] The present invention adopts the MADDPG algorithm to implement multi-agent deep reinforcement learning, where the state of each agent can be expressed as: ,in represents the observation vector of all agents on the current low-orbit satellite network environment state, represents the observation vector of all agents for the next state of the environment, represents the action vector of each agent in the current environment, Indicates the benefits of each agent after executing the action. If the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm includes N agents, each agent contains four deep neural networks, including the online policy network , Online Value Network , target strategy network and target value network , the parameters of the online policy network and the online value network The parameters of the target policy network and the target value network The synchronization between them adopts the soft update method, that is, ;in, represents the soft update coefficient;

[0099] Each agent transforms its local observation vector As input to the online policy network, it outputs an action , the action selection adopts Strategy, based on The value of selects whether to execute the action calculated by the current online policy network, calculates the mean square error loss function of the temporal difference error and calculates the gradient of the parameters of the online policy network, and then uses gradient descent to update the parameters of the online policy network.

[0100] At the same time, the MADDPG algorithm concatenates the local observation vectors of each agent into an observation vector , concatenate all agent actions into an action vector ,Will As the input of the online value network, the output of the online value network is a one-dimensional output value , which is the Q value, and then calculated based on the samples in the experience playback pool , gradient descent is used to update the parameters of the online value network, that is, the strategies of other agents are considered to achieve collaborative decision-making.

[0101] The ultimate goal of multi-agent deep reinforcement learning is for each agent to continuously interact with the environment and learn to achieve a preset strategy. In order to better evaluate the benefits of task execution, a binary variable is introduced. and , Representation Task In the visible time window Whether it is scheduled for execution, Representation Task In the visible time window Is there a conflict in the task? In the visible time window Assigned to ground station On, then ,otherwise If the ground station If there is a conflict with other tasks, ,otherwise By tracking and The value of can be used to determine the task Successful execution, obviously, when When, the task Successfully executed. Therefore, the task The total number of completions can be expressed as:

[0102] (5);

[0103] in, Represents a set of tasks; represents the set of ground stations; Represents a set of visible time windows;

[0104] If the task If it is successfully executed, it will bring certain benefits ,in Indicates the gain factor. If a task conflicts, it will cause a certain loss. ,in, Represents the loss factor.

[0105] Therefore, the objective function of the networked measurement and control data transmission task scheduling method in the present invention is defined as:

[0106] (6);

[0107] in, Represents a set of tasks; represents the set of ground stations; Represents a set of visible time windows; represents the gain factor; Representation Task In the visible time window whether it is scheduled for execution; represents the loss factor; Representation Task In the visible time window whether conflicts occur; Indicates the agent chooses to perform tasks After the income.

[0108] The present invention adopts a ground station to realize low-orbit satellite measurement and control and data transmission, and uses inter-satellite links to forward measurement and control data transmission instructions and data packets, focusing on the scheduling of measurement and control data transmission tasks. Which low-orbit satellite is used to send the measurement and control data transmission data packet is a problem that the present invention needs to consider. Therefore, the present invention directly uses the shortest path routing method to obtain the inter-satellite path of the data packet.

[0109] Here, the objective function of the trained networked measurement and control data transmission task scheduling method is expressed as formula (6).

[0110] The total revenue of the multi-agent network trained here is expressed as:

[0111] (7);

[0112] in, Indicates the total time slot of the scene, represents the total number of agents, express Slotted Agent The reward after performing an action.

[0113] In a specific embodiment provided by the present invention, Figure 4 As shown, in a specific embodiment provided by the present invention, the specific process of training a multi-agent network in the inter-satellite link measurement and control data transmission task scheduling method for a low-orbit satellite constellation includes:

[0114] (1) Initialize the measurement and control data transmission in the network, including task set, resource set, constellation status information and ground station status information;

[0115] (2) The ground measurement and control center MCC passes Sparse matrix of time slot visibility query Query The set of inbound low-orbit satellites corresponding to each ground station in the time slot; Indicates the visible status of the star and the earth at that moment;

[0116] (3) Select based on priority The measurement and control and data transmission tasks are stored in the task queue. The task with the smaller number of hops between the task LEO satellite and the LEO satellite entering the current time slot has a higher priority. When selecting tasks, constraint C8 must be met, that is, a certain time interval must be maintained between tasks of the same type on the same LEO satellite.

[0117] (4) According to the selected The task information of each task, the status information of each ground station, the status information of the incoming low-orbit satellite, and the satellite-to-ground visible time window information of the low-orbit satellite and the ground station are used to analyze whether the ground station can execute the task. The constraints C3, C4, C5, C6, and C7 must be met when making the judgment. The task information of each task, the status information of all ground stations in the environment, the shortest number of hops from the incoming low-orbit satellite to the mission low-orbit satellite, and the equipment support status are used as the intelligent agent The local observation of Agents in The local observation vector of the time slot;

[0118] (5) The agent interacts with the environment. The local observation vector As input to the online policy network, the online policy network outputs a deterministic action , select the corresponding ground station and inbound low-orbit satellite according to formula (16). According to constraint C4, even if a ground station The time slot supports the execution of multiple tasks, and each agent cannot choose the same action. Finally, the reward is obtained according to formula (17): . The ground station status information table The status of the ground station that has been occupied is reduced by one, and the status of the newly occupied ground station is updated to the time required for mission execution. Low-orbit satellite status information table The usage status of the incoming low-orbit satellites in the table is also updated as above. The shortest time interval for the low-orbit satellites that have executed the mission is reduced by one, and the corresponding status of the low-orbit satellites for the newly executed mission is updated to the shortest time interval required to execute the same mission. According to constraint C1, a task can only be executed once, so the task set is deleted. According to steps (2) to (4), the state of the agent in the next time slot is obtained. , and the experience value of all agents ( ) is stored in the experience replay pool;

[0119] (6) Repeat step (5) according to the time slot. After multiple rounds of loop execution, select samples from the experience replay pool and The states of the agents are concatenated into observation vectors , concatenate the actions into action vectors , As the input of the online value network, it outputs a one-dimensional output value, namely the Q value, and updates the online value network through the gradient descent method; each agent converts its own local variables As the input of the online policy network, the online policy network is updated by gradient descent. The parameters of the online policy network and the online value network The parameters of the target policy network and the target value network The synchronization between them adopts the soft update method, that is, , based on this, update the target value network and target strategy network;

[0120] (7) After the entire process is completed, the scene is initialized again and training is continued based on the previous round of training until the execution reward converges and a trained multi-agent network is obtained.

[0121] Specifically, the above constraints C1, C2, C3, C4, C5, C6, C7, and C8 are expressed as:

[0122] C1: Each measurement and control task and data transmission task can only be executed once;

[0123] (8);

[0124] in, Represents a set of tasks; Representation Task In the visible time window whether conflicts occur; Representation Task In the visible time window whether it is scheduled for execution; Indicates tasks; Represents a collection of visible time windows.

[0125] C2: TT&C and data transmission data are transmitted between satellites using a one-satellite-four-link method, including one forward and one backward link within the same orbit, and one direction-finding link on the left and right different orbit planes. Although the mission low-orbit satellite can perform TT&C data transmission tasks outside the satellite-to-ground visibility time, the low-orbit satellite selected to send data must meet the satellite-to-ground visibility constraints;

[0126] (9);

[0127] in, Representation Task Start execution time, Representation Task End execution time, Indicates execution of tasks The time window start time, Indicates execution of tasks The time window end time.

[0128] C3: The visible time window length must be greater than the shortest execution time required for the task to ensure that there is sufficient time to complete data reception.

[0129] (10);

[0130] in, Representation Task In the visible time window whether it is scheduled for execution; Representation Task In the visible time window whether conflicts occur; Indicates the end time of the visible time window; Indicates the start time of the visible time window; Indicates the shortest task execution time.

[0131] C4: When the ground station receives TT&C data or sends TT&C instructions to a low-orbit satellite, it can only execute one task within a visible time window, that is, the ground station can execute at most one task at a time;

[0132] (11);

[0133] C5: Equipment that performs telemetry and data transmission tasks must support low-orbit satellite telemetry or data transmission operations;

[0134] (12);

[0135] in, Indicates the task type; Indicates the type of ground station; Representation Task In the visible time window Whether it is scheduled for execution.

[0136] C6: The orbital requirements of the measurement and control mission must be consistent with the mission low-orbit satellite's ascending and descending orbit, and have nothing to do with the low-orbit satellite's ascending and descending orbit state when selecting which low-orbit satellite to use as the target low-orbit satellite to send data; there is no ascending and descending orbit constraint for the data transmission mission;

[0137] (13);

[0138] in, Indicates the orbit status of the mission low-orbit satellite when performing measurement and control tasks. Indicates the orbit type required by the mission.

[0139] C7: The visible time window for establishing the satellite-to-ground link must meet the minimum elevation angle requirement of the mission;

[0140] (14);

[0141] in, Indicates the minimum elevation angle requirement for the mission; Indicates the maximum elevation angle in this time window.

[0142] C8: There must be a certain time interval between the same type of tasks of the same low-orbit satellite, that is, there must be a certain time interval between measurement and control tasks and between data transmission tasks.

[0143] (15);

[0144] in, It indicates the minimum time interval between the execution of the same type of missions by the same low-orbit satellite; Indicates the same type of mission of the same low-orbit satellite The time to perform the task; Indicates the same type of mission of the same low-orbit satellite The time to perform the task; Indicates the maximum time interval between the execution of the same type of missions on the same low-orbit satellite.

[0145] S104, obtaining a routing path between the inbound low-orbit satellite and the mission low-orbit satellite according to the shortest path routing table, and determining a scheduling sequence according to the routing path and the associated ground stations.

[0146] For example, the scene has Each ground station has measurement and control equipment, so at most tasks, set the agent in the scene to The state of each agent should include mission-related information, ground station-related information, low-orbit satellite-related information, and resource-related information. The elements of this information are extracted and shaped. The local observation vector of each agent at the time, the local observation vector Indicates ground station The status of the Indicates ground station Whether to support the planned mission. If the current ground station does not support the mission execution, there are three situations. The first is that the current device is executing other tasks. The second is that the current device does not support the corresponding function. The last is that there is no satellite-to-ground link to support data downlink or command uplink. They respectively represent the low-orbit satellite information visible to the ground station, including the target low-orbit satellite number, the number of hops between the low-orbit satellite and the target low-orbit satellite, Indicates selection If there are not enough visible low-orbit satellites, fill in 0 to ensure the state size is consistent.

[0147] The action space of the agent is expressed as formula (2). When the action space is hour, Indicates not to execute the task. Map the two-dimensional action space to a one-dimensional action space, that is , the corresponding relationship is:

[0148] (16);

[0149] in, Expressed as a ceiling function; Representing an Agent exist The mapping result of time slots; express The total number of visible LEO satellites in the selected time slot; Indicates selection ground stations, Indicates that the ground station The visible A low-orbit satellite.

[0150] The method proposed in this invention needs to continuously optimize relevant strategies to select appropriate actions to achieve efficient scheduling of measurement and control tasks. In order to maximize the long-term task reward, each agent needs to continuously obtain action feedback from the environment and implement state updates. Therefore, according to the goal of this scenario, the reward can be set as:

[0151] (17);

[0152] in, Representation Task In the visible time window whether it is scheduled for execution; represents the gain factor; Representation Task In the visible time window whether conflicts occur; represents the loss factor; Indicates the agent chooses to perform tasks The subsequent income; express Slotted Agent Rewards after performing actions; Representing an Agent exist Parameters of the time slot.

[0153] Number of hops between the source LEO satellite and the destination LEO satellite , Task Priority Related, the relationship is as follows:

[0154] (18);

[0155] in, represents normalization processing, When , it means that the current task directly executes the measurement and control or data transmission task through the satellite-to-ground link without the need for inter-satellite link forwarding.

[0156] In order to describe the task scheduling details in detail, the present invention divides the task scheduling time into The present invention uses intersatellite links to assist in the scheduling of measurement and control data transmission tasks, so it is necessary to obtain an intersatellite routing table. The intersatellite routing method of the present invention adopts the shortest path routing, so it is necessary to first obtain the adjacency matrix of the low-orbit satellite. , low-orbit satellites are connected through one satellite and four links, and the adjacency matrix is ​​as follows:

[0157] (19);

[0158] If a low-orbit satellite With low-orbit satellite If there is an edge between ,otherwise . Obtain inter-satellite routing table based on Floyd algorithm and the number of hops on the shortest path , the basic process is:

[0159] 1) Take the adjacency matrix as input and initialize a path matrix to record the intermediate vertices of the shortest path;

[0160] 2) Iterate each vertex and consider all vertex pairs , check if there is a vertex , so that from go through arrive The path is shorter than the known shortest path. If so, update The distance between them is estimated as the shorter path length;

[0161] 3) Output the shortest path length and intermediate vertices.

[0162] In addition, it is necessary to obtain the visible time window resource set between the giant LEO low-orbit satellite constellation and the ground station through STK (Satellite Tool Kit) In practical applications, the entire visible arc is usually used as a task execution operation. In order to locate the corresponding visible time window resources more quickly, it is only necessary to use a two-dimensional array to record whether the low-orbit satellite and each ground station enter the current time slot. 0 is used to represent a non-entry time slot, and 1 is used to represent a low-orbit satellite entry time slot. Since the entry status of the same time slot is much smaller than the space size of the two-dimensional array, a sparse matrix is ​​introduced. If there is a certain moment Low-orbit satellites enter the country, only A two-dimensional matrix of size Indicates the visible status of the star and the earth at that moment. The number of incoming low-orbit satellites, the number of ground stations and the total number of visible low-orbit satellites are stored in Storage The numbers of the incoming low-orbit satellites and the corresponding ground stations, that is, the low-orbit satellite number, the ground station number, and the number in the visible time window of the corresponding low-orbit satellite and ground station, can be quickly located to obtain the corresponding time window and its attributes, while also avoiding the repeated calculation process of each iteration during the training process. Through the above process, the visibility query sparse matrix can be obtained. Finally, it is necessary to define the ground station status information at the initial moment of the scene , low-orbit satellite status information , Task Collection and the number of agents in the scene ( Less than or equal to the number of ground devices in the scene).

[0163] The present invention proposes a method. First, the measurement and control data transmission task scheduling method is applicable to large-scale low-orbit satellite networks. Secondly, the measurement and control data transmission task scheduling process is expressed as a multi-agent deep reinforcement learning process. The strategy network and the value network are combined to establish a measurement and control data transmission task scheduling model suitable for giant LEO low-orbit satellite constellation scenarios. In addition, the "centralized training, distributed execution" characteristics of the MADDPG algorithm can take into account the impact of multiple ground stations and multiple tasks during task selection, and can improve the scheduling efficiency of measurement and control data transmission tasks in giant low-orbit satellite constellation scenarios, and ensure the timeliness of measurement and control data transmission task allocation.

[0164] (2) The present invention adopts a networked measurement, control and data transmission task scheduling method that combines inter-satellite links and satellite-to-ground links, which solves the problem of low completion rate of measurement, control and data transmission tasks due to the inability to deploy stations globally. At the same time, tasks that originally had to be executed by entering the country have become tasks that can be executed at any time, thereby improving the completion rate of measurement, control and data transmission tasks and resource utilization.

[0165] (3) The present invention describes the constraint satisfaction target model inherent in the scheduling of measurement and control data transmission tasks as a dynamic decision-making process that changes with time series through a multi-agent deep reinforcement learning framework. When the tasks in the environment change, the existing model can be used for training, which greatly shortens the model training time and ensures the scalability of the method.

[0166] The networked measurement and control data transmission task scheduling method proposed in the present invention expresses the measurement and control data transmission task scheduling process as a multi-agent deep reinforcement learning process, and combines the strategy network and the value network to establish a measurement and control data transmission task scheduling model suitable for the giant LEO low-orbit satellite constellation scenario. It takes into account the mutual influence between global tasks and resources and reduces the effectiveness of decision-making during execution, while improving the scheduling efficiency of measurement and control data transmission tasks in the giant low-orbit satellite constellation scenario.

[0167] The networked measurement and control data transmission task scheduling method proposed in the present invention is mainly aimed at giant low-orbit satellite constellations. In this scenario, resource competition is more intense, arc conflicts are greater, and it is difficult to meet large-scale measurement and control data transmission needs by relying solely on ground-based measurement and control, which reduces the task completion rate and resource utilization rate. Based on this scenario, the measurement and control data transmission scheduling method designed by the present invention combines the low-orbit satellite inter-satellite link to realize measurement and control instruction injection and data download, improves resource utilization and task completion rate, and makes tasks that originally require entry to be executed become tasks that can be executed at any time.

[0168] The method proposed in the present invention is combined with a multi-agent deep reinforcement learning framework to characterize the measurement, control and data transmission tasks and their resources in the giant LEO low-orbit satellite constellation scenario, and its task scheduling process is modeled as a reinforcement learning decision-making process that changes over time. The trained model is suitable for the scheduling of measurement, control and data transmission tasks in different scenarios, thereby improving the scalability of the algorithm.

[0169] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. All or part of the present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, mobile communication terminals, multi-processor systems, microprocessor-based systems, programmable electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc.

[0170] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the present invention.

Claims

1. A networked measurement, control and data transmission task scheduling method for giant low-orbit satellite constellations, characterized in that: include: Get the tasks to be planned; The task layer is used to extract elements of the task to be planned to obtain task elements, and the resource layer, equipment status and constellation status are used to extract the status of the measurement and control center, the status of the low-orbit satellite constellation and the status of the ground station to obtain resource attributes, ground station status and low-orbit satellite status; The task elements, the resource attributes, the ground station status and the low-orbit satellite status are respectively converted to obtain input data, and the input data is input into the trained multi-agent network to obtain the action corresponding to the current time slot, and the inbound low-orbit satellite and the associated ground station are determined according to the action; The training process of the multi-agent network includes: The multi-agent network includes: an online strategy network, an online value network, a target strategy network and a target value network; The input of the online policy network is a local observation vector, the output is an action, and the gradient descent method is used to update the parameters of the online policy network through the action to obtain a trained online policy network; The input of the online value network is an observation vector and an action vector, and the output is a one-dimensional output value. The gradient descent method is used to update the parameters of the online value network through the one-dimensional output value to obtain a trained online value network; Using a soft update method to complete the update of the target value network and the target policy network corresponding to the online value network and the online policy network, and obtain a trained target value network and a target policy network; Wherein, the local observation vector is a part of the state of each agent in the current time slot calculated by each agent according to available resources; The available resources are obtained by screening the task elements, resource attributes, ground station status and low-orbit satellite status corresponding to each time slot according to the first constraint condition; the available resources are different in each round of training, and the available resources in each round of training are calculated according to the second constraint condition and the reward calculation formula for each agent, and the available resources are updated according to the reward; here, the reward calculation formula is expressed as: ； in, Indicates the task In the visible time window whether it is scheduled for execution; represents the gain factor; Indicates the task In the visible time window whether conflicts occur; represents the loss factor; Indicates the agent chooses to perform tasks The subsequent income; express Slotted Agent Rewards after performing actions; Representing an Agent exist Parameters of the time slot; The task elements are obtained by performing data processing on the tasks in the training set using the task layer; The resource attributes, the ground station status and the low-orbit satellite status are obtained by extracting the status of the measurement and control center, the status of the low-orbit satellite constellation and the status of the ground station respectively using the resource layer, the equipment status and the constellation status; The observation vector is obtained by concatenating the agent states of each agent in the experience replay pool; The action vector is obtained by concatenating the output actions of each agent in the experience replay pool; The data stored in the experience replay pool is obtained by the online strategy network after multiple rounds of training; The routing path between the inbound low-orbit satellite and the mission low-orbit satellite is obtained according to the shortest path routing table, and a scheduling sequence is determined according to the routing path and the associated ground station.

2. The method for scheduling networked measurement, control and data transmission tasks for giant low-orbit satellite constellations according to claim 1 is characterized in that: The step of extracting elements of the task to be planned by using the task layer to obtain task elements, and extracting the status of the measurement and control center, the low-orbit satellite constellation and the ground station by using the resource layer, the equipment status and the constellation status to obtain resource attributes, the ground station status and the low-orbit satellite status, includes: The task layer includes a plurality of tasks to be planned, and the task types of the tasks to be planned include: measurement and control tasks and data transmission tasks; the task elements include: task type, shortest execution time, task priority and task minimum elevation angle; The resource layer includes visible time window resources and inter-satellite link resources, and the resource attributes include time window start time, time window end time, time window maximum elevation angle and inter-satellite route shortest hop count; The device status includes a ground station status and a ground station type, wherein the ground station status includes: a current execution status of the ground station and a functional abstraction possessed by the ground station; The constellation status includes: a low-orbit satellite status and a link status. The low-orbit satellite status includes: a time interval of the last measurement and control mission performed by the low-orbit satellite and a current execution status of the low-orbit satellite.

3. The networked measurement, control and data transmission task scheduling method for giant low-orbit satellite constellations according to claim 1 is characterized in that: The local observation vector is expressed as: ； in, Indicates whether there is a task; Indicates a mission low-orbit satellite; Indicates the task priority; Indicates ground station Status; Indicates ground station Whether to support tasks to be planned; Respectively represent the ground station Visible low-orbit satellite information; Indicates selection visible low-orbit satellites; express Slotted Agent The local observation vector of Representing an Agent exist Parameters of the time slot; Indicates ground station Status; Indicates ground station Whether to support tasks to be planned; Indicates that the ground station Visible low-orbit satellite information.

4. The networked measurement, control and data transmission task scheduling method for giant low-orbit satellite constellations according to claim 1 is characterized in that: The action is a two-dimensional tuple, which is represented by: ； in, Indicates selection ground stations; Indicates that the ground station The visible Low Earth Orbit Satellites; express Slotted Agent Actions; Representing an Agent exist Parameters of the time slot.

5. The networked measurement, control and data transmission task scheduling method for giant low-orbit satellite constellations according to claim 1 is characterized in that: The step of determining an incoming low-orbit satellite and an associated ground station according to the action includes: Mapping the action into a one-dimensional action space to obtain a mapping result; The corresponding relationship between the action and the mapping result is expressed as: ； in, Expressed as a ceiling function; Representing an Agent exist The mapping result of time slots; express The total number of visible LEO satellites in the selected time slot; Indicates selection ground stations; Indicates that the ground station The visible Low Earth Orbit Satellites; express Slotted Agent action.

6. The networked measurement, control and data transmission task scheduling method for giant low-orbit satellite constellations according to claim 1 is characterized in that: The objective function of the networked measurement and control data transmission task scheduling method is expressed as: ； in, Represents a set of tasks; represents the set of ground stations; Represents a set of visible time windows; represents the gain factor; Indicates the task In the visible time window whether it is scheduled for execution; represents the loss factor; Indicates the task In the visible time window whether conflicts occur; Indicates the agent chooses to perform tasks After the income.

Images