Satellite scheduling method and device based on reinforcement learning, equipment and storage medium

Abstract

Embodiments of the present disclosure provide a satellite beam scheduling method and device based on reinforcement learning, equipment and a storage medium. The method comprises: constructing a state space comprising the satellite identifier of each satellite in a navigation satellite constellation, the pitch angle of each satellite relative to a target area, the direction angle of each satellite relative to the target area, and the relative distance of each satellite to the target area; constructing an action space comprising a satellite identifier sequence of a satellite to be scheduled; designing a reward function based on a reward index comprising a coverage multiplicity reward and a PDOP reward; training a neural network for satellite beam scheduling based on a TD3 algorithm, according to the state space, the action space and the reward function, to obtain a trained policy network; inputting a real-time state vector of the target area into the trained policy network to obtain a real-time action, so as to control all satellite beams corresponding to the real-time action to point to the target area. The method can improve the service quality of the satellite.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of satellite navigation technology, and more specifically, to a satellite scheduling method, apparatus, device, and storage medium based on reinforcement learning. Background Technology

[0002] In modern satellite navigation systems, ground users are increasingly demanding higher accuracy, stability, and coverage for positioning services. To achieve high-precision positioning, it is necessary to ensure that a sufficient number of satellites cover a specific area at any given time, while keeping the Position Dilution of Precision (PDOP) within a low range.

[0003] Traditional satellite beam scheduling methods are mostly based on pre-set planning or simple optimization algorithms, such as scheduling rules based on fixed priorities. These methods are difficult to fully balance some interrelated and conflicting objectives such as coverage multiples and PDOP, resulting in a decrease in the quality of satellite services. Summary of the Invention

[0004] The embodiments described herein provide a reinforcement learning-based satellite scheduling method, apparatus, device, and storage medium that can balance the conflicting objectives of coverage multiplicity and PDOP, adaptively optimize satellite beam pointing, and thereby improve the quality of service of satellites.

[0005] Firstly, this disclosure provides a satellite scheduling method based on reinforcement learning, including:

[0006] A state space and an action space are constructed; a reward function is designed based on a reward metric; a satellite beam scheduling neural network is trained based on the TD3 algorithm according to the state space, the action space, and the reward function to obtain a trained policy network, which includes the policy network and a value network; the real-time state vector of the target area is input into the trained policy network to obtain real-time actions, thereby controlling all satellite beams corresponding to the real-time actions to point to the target area.

[0007] The state space includes the satellite identifiers of multiple satellites in the navigation satellite constellation, the elevation angle of each satellite relative to the target area, the azimuth angle of each satellite relative to the target area, and the relative distance between each satellite and the target area. The action space includes the sequence of satellite identifiers of the satellites to be scheduled. The reward indicators include coverage multiple reward and position accuracy attenuation factor (PDOP) reward.

[0008] In some embodiments of this disclosure, the step of training a satellite beam scheduling neural network based on the TD3 algorithm, according to the state space, the action space, and the reward function, to obtain a trained policy network includes:

[0009] The current state vector is input into the policy network to obtain the current action. The state space includes the state vector, and the action space includes the action. In the next moment, all satellite beams corresponding to the current action are controlled to point towards the target area. The next moment's state vector, the next moment's coverage multiplicity of the target area, and the next moment's PDOP are obtained. The current moment's reward value is determined based on the next moment's coverage multiplicity, the next moment's PDOP, and the reward function. The current moment's state vector, the current moment's action, the current moment's reward value, and the next moment's state vector are stored as a set of training samples in the experience replay pool. Multiple sets of training samples are randomly selected from the experience replay pool, and each set of training samples is input into the neural network. The network parameters of the neural network are updated based on the TD3 algorithm to obtain the trained policy network.

[0010] In some embodiments of this disclosure, updating the network parameters of the neural network based on the TD3 algorithm to obtain the trained policy network includes:

[0011] The current state vector and current action from each group of training samples are input into the value network to obtain an estimated value. The network parameters of the value network are adjusted by minimizing the error between the estimated value and the target value. Based on the feedback from the value network, the network parameters of the policy network are adjusted, and the process of randomly selecting multiple groups of training samples from the experience replay pool is repeated until the convergence condition is met and training ends, resulting in a trained policy network.

[0012] In some embodiments of this disclosure, the state space also includes the historical and future durations of continuous visibility of each of the satellites over the target area.

[0013] In some embodiments of this disclosure, the state space also includes the selection state information of each satellite at the previous moment.

[0014] In some embodiments of this disclosure, the reward metric also includes a satellite handover reward.

[0015] The reward function designed based on reward metrics includes:

[0016] The coverage multiple reward, the PDOP reward, and the satellite handover number reward are weighted separately; the weighted coverage multiple reward, PDOP reward, and satellite handover number reward are summed to obtain the reward function.

[0017] In some embodiments of this disclosure, when the coverage weight value is greater than or equal to a first preset value, the coverage weight reward value is positive; when the coverage weight value is less than the first preset value, the coverage weight reward value is negative; and the coverage weight reward value is positively correlated with the coverage weight value.

[0018] When the PDOP value is less than or equal to the second preset value, the PDOP reward value is positive; when the PDOP value is greater than the second preset value, the PDOP reward value is negative, and the PDOP reward value is negatively correlated with the PDOP value.

[0019] When the number of satellite handovers is less than or equal to a third preset value, the satellite handover reward value is positive; when the number of satellite handovers is greater than the third preset value, the satellite handover reward value is negative, and the satellite handover reward value is negatively correlated with the number of satellite handovers.

[0020] Secondly, this disclosure provides a satellite scheduling device based on reinforcement learning, comprising:

[0021] A module is established to construct a state space and an action space. The state space includes the satellite identifiers of multiple satellites in the navigation satellite constellation, the elevation angles of each satellite relative to the target area, the azimuth angles of each satellite relative to the target area, and the relative distances of each satellite to the target area. The action space includes a sequence of satellite identifiers of satellites to be scheduled.

[0022] The design module is used to design a reward function based on reward metrics, which include coverage multiplicity reward and location accuracy decay factor (PDOP) reward.

[0023] The training module is used to train a satellite beam scheduling neural network based on the TD3 algorithm, according to the state space, the action space and the reward function, to obtain a trained policy network, wherein the neural network includes the policy network and the value network.

[0024] The scheduling module is used to input the real-time state vector of the target area into the trained policy network to obtain real-time actions, so as to control all satellite beams corresponding to the real-time actions to point to the target area.

[0025] Thirdly, this disclosure provides an electronic device including a processor for executing a computer program stored in a memory, wherein the computer program, when executed by the processor, implements the steps of any of the methods provided in the first aspect.

[0026] Fourthly, this disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the methods provided in the first aspect.

[0027] The technical solution provided in this disclosure constructs a state space including the satellite identifiers of multiple satellites in the navigation satellite constellation, the elevation angles of each satellite relative to the target area, the azimuth angles of each satellite relative to the target area, and the relative distances of each satellite to the target area. It also constructs an action space including a sequence of satellite identifiers for the satellites to be scheduled. A reward function is designed based on reward metrics including coverage multiplicity reward and PDOP reward. Using the TD3 algorithm, a neural network for satellite beam scheduling is trained according to the state space, action space, and reward function to obtain a trained policy network. The real-time state vector of the target area is input into the trained policy network to obtain real-time actions, thereby controlling all satellite beams corresponding to the real-time actions to point towards the target area. This approach balances the conflicting objectives of coverage multiplicity and PDOP, and allows for continuous learning and timely adjustment of the satellite beam scheduling strategy, adaptively optimizing satellite beam pointing and thus improving the quality of service of the satellites. Attached Figure Description

[0028] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. It should be understood that the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure, wherein:

[0029] Figure 1 A flowchart illustrating a satellite beam scheduling method based on reinforcement learning provided in this disclosure embodiment;

[0030] Figure 2 This is a schematic diagram of a TD3 algorithm framework provided in an embodiment of this disclosure.

[0031] Figure 3 This is a flowchart illustrating another satellite beam scheduling method based on reinforcement learning provided in an embodiment of this disclosure.

[0032] Figure 4 A schematic diagram illustrating the convergence of the reward function during the training process provided for this disclosure.

[0033] Figure 5 A schematic diagram of a satellite beam scheduling device based on reinforcement learning provided for the implementation of this disclosure.

[0034] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0035] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are also within the scope of protection of this disclosure.

[0036] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this subject matter pertains. It will be further understood that terms such as those defined in commonly used dictionaries shall be interpreted as having meanings consistent with their meanings in the context of the specification and in the relevant art, and shall not be interpreted in an idealized or overly formal form unless otherwise explicitly defined herein. Furthermore, terms such as “first” and “second” are used only to distinguish one component (or part of a component) from another component (or another part of a component).

[0037] Figure 1 A flowchart illustrating a reinforcement learning-based satellite beam scheduling method provided in this disclosure is shown below. Figure 1 As shown, the specific steps of the reinforcement learning-based satellite beam scheduling method include:

[0038] S101, construct the state space and action space.

[0039] The state space includes the satellite identifiers of multiple satellites in the navigation satellite constellation, the elevation angle of each satellite relative to the target area, the azimuth angle of each satellite relative to the target area, and the relative distance between each satellite and the target area. The action space includes the sequence of satellite identifiers of the satellites to be scheduled.

[0040] For example, the elevation angle of the satellite relative to the target area describes the angle information in the vertical direction when the satellite beam is pointing to the target area, and the azimuth angle of the satellite relative to the target area describes the angle information in the horizontal direction when the satellite beam is pointing to the target area. Together, they determine the accurate direction of the satellite beam in space. By accurately measuring and utilizing the elevation angle and azimuth angle, the satellite beam can more accurately cover the target area, avoiding insufficient coverage or waste of resources caused by beam deviation.

[0041] Satellite identifiers are used to uniquely identify each satellite in a navigation satellite constellation, facilitating targeted operation and management of different satellites. The relative distance between a satellite and a target area describes their relative positions, masking scale inconsistencies caused by changes in absolute position. This provides reliable position information for algorithms, aiding in the assessment of satellite signal propagation strength and stability. During satellite beam scheduling, allocating resources rationally based on the relative distance between the satellite and the target area ensures that the target area receives high-quality satellite signals.

[0042] In practical applications, for each ground target area requiring service, satellite orbit data and the target area's location information can be used, along with mathematical methods such as trigonometric functions, to calculate the elevation and azimuth angles of each satellite relative to the target area. Simultaneously, the relative distance between the target area and the satellites can be calculated using spatial distance formulas. The calculated elevation, azimuth, and relative distances are then organized according to the definition of state space to form state vectors. These state vectors serve as the input to the agent, and together they constitute the state space.

[0043] For example, a navigation satellite constellation includes N satellites, and the satellite identifier of each satellite can be IDi, where i is an integer greater than 1 and less than or equal to N. For a certain target area, the elevation angle of the satellite corresponding to the satellite identifier IDi relative to the target area is αi, the azimuth angle is βi, and the relative distance to the target area is Li. Then the resulting state vector si is [IDi, αi, βi, Li].

[0044] The action space is defined as the sequence of satellite identifiers selected at a given moment to provide beam coverage for the selected satellite; that is, the sequence of satellite identifiers to be scheduled. This design closely aligns with the actual operational process of satellite beam scheduling. By selecting an appropriate satellite identifier sequence, the direction of the satellite beams can be directly controlled, achieving precise coverage of the target area. When it is determined that a specific number and combination of satellite beams are needed to cover a target area, the corresponding action, i.e., the satellite identifier sequence, is selected from the action space. This allows the satellites corresponding to the satellite identifier sequence to adjust their beams to point towards the target area, thus completing the satellite beam coverage task.

[0045] For example, based on the above embodiments, the action ai corresponding to the state vector si is [IDm, IDj, IDn], where m, j, and n are integers greater than 1 and less than or equal to N, and m, j, and n are different.

[0046] S102, Design a reward function based on reward indicators.

[0047] For example, the reward indicators include coverage weight reward and PDOP reward. The coverage weight reward is closely related to the coverage weight of the target area. When the coverage weight value is greater than or equal to a first preset value, the coverage weight reward value is positive. When the coverage weight value is less than the first preset value, the coverage weight reward value is negative. The coverage weight reward value is positively correlated with the coverage weight value, so as to enable more satellite beams to cover the target area, enhance the strength and stability of satellite signals, and improve positioning accuracy.

[0048] For example, when the coverage weight value reaches 6, a positive reward value for the base coverage weight is obtained. When the coverage weight value increases to 8, the reward value is increased on top of the positive reward value for the base coverage weight to incentivize the algorithm to maximize the coverage weight of the target area. Conversely, when the coverage weight value is less than 6, the positive reward value for the base coverage weight is reduced to obtain a negative reward value for the coverage weight, prompting the algorithm to adjust its satellite beam scheduling strategy and increase the satellite coverage weight.

[0049] The PDOP reward is related to the PDOP of the target area. When the PDOP value is less than or equal to the second preset value, the PDOP reward value is positive. When the PDOP value is greater than the second preset value, the PDOP reward value is negative. The PDOP reward value is negatively correlated with the PDOP value, so as to reduce the PDOP of the target area and meet the user's need for high-precision positioning.

[0050] For example, when the PDOP value reaches 3, a base PDOP positive reward value is obtained. When the PDOP value decreases to 2, the reward value is increased on top of the base PDOP positive reward value to incentivize the algorithm to reduce the PDOP in the target area as much as possible. Conversely, when the PDOP value is greater than 3, it is decreased on top of the base PDOP positive reward value to obtain a PDOP negative reward value, guiding the algorithm to adjust the satellite beam scheduling strategy and reduce the PDOP.

[0051] The reward function is obtained by weighting the coverage multiplicity reward and the PDOP reward separately, and then summing the weighted coverage multiplicity reward and the PDOP reward. Therefore, the weighted sum of the coverage multiplicity reward value and the PDOP reward value is the reward value of the reward function.

[0052] S103, based on the TD3 algorithm, trains a neural network for satellite beam scheduling according to the state space, action space, and reward function to obtain a trained policy network.

[0053] Neural networks include policy networks and value networks.

[0054] For example, before training, key parameters of the neural network need to be set, such as the number of layers and the number of neurons in each layer, to ensure that the neural network can effectively learn and process complex state information and action decisions. An appropriate learning rate needs to be set, as this parameter controls the step size of the TD3 algorithm's updates during the learning process. An excessively large learning rate may lead to algorithm instability, while an excessively small learning rate will result in a slow learning speed. Setting an appropriate learning rate can balance the algorithm's learning speed and stability.

[0055] A reasonable discount factor needs to be set, which determines how much importance the agent places on future rewards. A higher discount factor means the agent focuses more on long-term rewards, while a lower discount factor means the agent focuses more on short-term immediate rewards. The update frequency also needs to be determined, i.e., how often the policy network and value network should be updated.

[0056] Figure 2 This is a schematic diagram of a TD3 algorithm framework provided in an embodiment of this disclosure, as shown below. Figure 2 As shown, the TD3 algorithm includes a policy network and a value network. Based on its current state, the agent carefully selects an action from the action space, i.e., a sequence of satellite identifiers, using the policy network. For example, at a certain moment, the agent perceives the elevation angle of the satellite relative to the target area, the azimuth angle of the satellite relative to the target area, and the relative distance between the satellite and the target area in the current state space. The agent then evaluates various possible actions in the action space through the policy network and selects the optimal sequence of satellite identifiers as the corresponding action.

[0057] After executing the selected action, the environment evaluates the agent's behavior according to the reward function and assigns a corresponding reward value. Simultaneously, the environment provides feedback on new state information, including the coverage multiplicity of the target area after satellite beam adjustment, PDOP, the adjusted satellite elevation angle relative to the target area, the adjusted satellite azimuth angle, and the adjusted relative distance between the satellite and the target area. The agent records these states, actions, rewards, and experiences with the new states, storing them in an experience replay pool. As data accumulates in the experience replay pool, the agent periodically extracts a batch of data and updates the policy network and value network using the update rules of the TD3 algorithm.

[0058] like Figure 2 As shown, the TD3 algorithm also includes a target value network and a target policy network. The network parameters of the target value network and the target policy network are not updated in real time, but are updated periodically based on the value network and the policy network. For example, the target value network and the target policy network are periodically updated based on the value network and the policy network.

[0059] For example, a soft update coefficient is set, and at regular time steps, the network parameters of the value network are gradually integrated into the target value network according to the soft update coefficient, and the network parameters of the policy network are gradually integrated into the target policy network according to the soft update coefficient. This allows the target value network and the target policy network to gradually absorb the learning results of the value network and the policy network while maintaining relative stability, thus avoiding instability caused by the algorithm being updated too frequently.

[0060] For example, if the soft update coefficient is 0.005, the time step is 100, a certain network parameter of the policy network is θ, and the corresponding network parameter of the target policy network is θ', then during the update, the network parameter of the target policy network will be updated to (1-0.005)*θ'+0.005*θ.

[0061] Through this continuous learning and optimization process, the agent gradually masters the optimal satellite beam adjustment strategy that simultaneously satisfies coverage multiplicity and PDOP requirements, achieving intelligent and efficient scheduling of navigation satellite constellation beams. With each training iteration, the agent learns even better satellite beam scheduling strategies; each training session optimizes the previous strategy, and the agent continuously adjusts the strategy based on new data. For example, after 1000 training sessions, the agent's action selection under different states becomes more accurate, better balancing the conflicting objectives of coverage multiplicity and PDOP.

[0062] S104: Input the real-time state vector of the target area into the trained policy network to obtain the real-time action, so as to control all satellite beams corresponding to the real-time action to point to the target area.

[0063] For example, during actual communication between the target area and the satellites, a real-time state vector of the target area is acquired. This real-time state vector includes the satellite identifiers of each satellite in the navigation satellite constellation, the real-time elevation angles of each satellite relative to the target area, the real-time azimuth angles of each satellite relative to the target area, and the real-time relative distances between each satellite and the target area. This real-time state vector is then input into a trained policy network. Based on this policy network, a real-time action is output, which is a sequence of satellite identifiers for the satellites to be scheduled. At the next moment, all satellite beams corresponding to the real-time action are scheduled to point towards the target area, enabling all satellites corresponding to the real-time action to provide services to the target area.

[0064] In this embodiment, a state space is constructed that includes the satellite identifiers of multiple satellites in the navigation satellite constellation, the elevation angles of each satellite relative to the target area, the azimuth angles of each satellite relative to the target area, and the relative distances of each satellite to the target area. An action space is constructed that includes the sequence of satellite identifiers of the satellites to be scheduled. A reward function is designed based on reward indicators including coverage multiple reward and PDOP reward. Based on the TD3 algorithm, a neural network for satellite beam scheduling is trained according to the state space, action space, and reward function to obtain a trained policy network. The real-time state vector of the target area is input into the trained policy network to obtain real-time actions, thereby controlling all satellite beams corresponding to the real-time actions to point to the target area. This can balance the two conflicting objectives of coverage multiple and PDOP, and can also continuously learn and adjust the satellite beam scheduling strategy in a timely manner to adaptively optimize satellite beam pointing, thereby improving the quality of service of the satellites.

[0065] In some embodiments, Figure 3 This is a flowchart illustrating another satellite beam scheduling method based on reinforcement learning provided in an embodiment of this disclosure. Figure 3 for Figure 1 Based on the illustrated embodiment, a specific description of one possible implementation of S103 is as follows:

[0066] S201, input the current state vector into the policy network to obtain the current action.

[0067] For example, the agent first obtains the current state vector from the state space. This current state vector includes the satellite identifiers of each satellite in the navigation satellite constellation, the elevation angle of each satellite relative to the target area at the current time, the azimuth angle of each satellite relative to the target area at the current time, and the relative distance between each satellite and the target area at the current time. Then, a policy network evaluates each action in the action space, outputting a probability distribution for each action. For instance, the output layer of the policy network is a softmax function, which transforms the policy network's evaluation values ​​for different actions into a probability distribution, ensuring that the sum of the probabilities is 1. Each probability value represents the likelihood of the corresponding action being selected.

[0068] Then, an ε-greedy strategy is adopted, which randomly selects an action from the action space with a probability of ε, explores the regions in the action space that have not yet been fully discovered, and selects the action with the highest probability in the probability distribution with a probability of 1-ε, that is, the action that the policy network considers to be optimal. By making full use of the learned experience knowledge, the agent can obtain the action at the current moment. While continuously exploring new strategies, the agent can gradually optimize and consolidate the existing effective strategies.

[0069] For example, if ε is 0.1, the agent has a 10% probability of randomly choosing an action and a 90% probability of choosing the action with the highest probability.

[0070] S202, at the next moment, control all satellite beams corresponding to the action at the current moment to point to the target area.

[0071] For example, based on the actions taken at the current moment, the ground control center sends precise control commands to the corresponding satellite at the next moment to control the satellite to adjust its beam to point towards the target area. This process requires ensuring the accuracy and timeliness of command transmission, as well as the satellite's reliable execution of the commands. For instance, the ground control center uses a high-precision communication link to send control commands containing detailed information such as satellite identification and beam adjustment angles to the satellite. After receiving the commands, the satellite precisely adjusts its beam to point towards the target area through its internal attitude adjustment and beam control devices.

[0072] S203, obtain the next time-state vector, the next time-cover multiplicity of the target region, and the next time-PDOP.

[0073] For example, after the satellite beam performs its action, the environment obtains the actual coverage weight and PDOP of the target area through ground monitoring stations and sensors carried by the satellite itself, i.e., the coverage weight and PDOP at the next moment. Simultaneously, based on the actual situation after the satellite beam's action, the environment recalculates and feeds back a new state vector, i.e., the state vector at the next moment. This state vector includes the satellite identifiers of each satellite in the navigation satellite constellation, the adjusted elevation angles of each satellite relative to the target area, the adjusted azimuth angles of each satellite relative to the target area, and the adjusted relative distances between each satellite and the target area. The state vector at the next moment reflects the state change after the satellite beam performs its action at the current moment, providing an information basis for the agent's action decision at the next moment.

[0074] S204, determine the reward value at the current time step based on the next time step coverage multiplicity, the next time step PDOP, and the reward function.

[0075] For example, the environment substitutes the next time-step coverage multiplicity and the next time-step PDOP into the reward function to calculate the reward value of the reward function, which is the reward value of the current time step. For instance, if the next time-step coverage multiplicity is 8 and the next time-step PDOP is 2, a higher reward value for the current time step is obtained by combining the calculation rules of coverage multiplicity reward and PDOP reward in the reward function. Conversely, if the next time-step coverage multiplicity is less than 6 and the next time-step PDOP is greater than 3, a lower reward value for the current time step is obtained by combining the calculation rules of coverage multiplicity reward and PDOP reward in the reward function.

[0076] S205, store the current state vector, current action, current reward value and next state vector as a set of training samples into the experience replay pool.

[0077] For example, the first step is to create an experience replay pool. The pool can be designed using advanced data structures to efficiently store and retrieve data, providing rich data support for subsequent learning and policy updates. For instance, a circular queue can be used, automatically overwriting the oldest set of training samples when a new set is generated, ensuring that the data in the experience replay pool remains at a certain size and is timely.

[0078] Secondly, the current state vector, current action, current reward value, and next state vector are stored as a set of training samples in the experience replay pool to obtain a set of training samples. By repeatedly executing S201 to S205, multiple sets of training samples can be stored in the experience replay pool to obtain multiple sets of training samples. Furthermore, when the experience replay pool is full, the training samples in the experience replay pool can be updated in the order of their storage time from front to back.

[0079] S206, randomly select multiple sets of training samples from the experience replay pool.

[0080] For example, a certain number of training samples are randomly sampled from the experience replay pool according to certain random rules to ensure the randomness and representativeness of the sampling and avoid the impact of sampling bias on the learning effect. The random rules can be various random sampling methods, such as simple random sampling and stratified sampling. For example, simple random sampling can be used to randomly select 100 sets of training samples from the experience replay pool.

[0081] S207. Input each group of training samples into the neural network, update the network parameters of the neural network based on the TD3 algorithm, and obtain the trained policy network.

[0082] For example, the current state vector and current action from each group of training samples are input into the value network, and the value of the current state vector-current action pair is estimated based on the value network to obtain the estimated value.

[0083] Subsequently, the network parameters of the value network are adjusted by minimizing the error between the estimated value and the target value, such as the mean squared error. For example, in a certain training round, the estimated value obtained by the value network for the current state vector-current action pair is V1, while the target value calculated by the TD3 algorithm is V2. The error between the estimated value V1 and the target value V2 is calculated using the mean squared error formula, and then the weights of the value network are adjusted by backpropagation to make the estimated value V1 closer to the target value V2.

[0084] Then, based on the feedback from the value network, the network parameters of the policy network are adjusted so that the generated actions can obtain higher long-term cumulative rewards as the update direction, so as to generate actions that maximize long-term rewards in this state.

[0085] After training on all randomly selected training samples in a single iteration, it is determined whether the convergence condition is met. The convergence condition can be either that the training iterations have reached a preset number, or that the reward function has converged. For example... Figure 4 A schematic diagram illustrating the convergence of the reward function during the training process provided for this disclosure is shown below. Figure 4 As shown, the reward function begins to converge when the number of training iterations exceeds 5000. If the convergence condition is not met, the process returns to step S206 until the convergence condition is met, at which point the training ends, and the trained policy network is obtained.

[0086] In some embodiments, the state space also includes the historical and future durations of continuous visibility of each satellite over the target area.

[0087] For example, the historical continuous visibility duration of a satellite over a target area records the duration during which the satellite has been continuously visible over the target area before the current moment. By taking this information into account, satellites with longer visibility durations over the target area can be prioritized during the scheduling process, reducing unnecessary satellite switching, maintaining the continuity of satellite missions, and improving the stability and reliability of satellite services.

[0088] The forecast of the future visibility duration of a satellite over a target area indicates the satellite's visibility over the target area in the future. With this information, the timing of satellite beam switching can be planned in advance to avoid coverage interruptions caused by the satellite becoming invisible, thus ensuring the continuity of satellite positioning services.

[0089] In practical applications, advanced orbit prediction models and signal propagation analysis techniques can accurately record the duration of satellite visibility over a target area before the current moment (i.e., the historical duration of satellite visibility over the target area) and the duration of satellite visibility over the target area after the current moment (i.e., the future duration of satellite visibility over the target area). Predicting the future duration of satellite visibility over a target area requires precise simulation of the satellite's orbital motion and consideration of the impact of factors such as Earth's obstruction and atmospheric refraction on signal propagation. For example, by establishing a complex satellite orbital dynamics model and combining it with real-time Earth attitude data and atmospheric parameters, the visibility of the satellite over the target area in the future can be predicted, thereby accurately recording the future duration of satellite visibility over the target area.

[0090] Based on the above embodiments, the elevation angle of the satellite relative to the target area, the elevation azimuth angle of the satellite relative to the target area, the relative distance between the satellite and the target area, the historical continuous visibility duration of the satellite to the target area, and the future continuous visibility duration of the satellite to the target area are organized in an orderly manner according to the definition of the state space to form a state vector.

[0091] In this embodiment of the disclosure, by constructing a state space that includes the historical and future durations of continuous visibility of the satellite over the target area, unnecessary satellite switching can be reduced, the continuity of satellite missions can be maintained, the stability and reliability of satellite services can be improved, coverage interruptions caused by the satellite becoming invisible can be avoided, the continuity of satellite positioning services can be guaranteed, and the quality of satellite services can be further improved.

[0092] In some embodiments, the state space also includes the selected state information of each satellite in the previous moment.

[0093] For example, the selection status information is identified using binary, where 0 represents not selected and 1 represents selected, providing a continuity constraint for action selection. By referring to the satellite selection status information of the previous moment, frequent switching of satellite beams can be effectively avoided, reducing losses and improving resource utilization. For instance, if a satellite was selected in the previous moment and the coverage of the target area was good, the algorithm tends to continue selecting that satellite in the current moment, maintaining its service to the target area, unless there are special circumstances.

[0094] Based on the above embodiments, a specific description of a possible implementation of S102 is as follows:

[0095] The coverage multiple reward, PDOP reward, and satellite handover reward are weighted separately, and the weighted coverage multiple reward, PDOP reward, and satellite handover reward are summed to obtain the reward function. Thus, the weighted sum of the coverage multiple reward value, PDOP reward value, and satellite handover reward value is the reward value of the reward function.

[0096] Among them, the satellite handover reward is related to the number of satellite handovers. When the number of satellite handovers is less than or equal to the third preset value, the satellite handover reward value is positive. When the number of satellite handovers is greater than the third preset value, the satellite handover reward value is negative. The satellite handover reward value is negatively correlated with the number of satellite handovers, so as to reduce unnecessary satellite handovers and improve the utilization efficiency and stability of satellite resources.

[0097] Throughout the mission, a negative reward value is given for each satellite switch to incentivize the algorithm to maintain the continuity of the satellite mission. For example, within a mission period, when the number of satellite switches is kept at a relatively low level, the reward value for the number of satellite switches is higher. Conversely, when the number of satellite switches is too high, the reward value for the number of satellite switches will be significantly reduced.

[0098] In this embodiment, by constructing a state space that includes the selected state information of each satellite at the previous moment, and designing a reward function based on a reward index that includes satellite switching rewards, the three conflicting objectives of coverage multiples, PDOP, and satellite switching times can be balanced, further improving the service quality of the satellites. It can also reduce the number of satellite switching times, thereby reducing the wear and tear on the equipment caused by frequent switching, extending the equipment's service life, and improving the utilization efficiency of satellite resources. This enables the satellites to provide stable and efficient continuous and reliable services to the target area, reducing operating costs and enhancing the economic and social benefits of the satellite navigation system.

[0099] In some embodiments, the method further includes the following steps before performing S103:

[0100] Based on the operational status of the navigation satellite constellation and changes in the demand of the target area, the parameters of the TD3 algorithm and the task priority of the target area are dynamically adjusted. For example, when some satellites in the navigation satellite constellation malfunction, the selection probability of these malfunctioning satellites in the action space is reduced accordingly, while the weight of normal satellites is increased. When the positioning demand of a certain target area suddenly increases, the task priority of that target area is increased, so that the algorithm prioritizes meeting the coverage multiple and PDOP requirements of that target area when allocating resources. Through this dynamic adjustment mechanism, the scheduling effect is further optimized, ensuring that the satellite navigation system can adapt to complex and ever-changing application scenarios.

[0101] This disclosure also provides a satellite beam scheduling device based on reinforcement learning. Figure 5 A schematic diagram of a satellite beam scheduling device based on reinforcement learning provided in this disclosure embodiment is shown below. Figure 5 As shown, the satellite beam scheduling device based on reinforcement learning includes:

[0102] Module 110 is established to construct the state space and action space. The state space includes the satellite identifiers of multiple satellites in the navigation satellite constellation, the elevation angles of each satellite relative to the target area, the azimuth angles of each satellite relative to the target area, and the relative distances of each satellite to the target area. The action space includes the sequence of satellite identifiers of the satellites to be scheduled.

[0103] Design module 120 is used to design a reward function based on reward metrics, which include coverage multiplicity reward and location accuracy decay factor (PDOP) reward.

[0104] Training module 130 is used to train a satellite beam scheduling neural network based on the TD3 algorithm, according to the state space, action space and reward function, to obtain a trained policy network. The neural network includes a policy network and a value network.

[0105] The scheduling module 140 is used to input the real-time state vector of the target area into the trained policy network to obtain real-time actions, so as to control all satellite beams corresponding to the real-time actions to point to the target area.

[0106] In some embodiments, the training module 130 is further configured to input the current state vector into the policy network to obtain the current action, wherein the state space includes the state vector and the action space includes the action; at the next time step, control all satellite beams corresponding to the current action to point to the target area; obtain the next time step state vector, the next time step coverage multiplicity of the target area, and the next time step PDOP; determine the current time step reward value based on the next time step coverage multiplicity, the next time step PDOP, and the reward function; store the current time step state vector, the current time step action, the current time step reward value, and the next time step state vector as a set of training samples in the experience replay pool; randomly select multiple sets of training samples from the experience replay pool, input each set of training samples into the neural network, update the network parameters of the neural network based on the TD3 algorithm, and obtain the trained policy network.

[0107] In some embodiments, the training module 130 is further configured to input the current state vector and current action from each group of training samples into the value network to obtain the estimated value; adjust the network parameters of the value network by minimizing the error between the estimated value and the target value; adjust the network parameters of the policy network based on the feedback of the value network; and return to execute the random selection of multiple groups of training samples from the experience replay pool until the convergence condition is met and the training ends, thereby obtaining the trained policy network.

[0108] In some embodiments, the state space also includes the historical and future durations of continuous visibility of each satellite over the target area.

[0109] In some embodiments, the state space also includes the selected state information of each satellite in the previous moment.

[0110] In some embodiments, the reward metric may also include a reward for the number of satellite handovers.

[0111] The design module 120 is further used to weight the coverage multiple reward, PDOP reward and satellite handover number reward respectively; and to sum the weighted coverage multiple reward, PDOP reward and satellite handover number reward to obtain the reward function.

[0112] In some embodiments, when the coverage weight value is greater than or equal to a first preset value, the coverage weight reward value is positive; when the coverage weight value is less than the first preset value, the coverage weight reward value is negative; and the coverage weight reward value is positively correlated with the coverage weight value.

[0113] When the PDOP value is less than or equal to the second preset value, the PDOP reward value is positive; when the PDOP value is greater than the second preset value, the PDOP reward value is negative, and the PDOP reward value is negatively correlated with the PDOP value.

[0114] When the number of satellite handovers is less than or equal to the third preset value, the satellite handover reward value is positive; when the number of satellite handovers is greater than the third preset value, the satellite handover reward value is negative, and the satellite handover reward value is negatively correlated with the number of satellite handovers.

[0115] In some embodiments, the reinforcement learning-based satellite beam scheduling apparatus further includes:

[0116] The adjustment module is used to dynamically adjust the parameters of the TD3 algorithm and the task priority of the target area based on the operation of the navigation satellite constellation and changes in the needs of the target area.

[0117] The apparatus provided in this disclosure is used to perform the steps provided in any of the above method embodiments, has the functional modules corresponding to the method embodiments, and has the beneficial effects of the method embodiments, which will not be repeated here.

[0118] This disclosure also provides an electronic device, including: a processor, the processor being configured to execute a computer program stored in a memory, the computer program being executed by the processor to implement the steps of the method embodiments of this disclosure.

[0119] Figure 6 This is a schematic diagram of the structure of an electronic device provided in this disclosure. Figure 6 A block diagram is shown that is suitable for implementing embodiments of the present disclosure. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0120] like Figure 6 As shown, the electronic device 12 is represented in the form of a general-purpose computing device. The components of the electronic device 12 may include, but are not limited to: one or more processors 16, system memory 28, and bus 18 connecting different system components (including system memory 28 and processor 16).

[0121] Bus 18 represents one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. For example, these architectures include, but are not limited to, the Industry Standard Architecture (ISA) bus, the Micro Channel Architecture (MAC) bus, the Enhanced ISA bus, the Video Electronics Standards Association (VESA) local bus, and the Peripheral Component Interconnect (PCI) bus.

[0122] Electronic device 12 typically includes a variety of computer system readable media. These media can be any media that can be accessed by electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0123] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write non-removable, non-volatile magnetic media (commonly referred to as "hard disk drives"). Disk drives for reading and writing to removable non-volatile disks (e.g., "floppy disks") and optical disk drives for reading and writing to removable non-volatile optical disks (e.g., CD-ROMs, DVD-ROMs, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. System memory 28 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of various embodiments of this disclosure.

[0124] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. Program modules 42 typically perform the functions and / or methods described in the embodiments of this disclosure.

[0125] The processor 16 performs various functional applications and data processing by running at least one of a plurality of programs stored in the system memory 28, such as implementing the method embodiments provided in this disclosure.

[0126] This disclosure also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method embodiments.

[0127] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0128] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0129] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0130] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or wide area network (WAN) domain—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0131] This disclosure also provides a computer program product that, when run on a computer, causes the computer to perform the steps of the above-described method embodiments.

[0132] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatuses and methods according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0133] Unless otherwise expressly indicated by the context, the singular form of words used herein and in the appended claims includes the plural form, and vice versa. Thus, when referring to the singular, the plural form of the corresponding term is generally included. Similarly, the terms “comprising” and “including” shall be interpreted as including rather than exclusively. Likewise, the terms “including” and “or” shall be interpreted as including unless such interpretation is expressly prohibited herein. Where the term “example” is used herein, particularly when it follows a set of terms, the “example” is merely exemplary and illustrative and should not be considered exclusive or extensive.

[0134] Further aspects and scope of adaptation become apparent from the description provided herein. It should be understood that various aspects of this disclosure may be implemented individually or in combination with one or more other aspects. It should also be understood that the descriptions and specific embodiments herein are for illustrative purposes only and are not intended to limit the scope of this disclosure.

[0135] Several embodiments of this disclosure have been described in detail above. However, it is obvious that those skilled in the art can make various modifications and variations to the embodiments of this disclosure without departing from the spirit and scope of this disclosure. The scope of protection of this disclosure is defined by the appended claims.

Claims

1. A satellite beam scheduling method based on reinforcement learning, characterized in that, include: Construct a state space and an action space. The state space includes the satellite identifiers of multiple satellites in the navigation satellite constellation, the elevation angle of each satellite relative to the target area, the azimuth angle of each satellite relative to the target area, and the relative distance between each satellite and the target area. The action space includes the sequence of satellite identifiers of the satellites to be scheduled. The reward function is designed based on the reward indicators, which include the coverage multiplicity reward and the location accuracy attenuation factor (PDOP) reward. Based on the TD3 algorithm, a neural network for satellite beam scheduling is trained according to the state space, the action space, and the reward function to obtain a trained policy network, wherein the neural network includes the policy network and the value network; The real-time state vector of the target area is input into the trained policy network to obtain real-time actions, thereby controlling all satellite beams corresponding to the real-time actions to point towards the target area.

2. The method according to claim 1, characterized in that, The method of training a satellite beam scheduling neural network based on the TD3 algorithm, according to the state space, the action space, and the reward function, to obtain a trained policy network includes: The current state vector is input into the policy network to obtain the current action. The state space includes the state vector, and the action space includes the action. In the next moment, control all satellite beams corresponding to the action at the current moment to point towards the target area; Obtain the next time-state vector, the next time-cover multiplicity of the target region, and the next time-PDOP; The reward value at the current time step is determined based on the next time step coverage multiplicity, the next time step PDOP, and the reward function. The current state vector, the current action, the current reward value, and the next state vector are stored as a set of training samples in the experience replay pool. Multiple sets of training samples are randomly selected from the experience replay pool, and each set of training samples is input into the neural network. The network parameters of the neural network are updated based on the TD3 algorithm to obtain the trained policy network.

3. The method according to claim 2, characterized in that, The step of updating the network parameters of the neural network based on the TD3 algorithm to obtain the trained policy network includes: The current state vector and the current action from each group of training samples are input into the value network to obtain the estimated value. The network parameters of the value network are adjusted by minimizing the error between the estimated value and the target value; Based on the feedback from the value network, the network parameters of the policy network are adjusted, and the process of randomly selecting multiple sets of training samples from the experience replay pool is repeated until the convergence condition is met and training ends, resulting in the trained policy network.

4. The method according to claim 1, characterized in that, The state space also includes the historical and future durations of continuous visibility of each satellite over the target area.

5. The method according to claim 1 or 4, characterized in that, The state space also includes the selected state information of each satellite at the previous moment.

6. The method according to claim 5, characterized in that, The reward indicators also include rewards for the number of satellite handovers; The reward function designed based on reward metrics includes: The coverage multiple reward, the PDOP reward, and the satellite handover number reward are weighted respectively; The reward function is obtained by summing the weighted coverage multiple reward, the PDOP reward, and the satellite handover number reward.

7. The method according to claim 6, characterized in that, When the coverage weight value is greater than or equal to the first preset value, the coverage weight reward value is positive; when the coverage weight value is less than the first preset value, the coverage weight reward value is negative, and the coverage weight reward value is positively correlated with the coverage weight value. When the PDOP value is less than or equal to the second preset value, the PDOP reward value is positive; when the PDOP value is greater than the second preset value, the PDOP reward value is negative, and the PDOP reward value is negatively correlated with the PDOP value. When the number of satellite handovers is less than or equal to a third preset value, the satellite handover reward value is positive; when the number of satellite handovers is greater than the third preset value, the satellite handover reward value is negative, and the satellite handover reward value is negatively correlated with the number of satellite handovers.

8. A satellite beam scheduling device based on reinforcement learning, characterized in that, include: A module is established to construct a state space and an action space. The state space includes the satellite identifiers of multiple satellites in the navigation satellite constellation, the elevation angles of each satellite relative to the target area, the azimuth angles of each satellite relative to the target area, and the relative distances of each satellite to the target area. The action space includes a sequence of satellite identifiers of satellites to be scheduled. The design module is used to design a reward function based on reward indicators, which include coverage multiplicity reward and location accuracy decay factor (PDOP) reward. The training module is used to train a satellite beam scheduling neural network based on the TD3 algorithm, according to the state space, the action space and the reward function, to obtain a trained policy network, wherein the neural network includes the policy network and the value network; The scheduling module is used to input the real-time state vector of the target area into the trained policy network to obtain real-time actions, so as to control all satellite beams corresponding to the real-time actions to point to the target area.

9. An electronic device, characterized in that, include: A processor for executing a computer program stored in a memory, wherein the computer program, when executed by the processor, implements the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-7.

Images