Satellite round game decision-making method based on alpha-zero, terminal and medium

Abstract

The application relates to the technical field of satellite autonomous control, and discloses a satellite round game pursuit-escape game decision method based on Alpha-Zero, a terminal and a medium. The method sets physical constants and game rules, establishes a relative orbit motion model of satellites of both sides of pursuit and escape based on a C-W equation, and thus constructs a satellite round game pursuit-escape game environment; a strategy value neural network based on an Alpha-Zero framework is constructed, and when impulse orbit change decision needs to be made in each round, Monte Carlo tree search is performed based on the satellite round game pursuit-escape game environment and the strategy value neural network; the parameters of the strategy value neural network are iteratively optimized through self-game iteration, and an optimal strategy value neural network is obtained; after the real-time collected satellite orbit state is input into the optimal network after feature coding, an impulse orbit change strategy is generated and the satellite is controlled to execute. The application realizes efficient autonomous game of the satellite in a complex space environment.

Description

Technical Field

[0001] This invention relates to the field of satellite autonomous control technology, specifically to a satellite turn-based pursuit and escape game decision-making method, terminal, and medium based on Alpha-Zero. Background Technology

[0002] With the development of aerospace technology, satellite missions in orbit are becoming increasingly complex. Traditional satellite control relies on remote control from ground stations, which suffers from high communication latency and weak anti-interference capabilities, making it difficult to meet the real-time pursuit and escape requirements in complex space environments. Satellite pursuit and escape game, as a typical zero-sum game problem, is characterized by a large state space, asymmetric actions, and high timeliness of decision-making. Currently, methods for solving such problems mainly include traditional optimization algorithms (such as Monte Carlo Tree Search (MCTS)) and artificial intelligence methods based on reinforcement learning; however, both have certain limitations in practical applications.

[0003] First, traditional Monte Carlo tree search algorithms are inefficient and time-consuming when dealing with scenarios like satellite orbit changes, which involve a vast action space and multiple rounds of long-term game theory, making it difficult to meet the needs of real-time on-orbit decision-making. Because satellites have relatively limited computing power, the high latency of this algorithm can prevent satellites from generating orbit change commands within the extremely short game window, thus missing the optimal opportunity for capture or escape.

[0004] Secondly, while general reinforcement learning-based artificial intelligence algorithms possess strong self-learning capabilities, they often exhibit "black box" characteristics, resulting in weak model interpretability and a training process that typically relies heavily on prior information. In the high-speed relative motion of space, outputting abnormal commands that deviate from the laws of physics could easily trigger irreversible safety accidents such as satellite collisions or spacetime drift. Summary of the Invention

[0005] To address the technical problems existing in the prior art, this invention provides a satellite turn-based pursuit and escape game decision-making method, terminal, and medium based on Alpha-Zero. It improves the efficiency of satellite strategy search in orbit through neural networks and combines MCTS with physical models to achieve efficient autonomous game playing in complex space environments.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] This invention discloses a satellite-based turn-based pursuit and escape game decision-making method based on Alpha-Zero, including steps S1 to S4.

[0008] S1. Set physical constants and game rules, and establish a relative orbital motion model of the satellites pursuing and fleeing based on the CW equation, thereby constructing a satellite turn-based pursuit and fleeing game environment; wherein, the game rules include: setting the maximum number of game rounds and the capture threshold, in each round one satellite takes action by applying a pulse orbit change, and the other satellite drifts naturally, the two sides take turns to execute, each round lasts for a set time, and when the satellites of both sides are less than the capture threshold, the capture is considered successful, and the asymmetric action space of the satellites pursuing and fleeing is defined.

[0009] S2. Construct a strategy value neural network based on the Alpha-Zero framework, and when pulse orbit change decisions are required in each round, perform Monte Carlo tree search based on the satellite turn-based pursuit and escape game environment and the strategy value neural network; the strategy value neural network is used to receive the encoded satellite orbit state features and output the prior probability of actions and state value of the satellites of both the pursuit and escape sides.

[0010] S3. The parameters of the policy value neural network are optimized through self-play iterative optimization to obtain the optimal policy value neural network.

[0011] S4. The real-time acquired satellite orbit status is input into the optimal strategy value neural network after feature encoding. Monte Carlo tree search is performed based on the probability distribution output by the network. The action with the highest number of visits in the search results is used as the pulse orbit change strategy for the current round and the satellite is controlled to execute it.

[0012] As a further improvement to the above scheme, in step S2, each node in the Monte Carlo tree stores the following statistics: total number of visits. ,action Number of visits ,action Total value accumulation Average value And the prior probability of actions predicted by the policy value neural network , The process of performing a Monte Carlo tree search involves the following four stages:

[0013] Selection Phase: Based on the statistical information stored in the nodes, the UCB algorithm is used to select child nodes downwards until a leaf node is reached; wherein, the node with the highest UCB score is selected as the child node at each step, and the formula for calculating the UCB score is:

[0014] ;

[0015] In the formula, For UCB scores; the value of satellites used by both sides in the pursuit and escape. They are opposites; To explore and utilize the balance coefficient;

[0016] Expansion Phase: Input the orbital state features corresponding to the leaf nodes into the policy value neural network to obtain the prior probability of the action corresponding to that node. and predicted state value and initialize the node. , , and ;

[0017] Simulation Phase: Starting from the current round, the set Monte Carlo tree search depth is extrapolated downwards, the changes in relative distances are calculated, and the predicted state value is combined. Calculate the value of the smoothed state :

[0018] ;

[0019] In the formula, From the current round To the Relative distance after the round, Determine the depth for the Monte Carlo tree search; This represents the initial relative distance; As the weight of distance value, Weights for predicting state values; It is a symbolic function;

[0020] Backtracking phase: The smoothed state value is backpropagated to update the number of visits, total value, and average value of each node in the path;

[0021] After repeating the above four stages a preset number of times, the search strategy for this decision is generated based on the distribution of the number of visits to each action at the root node.

[0022] As a further improvement to the above scheme, Dirichlet noise is added at the root node during the selection phase to smooth the prior probabilities of actions, expressed as follows:

[0023] ;

[0024] In the formula, The prior probability of the corrected action after adding noise; Dirichlet noise, For noise parameters; These are the weight parameters.

[0025] As a further improvement to the above scheme, in step S1, the physical constants include the Earth's gravitational constant. Reference orbit radius and orbital angular velocity and satisfy The state transition matrix of the relative orbital motion model satisfies:

[0026] ;

[0027] In the formula, Let be the state transition matrix at time t; As an intermediate quantity, ; Set the duration for each round.

[0028] As a further improvement to the above scheme, in step S1, the asymmetric action space of the satellites of both the pursuing and fleeing parties is defined as follows:

[0029] The tracking satellite and the escaping satellite have different maximum pulse velocity increment limits and discretization levels, with the maximum pulse velocity increment of the tracking satellite being greater than that of the escaping satellite; the total strategy number for both satellites is expressed by the following formula:

[0030] ;

[0031] ;

[0032] In the formula, To track the total number of strategies for the satellite, To track the discrete series of satellite pulse velocity increments, To determine the discrete number of thrust deflection angles of the tracking satellite, the maximum pulse velocity increment of the tracking satellite is used. Strategy discretization Level pulse velocity increment; The total number of strategies for the escaping satellite; Let be the discrete series of the velocity increment of the escaping satellite pulse. The discrete number of thrust deflection angles of the escaping satellite is represented by the maximum pulse velocity increment of the escaping satellite. Strategy discretization The velocity increment of the pulse; the velocity update formula for satellite pulse orbit change is:

[0033] ;

[0034] In the formula, The speed after the trajectory change. Speed ​​before orbital change; This represents the pulse velocity increment after discretization. The thrust deflection angle ranges from 0° to 360°; the satellite's orbital state evolves via the CW equations after pulse orbital maneuvers; superscript This is the transpose symbol.

[0035] As a further improvement to the above scheme, in step S2, the encoding rule for the satellite orbital state characteristics is as follows:

[0036] The four orbital components of the current step of the tracking satellite and the escaping satellite. The four orbital components of the previous historical state, the four orbital components of the previous two historical states, and the current round number are concatenated to form a 25-dimensional original feature vector; then the positional components in the original feature vector are... and velocity components Normalized to Range, normalize the current round number to The normalized feature vector is the encoded satellite orbital state feature.

[0037] As a further improvement to the above scheme, the policy value neural network includes a shared backbone network, a tracking policy head, an escape policy head, and a value head. The shared backbone network contains three cascaded fully connected residual blocks. The input of this network is a 25-dimensional satellite orbital state feature, and the output is a 256-dimensional general feature. The inputs of the tracking policy head and the escape policy head are both 256-dimensional general features. The tracking policy head is used to output... The policy logits of the dimension, the escape policy head is used for output. The strategy for the dimension is logits; the value header is used for output. The state value of the range, where 1 represents the absolute advantage of the pursuer and -1 represents the absolute advantage of the escaper.

[0038] As a further improvement to the above scheme, step S3 specifically includes:

[0039] During the self-play process, each step selects a strategy based on the number of times an action is accessed, and the replay buffer stores the strategy based on the average value of the action.

[0040] During the training phase, samples are randomly sampled from the replay buffer, the value loss is calculated using MSE loss, the policy loss is calculated using cross-entropy loss, and the network parameters are updated using the SGD optimizer.

[0041] The average final value is used to evaluate the strategy value neural network for a set number of rounds. In each round, the satellites of both the pursuing and fugitive sides alternately use the current strategy value neural network and the optimal strategy value neural network to generate strategies. If the average value of the current strategy value neural network is higher than that of the optimal strategy value neural network by more than a set threshold, the current strategy value neural network is updated to the optimal strategy value neural network and saved.

[0042] The present invention also discloses a computer terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the Alpha-Zero-based satellite turn-based pursuit and escape game decision-making method as described above.

[0043] The present invention also discloses a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the Alpha-Zero-based satellite turn-based pursuit and escape game decision-making method as described above.

[0044] Compared with the prior art, the beneficial effects of the present invention are:

[0045] 1. This invention discloses a satellite turn-based pursuit-escape game decision-making method based on Alpha-Zero. It represents the satellite pursuit-escape game as a multi-round zero-sum game in a discrete space. Through a fusion architecture of deep neural networks and Monte Carlo tree search, it provides the value of a state and the prior probabilities of actions in the action space under that state, improving the efficiency of deep search strategies for both the pursuer and the escaper. This method generates training samples based on a self-play mechanism, eliminating the need for manually provided complex prior strategies. It solves the problems of long Monte Carlo tree search time and low efficiency in complex action environments, as well as the low interpretability of traditional reinforcement learning. Simulation tests show that the proposed decision-making method can autonomously and efficiently learn pursuit-escape strategies, enabling the satellite to complete complex game decisions within a limited control period, overcoming the response lag problem caused by excessive computation time in traditional methods.

[0046] 2. This invention constructs a relative orbital motion model based on the CW equations as the foundation of the search environment, and directly embeds physical constraints (such as the maximum pulse velocity increment limit) into the game rules. Compared with conventional end-to-end reinforcement learning "black box" models, this invention combines the explicit inference trajectory of MCTS, ensuring that each orbital change decision has a clear physical evolution path and traceability. This high interpretability and physical consistency effectively avoids the risk of collision or loss of control caused by abnormal algorithm output commands during high-speed relative motion, ensuring the safety of on-orbit operation.

[0047] 3. To address the challenges of "difficult-to-determine outcome and sparse rewards" in satellite pursuit games, this invention introduces a smoothed value calculation mechanism that combines the rate of change of relative distance with neural network predictions. This mechanism replaces the extreme reward system in traditional games that only provides a return at the end, enabling real-time evaluation of the current strategy's merits during the middle stages of the game. This allows satellites to efficiently find the optimal strategy in complex game decision-making. Attached Figure Description

[0048] Figure 1This is a flowchart of the satellite turn-based pursuit and escape game decision-making method based on Alpha-Zero in Embodiment 1 of the present invention.

[0049] Figure 2 This is a diagram illustrating the reasoning and training framework for satellite turn-based pursuit and escape game decision-making based on Alpha-Zero in Embodiment 1 of the present invention.

[0050] Figure 3 This is a schematic diagram of the satellite pursuit trajectory in scenario example one of Embodiment 1 of the present invention.

[0051] Figure 4 This is a schematic diagram showing the change of relative distance with the number of rounds in scenario example one of Embodiment 1 of the present invention.

[0052] Figure 5 This is a schematic diagram of the satellite pursuit trajectory in scenario example two of embodiment 1 of the present invention.

[0053] Figure 6 This is a schematic diagram showing the change of relative distance with the number of rounds in scenario example two of embodiment 1 of the present invention.

[0054] Figure 7 This is a schematic diagram illustrating the change in the relative distance between satellites in pursuit and escape under different strategies in Scenario Example 3 of Embodiment 1 of the present invention, as a function of the number of rounds.

[0055] Figure 8 This is a comparison chart of the calculation time for strategy example three in Embodiment 1 of the present invention.

[0056] Figure 9 This is a schematic diagram of the structure of the computer terminal in Embodiment 2 of the present invention. Detailed Implementation

[0057] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0058] Example 1

[0059] This embodiment provides a satellite turn-based pursuit and escape game decision-making method based on Alpha-Zero, which solves the problems of low efficiency and shallow search depth of traditional Monte Carlo tree search and weak interpretability of conventional reinforcement learning algorithms. Through continuous self-play training in a constructed satellite pursuit and escape simulation environment, it explores how to utilize neural network prior guidance and smoothing value mechanisms to achieve efficient and reliable pursuit and escape decision-making strategies, adapting to the autonomous game requirements in complex space environments.

[0060] Please see Figure 1 The decision-making method of the present invention includes steps S1 to S4.

[0061] S1. Set physical constants and game rules, and establish a relative orbital motion model of the satellites of the pursuing and fleeing parties based on the CW equation, thereby constructing a satellite turn-based pursuit and fleeing game environment.

[0062] The physical constants include the Earth's gravitational constant. Reference orbit radius (Geosynchronous orbit) and orbital angular velocity and satisfy .

[0063] Setting the game rules includes: setting the maximum number of game rounds G. max The Monte Carlo tree search depth is 13 rounds. The capture threshold is 7 rounds. In each round, one satellite performs an action by applying a pulse to change its orbit, while the other satellite drifts away naturally. The two sides take turns performing this action, and each round lasts for a set time. The term "second" refers to the time after each orbital maneuver, during which the spacecraft drifts naturally according to the CW equations for 600 seconds before the opponent makes a decision. A successful capture is determined when both satellites are below the capture threshold. The asymmetric action space for both pursuing and fleeing satellites is defined as follows:

[0064] The tracking satellite and the escaping satellite have different maximum pulse velocity increment limits and discretization levels, with the tracking satellite's maximum pulse velocity increment being greater than that of the escaping satellite. The total strategy number for both satellites is expressed by the following formula:

[0065] ;

[0066] ;

[0067] In the formula, To track the total number of strategies for the satellite, To track the discrete series of satellite pulse velocity increments, To determine the discrete number of thrust deflection angles of the tracking satellite, the maximum pulse velocity increment of the tracking satellite is used. Strategy discretization Level pulse velocity increment; The total number of strategies for the escaping satellite; Let be the discrete series of the velocity increment of the escaping satellite pulse. The discrete number of thrust deflection angles of the escaping satellite is represented by the maximum pulse velocity increment of the escaping satellite. Strategy discretization The increment of the pulse velocity.

[0068] In this embodiment, the maximum pulse velocity increment of the tracking satellite The maximum speed is 0.003 km / s, the maximum pulse deflection angle is 360°, the strategy is discretized into 2 levels, the thrust deflection angle is discretized into 8 levels, and the total number of strategies is... The maximum pulse velocity increment of the escaping satellite The maximum speed is 0.001 km / s, the maximum pulse deflection angle is 360°, the strategy is discretized into 2 levels, the thrust deflection angle is discretized into 6 levels, and the total number of strategies is... .

[0069] In this embodiment, it is assumed that all satellites are in geosynchronous orbits, and the distance between the satellites is much smaller than the semi-major axis of their orbits. Therefore, the CW (Clohessy Wiltshire) equation is used to approximate the relative motion between them. The x-axis is defined as the direction from the Earth's center to the spacecraft, and the y-axis as the direction of the orbital velocity. The state variables are defined as follows: , , These represent the relative positions in the x and y directions, respectively, with units of km; , These represent the velocity components in the x and y directions, respectively, with units of km / s.

[0070] The initial state of the tracking party relative to the reference satellite is defined as follows: The initial state of the escaping party relative to the reference satellite is Initial relative distance km.

[0071] The relative orbital motion model is constructed based on the CW equations to establish a relative motion model between the pursuer and the pursuer, satisfying:

[0072] ;

[0073] In the formula, , They are respectively , The second derivative, , They are respectively , The first derivative, and They are respectively , The control acceleration component applied in the direction.

[0074] The state transition matrix is ​​obtained by directly solving its analytical solution. The orbital evolution after each pulse orbital change can be efficiently calculated using the state transition matrix:

[0075] ;

[0076] In the formula, For the satellite in t The state at any given moment; This represents the satellite's state at time t after performing a pulse maneuver. satisfy:

[0077] ;

[0078] In the formula, Let be the state transition matrix at time t; As an intermediate quantity, ; The duration of each round is a set time. Impulse maneuvers are employed, assuming an instantaneous change in velocity components, specifically manifested as the state after a trajectory change. satisfy:

[0079] ;

[0080] In the formula, This represents the satellite's state at time t before performing the pulse maneuver; This represents the pulse velocity increment after discretization. The thrust deflection angle ranges from 0° to 360°; the satellite's orbital state evolves via the CW equations after pulse orbital maneuvers; superscript This is the transpose symbol.

[0081] Set a termination condition; the game ends when any of the following conditions are met:

[0082] (1) The relative distance between the pursuer and the fugitive is less than the arrest threshold. km (Successful capture);

[0083] (2) Reach the maximum number of rounds =15 (timeout).

[0084] S2. Construct a strategy value neural network based on the Alpha-Zero framework, and when pulse orbit change decisions are required in each round, perform Monte Carlo tree search based on the satellite turn-based pursuit and escape game environment and the strategy value neural network; the strategy value neural network is used to receive the encoded satellite orbit state features and output the prior probability of actions and state value of the satellites of both the pursuit and escape sides.

[0085] The coding rules for satellite orbital state characteristics are as follows:

[0086] The four orbital components of the current step of the tracking satellite and the escaping satellite. The four orbital components of the previous historical state, the four orbital components of the previous two historical states, and the current round number are concatenated to form a 25-dimensional original feature vector; then the positional components in the original feature vector are... and velocity components Normalized to Range, normalize the current round number to The normalized feature vector is the encoded satellite orbital state feature.

[0087] In this embodiment, the current state (8-dimensional), the previous step's historical state (8-dimensional), the previous two steps' historical state (8-dimensional), and the current round number (1-dimensional) of both the pursuer and the pursuer are spliced ​​together to form a 25-dimensional original feature:

[0088] ;

[0089] in, For the first two historical states, , The first two steps are for tracking the relative positions of the objects. For the velocity components of the first two tracking steps, The relative positions of the two escapers in the first two steps. These are the velocity components of the first two escape steps; This refers to the previous historical state. , To determine the relative position of the tracking party in the previous step, For the velocity component of the previous tracking step, This refers to the relative position of the party that escaped in the previous step. The velocity component of the escape vehicle in the previous step; The current step state, , For the current step, the relative position of the tracking party. For the current step tracking velocity component, The relative position of the current escape route. The current escape velocity component; This represents the current round number. Normalize each feature component to... Scope, of which and The normalized ranges are [-30, 50] km and [-25, 150] km, respectively; velocity components and The normalized ranges are [-0.048, -0.048] km / s and [-0.016, 0.016] km / s, respectively, and the normalized range of the number of rounds is [0, G]. max ].

[0090] The policy value neural network of the present invention adopts a "shared backbone network + multi-task head" architecture, including a shared backbone network, a tracking policy head, an escape policy head, and a value head.

[0091] The shared backbone network consists of three cascaded fully connected residual blocks. Each residual block has the structure "Linear (256) → BatchNorm1d → LeakyReLU (0.1) → Dropout (0.1) → Linear (256) → BatchNorm1d", with the residual connection defined as output = LeakyReLU(x + block (x)). Here, Linear represents the fully connected layer, 256 indicates the output feature dimension of that layer, BatchNorm1d is a one-dimensional batch normalization layer, LeakyReLU is a modified linear unit activation function with leakage, Dropout is a random deactivation layer, output is the output, and block(·) represents the main branch operation within a single residual block. The input to the shared backbone network is 25-dimensional satellite orbital state features, and the output is 256-dimensional general features.

[0092] Both the tracking and escape policy heads are input to 256-dimensional general features. The tracking policy head is output after passing through a softmax function. The policy logits of the escape direction are output after passing through the softmax function. The strategy logits of the dimension; the value head is output through the tanh function. The state value of the range, where 1 represents the absolute advantage of the pursuer and -1 represents the absolute advantage of the escaper.

[0093] All fully connected layer weights are initialized using a He normal distribution, BatchNorm1d layer weights are initialized to 1, and biases are initialized to 0; the negative slope of all LeakyReLU layers is uniformly set to 0.1, and the dropout probability is set to 0.1.

[0094] Each node in a Monte Carlo tree stores the following statistics: total number of visits. ,action Number of visits ,action Total value accumulation Average value And the prior probability of actions predicted by the policy value neural network , The state is represented; the process of performing a Monte Carlo tree search includes four stages: selection, expansion, simulation, and backtracking.

[0095] Selection Phase: Based on the statistical information stored in the nodes, the UCB (Upper Confidence Bound) algorithm is used to select child nodes downwards until a leaf node is reached; wherein, the node with the highest UCB score is selected as the child node at each time, and the formula for calculating the UCB score is:

[0096] ;

[0097] In the formula, For UCB scores; the value of satellites used by both sides in the pursuit and escape. The numbers are opposites; the tracking number is positive, and the escaping number is negative. To explore and utilize the balance coefficient, it can be set to 1.2.

[0098] During the selection phase, to avoid local optima, Dirichlet noise is added at the root node to smooth the prior probabilities of actions, expressed as follows:

[0099] ;

[0100] In the formula, The prior probability of the corrected action after adding noise; Dirichlet noise, For noise parameters, It can be set to 0.03; This is a weighting parameter, which can be set to 0.15, and the coefficient is... and By controlling the weights of the original prior probability and the noise respectively, an exploration perturbation is introduced while ensuring the stability of the strategy.

[0101] Expansion Phase: Input the orbital state features (25 dimensions) corresponding to the leaf node into the policy value neural network to obtain the prior probability of the action corresponding to that node. and predicted state value and initialize the node. , , and .

[0102] Simulation Phase: Starting from the current round, the set Monte Carlo tree search depth is extrapolated downwards, the changes in relative distances are calculated, and the predicted state value is combined. Calculate the value of the smoothed state .

[0103] In this embodiment, the relative distance satisfies:

[0104] ;

[0105] In the formula, Indicates relative distance; It is an L2 norm.

[0106] Smooth state value The calculation formula is:

[0107] ;

[0108] In the formula, From the current round To the Relative distance after the round; The Monte Carlo tree search depth is set to 7; This represents the initial relative distance; The weight for distance value is set to 0.8; The weight for predicting state value is set to 0.2; It is a symbolic function.

[0109] The simulation phase is based on the improved UCB value, from the current round r i Search down to the rth node i +7 rounds, save the search path and the value of the nodes in the path.

[0110] Backtracking phase: The smoothed state value is backpropagated to update the number of visits, total value, and average value of each node in the path.

[0111] After repeating the above four stages a preset number of times, the search strategy for this decision is generated based on the distribution of the number of visits to each action at the root node.

[0112] S3. The parameters of the policy value neural network are optimized through self-play iterative optimization to obtain the optimal policy value neural network.

[0113] The overall steps during training are as follows: Figure 2 As shown, specifically:

[0114] Training samples are generated through self-play. During training, each round of self-play involves an 800-step Monte Carlo tree search to select a strategy, after which the strategy is returned. After the self-play ends, the final value is returned from the hunter's perspective. The strategy used in self-play differs from the strategy placed in the replay buffer. A dual-model structure is employed during model training. Training samples are generated through self-play by the optimal model, and the sub-model is updated in real-time based on these samples. After a certain number of moves, the optimal model and the sub-model take turns playing the hunter and the fleeing player, respectively. Model performance is evaluated based on the average final value, and network parameters are iteratively optimized.

[0115] During the self-play process, the strategy selected at each step is based on sampling the number of times an action is accessed, specifically: For a strategy distribution based on the number of times an action is accessed, the probability of the action with the highest number of accesses is: The probabilities of the second and third largest actions are both 0. ,set up It is 0.6.

[0116] Sample storage: (state features) Strategy distribution Actual final value The triple is stored in the replay buffer. The replay buffer stores the average value of the actions. The sampling strategy is as follows: Strategy For a policy distribution based on the average value of actions, satisfying the average value of actions The highest probability of action is The probabilities of the second and third largest actions are both 0. , It is 0.6.

[0117] When the number of samples in the replay buffer reaches 1000, network training begins. During the training phase, samples are randomly sampled from the replay buffer with a batch size of 256, 8 training epochs, a learning rate of 0.005, a momentum of 0.9, and a weight decay of 1e-5. State features are then input into the neural network. To obtain predictive value Predicting the prior probability distribution Define total loss The sum of value loss and strategy loss:

[0118] ;

[0119] In the formula, For strategic losses, For the loss of value, Given the network parameters, the cross-entropy loss strategy is used to calculate the loss:

[0120] ;

[0121] In the formula, For policy distribution based on average action value, This represents the prior probability distribution predicted by the neural network.

[0122] During the training phase, samples are randomly sampled from the playback buffer, and the value loss is calculated using MSE loss. , For actual endgame value, Predict the value of the network; update the network parameters using the SGD (Stochastic Gradient Descent) optimizer.

[0123] Every 50 rounds, the strategy value neural network evaluation uses 20 rounds of cross-game (the new network and the current best network take turns playing the roles of pursuer and pursuer, and the average value of the final game is compared. When the average value of the new network is higher than the update threshold of 0.005, the best network is updated and saved).

[0124] The average final value is used to evaluate the strategy value neural network for a set number of rounds. In each round, the satellites of both the pursuing and fugitive sides alternately use the current strategy value neural network and the optimal strategy value neural network to generate strategies. If the average value of the current strategy value neural network is higher than that of the optimal strategy value neural network by more than a set threshold, the current strategy value neural network is updated to the optimal strategy value neural network and saved.

[0125] S4. The real-time acquired satellite orbit status is input into the optimal strategy value neural network after feature encoding. Monte Carlo tree search is performed based on the probability distribution output by the network. The action with the highest number of visits in the search results is used as the pulse orbit change strategy for the current round and the satellite is controlled to execute it.

[0126] This embodiment also verifies the effectiveness of the above-mentioned Alpha-Zero-based satellite turn-based pursuit and escape game decision-making method. Hardware environment: a general-purpose computer device with a multi-core central processing unit (CPU frequency ≥ 2.0 GHz), random access memory (RAM ≥ 32 GB), and high-performance graphics processing unit (GPU computing power ≥ 20 TFLOPS, video memory ≥ 8 GB). Software environment: built based on Python 3.11 series version interpreter; core dependency libraries include NumPy library for numerical calculation, PyTorch library for deep learning model building and training, and custom game environment module for game process simulation.

[0127] In Example 1, the traditional MCTS algorithm, using the same value function and UCB formula, searches 50 times before each decision step, with a search depth of 7. Figure 3 A schematic diagram of the satellite pursuit trajectory generated for it. Figure 4 The curve showing the change in their relative distance with the number of rounds.

[0128] In Example 2, the traditional MCTS algorithm, using the same value function and UCB formula, searches 1000 times before each decision step, with a search depth of 7. Figure 5 A schematic diagram of the satellite pursuit trajectory generated for it. Figure 6 The curve showing the change in their relative distance with the number of rounds.

[0129] Because the escapee's speed changes relatively little, the small differences in early decisions accumulate over time, significantly impacting the later game situation. Therefore, the focus is on analyzing the escapee's trajectory strategy: Figures 3 to 4In the traditional MCTS (50 searches per step, depth 7), the escapee initially moves in the positive x-axis direction, which can gain a short-term advantage, but later it gets caught up in a simple speed contest with the pursuer in the x-direction, and ultimately the disadvantage is great. Figures 5 to 6 In the traditional MCTS (1000 searches per step, depth 7), the escapee anticipated the limitations of the later-stage strategy in Example 1 through thorough searching, recognizing that the pursuer would inevitably accelerate in the positive x-axis direction to complete the capture, and thus adopted an x-axis velocity alternation strategy, gaining the initiative in the later stages of the game.

[0130] Comparing the two sets of examples above, it can be seen that when the number of searches is insufficient, the decision-making of traditional MCTS is prone to short-term bias and it is difficult to take into account long-term benefits. However, the algorithm of this invention based on the AlphaZero framework only requires 200 searches per step and a depth of 7 (far less than the 1000 searches of traditional MCTS) to achieve a similar trend but better details, which significantly improves search efficiency.

[0131] Based on the above parameters, Example 3 further conducts a direct comparison experiment between the traditional MCTS algorithm and the algorithm based on the AlphaZero framework of this invention: the traditional MCTS algorithm uses the same value function and UCB formula, searching 1000 times before each decision step with a search depth of 7; the algorithm of this invention searches only 200 times per step, maintaining a search depth of 7. The game results of the two algorithms as the pursuer and the escapee are as follows: Figure 7 As shown: the blue line represents experimental data with the algorithm of this invention as the pursuer and the traditional MCTS algorithm as the escaper, and the green line represents experimental data with the traditional MCTS algorithm as the pursuer and the algorithm of this invention as the escaper. The results show that even though the number of searches for the algorithm of this invention is only one-fifth that of the traditional MCTS, it still exhibits superior game performance and completes the capture two rounds earlier than the traditional MCTS.

[0132] Figure 8 The comparison of the average search time per step between the two algorithms in Example 3 is presented intuitively, further demonstrating the significant efficiency advantage of the algorithm based on the AlphaZero framework in this invention.

[0133] Example 2

[0134] This embodiment provides a computer terminal, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the satellite turn-based pursuit and escape game decision-making method based on Alpha-Zero as described in Embodiment 1.

[0135] like Figure 9As shown, the computer terminal provided in this embodiment includes: at least one processor 101, and a memory 102 connected to at least one processor 101. This embodiment does not limit the specific connection medium between the processor 101 and the memory 102. Figure 9 The example shown is the connection between processor 101 and memory 102 via bus 100. Bus 100 is... Figure 9 The connections between other components are shown in bold lines and are for illustrative purposes only, not as limiting information. Bus 100 can be divided into address bus, data bus, control bus, etc., for ease of representation. Figure 9 The bus is represented by a single thick line, but this does not indicate that there is only one bus or one type of bus. Alternatively, the processor 101 may also be called a controller; there is no restriction on the name.

[0136] In this embodiment, the memory 102 stores instructions that can be executed by at least one processor 101. The at least one processor 101 can execute the aforementioned method by executing the instructions stored in the memory 102.

[0137] The processor 101 is the control center of the device. It can connect to various parts of the control device through various interfaces and lines. By running or executing instructions stored in memory 102 and calling data stored in memory 102, the processor can perform various functions and process data, thereby monitoring the device as a whole.

[0138] In one possible design, processor 101 may include one or more processing units. Processor 101 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into processor 101. In some embodiments, processor 101 and memory 102 may be implemented on the same chip; in some embodiments, they may also be implemented on separate chips.

[0139] Processor 101 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the Alpha-Zero-based satellite turn-based pursuit and escape game decision-making method disclosed in Embodiment 1 can be directly manifested as execution by the hardware processor, or executed by a combination of hardware and software modules in processor 101.

[0140] Memory 102, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 102 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 102 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In this embodiment, memory 102 can also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0141] By designing and programming the processor 101, the code corresponding to the Alpha-Zero-based satellite turn-based pursuit and escape game decision-making method described in the aforementioned embodiments can be embedded into the chip, thereby enabling the chip to execute the code during runtime. Figure 1 The steps of the Alpha-Zero-based satellite turn-based pursuit and escape game decision-making method are shown. How to design and program the processor 101 is a technique well-known to those skilled in the art and will not be described further here.

[0142] Example 3

[0143] This embodiment provides a computer-readable storage medium storing a computer program thereon. When the program is executed by a processor, it implements the steps of the satellite turn-based pursuit and escape game decision-making method based on Alpha-Zero as described in Embodiment 1.

[0144] The computer-readable storage medium may include flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the storage medium may be an internal storage unit of a computer device, such as the hard disk or memory of the computer device. In other embodiments, the storage medium may also be an external storage device of the computer device, such as a plug-in hard disk, smart memory card, secure digital card, flash memory card, etc., provided on the computer device. Of course, the storage medium may include both internal storage units and external storage devices of the computer device. In this embodiment, the memory is typically used to store the operating system and various application software installed on the computer device. In addition, the memory can also be used to temporarily store various types of data that have been output or will be output.

[0145] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A satellite-based turn-based pursuit and escape game decision-making method based on Alpha-Zero, characterized in that, include: S1. Set physical constants and game rules, and establish a relative orbital motion model of the satellites pursuing and fleeing based on the CW equation, thereby constructing a satellite turn-based pursuit and fleeing game environment; wherein, the game rules include: setting the maximum number of game rounds and the capture threshold, in each round one satellite takes action by applying a pulse orbit change, the other satellite drifts naturally, the two sides take turns to execute, each round lasts for a set time, when both satellites are less than the capture threshold, the capture is considered successful, and the asymmetric action space of the satellites pursuing and fleeing is defined; S2. Construct a strategy value neural network based on the Alpha-Zero framework, and when pulse orbit change decisions are required in each round, perform Monte Carlo tree search based on the satellite turn-based pursuit and escape game environment and the strategy value neural network; the strategy value neural network is used to receive the encoded satellite orbit state features and output the prior probability of actions and state value of the pursuing and escaping satellites. S3. The parameters of the strategy value neural network are optimized through self-play iterative optimization to obtain the optimal strategy value neural network; S4. The real-time acquired satellite orbit status is input into the optimal strategy value neural network after feature encoding. Monte Carlo tree search is performed based on the probability distribution output by the network. The action with the highest number of visits in the search results is used as the pulse orbit change strategy for the current round and the satellite is controlled to execute it.

2. The satellite-based turn-based pursuit and escape game decision-making method based on Alpha-Zero as described in claim 1, characterized in that, In step S2, each node in the Monte Carlo tree stores the following statistics: total number of visits. ,action Number of visits ,action Total value accumulation Average value And the prior probability of actions predicted by the policy value neural network , The process of performing a Monte Carlo tree search involves the following four stages: Selection Phase: Based on the statistical information stored in the nodes, the UCB algorithm is used to select child nodes downwards until a leaf node is reached; wherein, the node with the highest UCB score is selected as the child node at each step, and the formula for calculating the UCB score is: In the formula, For UCB scores; the value of satellites used by both sides in the pursuit and escape. They are opposites; To explore and utilize the balance coefficient; Expansion Phase: Input the orbital state features corresponding to the leaf nodes into the policy value neural network to obtain the prior probability of the action corresponding to that node. and predicted state value and initialize the node. , , and ; Simulation Phase: Starting from the current round, the set Monte Carlo tree search depth is extrapolated downwards, the changes in relative distances are calculated, and the predicted state value is combined. Calculate the value of the smoothed state : In the formula, From the current round To the Relative distance after the round, Determine the depth for the Monte Carlo tree search; This represents the initial relative distance; As the weight of distance value, Weights for predicting state values; It is a symbolic function; Backtracking phase: The smoothed state value is backpropagated to update the number of visits, total value, and average value of each node in the path; After repeating the above four stages a preset number of times, the search strategy for this decision is generated based on the distribution of the number of visits to each action at the root node.

3. The satellite-based turn-based pursuit and escape game decision-making method based on Alpha-Zero as described in claim 2, characterized in that, During the selection phase, Dirichlet noise is added at the root node to smooth the prior probabilities of actions, expressed as follows: In the formula, The prior probability of the corrected action after adding noise; Dirichlet noise, For noise parameters; These are the weight parameters.

4. The satellite-based turn-based pursuit and escape game decision-making method based on Alpha-Zero according to claim 1, characterized in that, In step S1, the physical constants include the Earth's gravitational constant. Reference orbit radius and orbital angular velocity and satisfy ; The state transition matrix of the relative orbital motion model satisfies: In the formula, Let be the state transition matrix at time t; As an intermediate quantity, ; Set the duration for each round.

5. The satellite-based turn-based pursuit and escape game decision-making method based on Alpha-Zero according to claim 1, characterized in that, In step S1, the asymmetric action space of the satellites of both the pursuing and fugitive sides is defined as follows: The tracking satellite and the escaping satellite have different maximum pulse velocity increment limits and discretization levels, with the maximum pulse velocity increment of the tracking satellite being greater than that of the escaping satellite; the total strategy number for both satellites is expressed by the following formula: In the formula, To track the total number of strategies for the satellite, To track the discrete series of satellite pulse velocity increments, To determine the discrete number of thrust deflection angles of the tracking satellite, the maximum pulse velocity increment of the tracking satellite is used. Strategy discretization Level pulse velocity increment; The total number of strategies for the escaping satellite; Let be the discrete series of the velocity increment of the escaping satellite pulse. The discrete number of thrust deflection angles of the escaping satellite is represented by the maximum pulse velocity increment of the escaping satellite. Strategy discretization The velocity increment of the pulse; the velocity update formula for satellite pulse orbit change is: In the formula, The speed after the trajectory change. Speed ​​before orbital change; This represents the pulse velocity increment after discretization. The thrust deflection angle ranges from 0° to 360°; the satellite's orbital state evolves via the CW equations after pulse orbital maneuvers; superscript This is the transpose symbol.

6. The satellite-based turn-based pursuit and escape game decision-making method based on Alpha-Zero according to claim 5, characterized in that, In step S2, the encoding rule for the satellite orbital state characteristics is as follows: The four orbital components of the current step of the tracking satellite and the escaping satellite. The four orbital components of the previous historical state, the four orbital components of the previous two historical states, and the current round number are concatenated to form a 25-dimensional original feature vector; then the positional components in the original feature vector are... and velocity components Normalized to Range, normalize the current round number to The normalized feature vector is the encoded satellite orbital state feature.

7. The satellite-based turn-based pursuit and escape game decision-making method based on Alpha-Zero according to claim 6, characterized in that, The policy value neural network comprises a shared backbone network, a tracking policy head, an escape policy head, and a value head. The shared backbone network consists of three cascaded fully connected residual blocks. Its input is a 25-dimensional satellite orbital state feature, and its output is a 256-dimensional general feature. Both the tracking and escape policy heads input the 256-dimensional general feature, and the tracking policy head is used to output... The policy logits of the dimension, the escape policy head is used for output. The strategy for the dimension is logits; the value header is used for output. The state value of the range, where 1 represents the absolute advantage of the pursuer and -1 represents the absolute advantage of the escaper.

8. The satellite-based turn-based pursuit and escape game decision-making method based on Alpha-Zero according to claim 2, characterized in that, Step S3 specifically includes: During the self-play process, each step selects a strategy based on the number of times an action is accessed, and the replay buffer stores the strategy based on the average value of the action. During the training phase, samples are randomly sampled from the replay buffer, the value loss is calculated using MSE loss, the policy loss is calculated using cross-entropy loss, and the network parameters are updated using the SGD optimizer. The average final value is used to evaluate the strategy value neural network for a set number of rounds. In each round, the satellites of both the pursuing and fugitive sides alternately use the current strategy value neural network and the optimal strategy value neural network to generate strategies. If the average value of the current strategy value neural network is higher than that of the optimal strategy value neural network by more than a set threshold, the current strategy value neural network is updated to the optimal strategy value neural network and saved.

9. A computer terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the satellite turn-based pursuit and escape game decision-making method based on Alpha-Zero as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the satellite turn-based pursuit and escape game decision-making method based on any one of claims 1 to 8.

Images