A low earth orbit space target pre-avoidance system based on a QMix algorithm

Abstract

The application belongs to the field of aerospace, and relates to a low-orbit space target pre-avoidance system based on a QMix algorithm, comprising: a space target generation module for generating orbit parameters of a protected space target and fragments with a normal distribution based on a position of the protected space target; an orbit environment deduction module for evolving orbits of the earth, the protected space target and the fragments, and generating a warning time window; a collision calculation module for calculating a collision probability between the protected space target and the fragments; an orbit corridor definition module for determining a movable range of the protected space target in a single maneuver; a collision pre-avoidance algorithm module based on QMix for modeling multiple maneuver actions as intelligent agents working cooperatively, adopting a centralized training and decentralized execution architecture, optimizing a collision avoidance strategy through a collision optimization reward function defined in stages, and outputting maneuver instructions; and a system visualization module for displaying orbit states, collision risks and reward information in real time.

Description

Technical Field

[0001] This invention belongs to the field of aerospace, specifically relating to a low-Earth orbit space target pre-avoidance system based on the QMix algorithm. Background Technology

[0002] With the continuous development of spacecraft technology and the increasing number of space launches, the amount of space debris is rapidly increasing as humanity explores and utilizes space. Currently, the number of space debris is in the millions, posing a significant threat to the normal operation of spacecraft. Since space debris can travel at speeds of thousands of meters per second relative to spacecraft, a collision between a space target and debris can cause devastating damage, necessitating urgent research into the safety of space targets.

[0003] Currently, the mainstream approach to ensuring the safety of space targets is through orbit prediction, which involves anticipating the positions of space targets and space debris in advance, and then quantifying collision risks using collision probabilities. Traditional methods for calculating collision probabilities include two-dimensional area integration, central probability density area integration, single-integral integration, series integration, rectangular domain approximation, line integration, and Simpson's formula integration. Reinforcement learning methods include the Chen-Bai method. However, traditional collision avoidance methods face three problems when dealing with large-scale constellations: First, their suddenness leads to passive warnings; high-threat collisions are often only confirmed shortly before impact, forcing control centers to take emergency measures in a very short time, resulting in low fault tolerance. Second, fuel consumption is high; each rapid avoidance maneuver consumes a large amount of propellant. For constellations with tens of thousands of space targets, the cumulative fuel cost from frequent maneuvers is enormous. Third, there is a lack of system coordination; traditional avoidance methods are usually slow or only applicable to one-to-one or one-to-many situations, making it difficult to achieve many-to-many system-level coordinated avoidance. Furthermore, single-target maneuvers can trigger chain reactions, creating new collision risks for other space targets or debris, lacking global collaborative optimization capabilities. Therefore, there is an urgent need to propose a method and system for pre-avoiding low-Earth orbit targets. This system should use algorithms to anticipate potential risks, enabling proactive avoidance before a collision warning occurs, reducing fuel consumption, and preventing emergency situations requiring passive maneuvering. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of existing collision avoidance technologies, such as strong unpredictability leading to passive early warning and high fuel costs, and to propose a low-orbit space target pre-avoidance system based on the QMix algorithm.

[0005] In view of this, the present invention provides a low-Earth orbit target pre-avoidance system based on the QMix algorithm, comprising: The space target generation module is used to generate orbital parameters for protecting space targets and debris that is normally distributed based on the locations of the protected space targets. The orbital environment simulation module is used to analyze the evolution of the orbits of Earth, protected space targets, and debris, and to generate early warning time windows. The collision calculation module is used to calculate the collision probability between protected space targets and debris; The track corridor definition module is used to determine the movable range of protected space targets during a single maneuver; The collision pre-avoidance algorithm module based on QMix is ​​used to model multiple maneuvers as a cooperative agent. It employs a centralized training and distributed execution architecture, optimizes the collision avoidance strategy through a hierarchically defined collision optimization reward function, and outputs maneuver commands. The system visualization module is used to display track status, collision risk, and reward information in real time.

[0006] As an improvement to the above system, the orbital parameters generated by the space target generation module include: simulation start time, simulation end time, collision time period, orbital parameter identifier, Kepler element of the protected space target, and debris data that is normally distributed based on the position of the protected space target, wherein the debris data includes position and velocity.

[0007] As an improvement to the above system, the orbital environment simulation module adopts the MJD2000 time system to perform dual simulations with and without maneuvering, and generates an early warning time window by combining the simulation results, and issues a warning before the nth orbital cycle.

[0008] As an improvement to the aforementioned system, the collision calculation module employs the Chen-Bai method to calculate the distance and relative motion geometric parameters between the protected space target and debris at the nearest approach point. Combining this with positional errors, it uses a two-dimensional normal distribution model to determine the collision probability between the protected space target and debris. for:

[0009] in, The mean of the nearest points in the x and y directions. This represents the error variance in the corresponding direction. To protect the sum of the radii of space targets and debris.

[0010] As an improvement to the above system, the single maneuver is the minimum of two constraints, wherein the first constraint is that the maneuver does not exceed 50% of the spacecraft's remaining fuel weight, and the second constraint is the percentage of the distance between the protected space target and debris relative to the remaining fuel's navigable range. The minimum value of the two constraints is used as the orbital corridor threshold to determine the movable range of the protected space target in a single maneuver.

[0011] As an improvement to the aforementioned system, the QMix-based collision pre-avoidance algorithm module includes, during the training phase: a reward function setting submodule, a continuous action discretization submodule, a loss function calculation submodule, an agent network, a hybrid network, and a threat pre-avoidance submodule. The reward function setting submodule is used to consider at least one of the collision threshold, fuel consumption threshold and orbit consumption threshold, and obtain the overall reward based on the reward weight. Linear reward is used within the threshold and incremental penalty is used outside the threshold. The continuous motion discretization submodule is used to discretize the velocity increment dimension into M levels and the time dimension into N levels, and to perform maneuvers in the x, y, and z directions for the protected space target, resulting in... A combination of discrete actions is selected by an agent to find the optimal solution, and then the discrete actions are made continuous to obtain the optimal method. The loss function calculation module is used to calculate the loss function; An agent network, consisting of multiple agents, is used to calculate the Q-value for each independent maneuver and select the optimal discrete action. Hybrid networks are used to synthesize the Q-values ​​calculated by individual agents into a global Q-value, and ensure that the synthesis satisfies the monotonicity constraint. The threat pre-avoidance submodule is used to output maneuver commands; Each of the aforementioned agents includes an agent_net network module and a target_agent_net network module with identical structures; wherein... The agent_net network module is used to calculate the current Q value and select actions during the online learning phase, and to update the loss function based on backpropagation. The target_agent_net network module is used to calculate the target Q value during the offline policy learning process based on experience playback and provide it to the agent_net network module, which remains unchanged during the update interval.

[0012] As an improvement to the aforementioned system, the QMix-based collision pre-avoidance algorithm module includes, in the application phase: an input layer and an agent network; wherein, The input layer is used to preprocess the environmental information generated by the orbital environment simulation module and to receive the encoded identifier of the current intelligent agent. The agent network includes only the agent_net network module, which is used to output maneuver commands.

[0013] As an improvement to the above system, the agent_net network module is a three-layer fully connected network, including: an input layer, a hidden layer, a ReLU layer, a hidden layer, a ReLU layer, and an output layer.

[0014] As an improvement to the above system, the offline policy learning process based on experience replay includes: sampling historical interaction data including state, action, reward and next state from the experience replay buffer; updating the parameters of the agent_net network module through time difference learning; and using the target_agent_net network module to provide a stable target Q value, and learning the mapping relationship between state and action value. During the online learning phase, the following methods were adopted: The strategy involves the agent_net network module selecting actions based on the current state, interacting with the environment to collect experience, and adapting to changes in the fragment trajectory in real time.

[0015] As an improvement to the above system, the loss function for:

[0016] in, For expectation operator, For states and joint actions, For rewards and the next state, For the next action, The global Q value, For the current network parameters, For the target network parameters, Here, D represents the experience replay pool, which is the discount factor. Represents a joint action for all agents. Find the maximum value.

[0017] Compared with the prior art, the advantages of the present invention are: 1. This invention proposes a spacecraft orbital pre-avoidance system based on the QMix algorithm, which extends the collision warning time from hours in traditional methods to several orbital periods through multi-step prediction. This invention applies a small thrust to the spacecraft to achieve collision avoidance over several days of orbital evolution, while optimizing fuel use.

[0018] 2. The QMIX algorithm employs a framework of centralized training and distributed execution to effectively address the dimensionality explosion problem in the joint action space. In this invention, each maneuver is treated as an independent agent, and its individual Q-values ​​are synthesized into a global Q-value through a hybrid network. The method introduces offline reinforcement learning, utilizing historical orbital maneuver data for pre-training to improve learning efficiency and avoid instability in the initial exploration phase; during the online learning phase, an ε-greedy strategy is adopted to achieve a balance between utilization and exploration, and to adapt in real time to dynamic environmental changes such as debris trajectories and fuel consumption.

[0019] 3. A graded collision reward function can provide better reward strategies for different levels of emergency avoidance situations, such as prioritizing fuel consumption or collision avoidance penalties, or quickly avoiding high-speed debris. This allows spacecraft to make more flexible judgments and more appropriate choices. Attached Figure Description

[0020] Figure 1 This is a flowchart of the low-orbit space target pre-avoidance system based on the QMix algorithm of this invention; Figure 2 This is a flowchart of the core modules; Figure 3 It is the agent_net network module structure; Figure 4 These are the experimental results from Qmix. Detailed Implementation

[0021] The low-Earth orbit space target pre-avoidance system proposed in this invention includes a space target generation module, an orbital environment simulation module, a collision calculation module, an orbital corridor definition module, a collision pre-avoidance algorithm module based on Qmix, and a system visualization module.

[0022] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0023] Example 1 Embodiment 1 of the present invention provides a low-Earth orbit space target pre-avoidance system based on the QMix algorithm, comprising: The space target generation module is used to generate orbital parameters for protecting space targets and debris that is normally distributed based on the locations of the protected space targets. The orbital environment simulation module is used to analyze the evolution of the orbits of Earth, protected space targets, and debris, and to generate early warning time windows. The collision calculation module is used to calculate the collision probability between protected space targets and debris; The track corridor definition module is used to determine the movable range of protected space targets during a single maneuver; The collision pre-avoidance algorithm module based on QMix is ​​used to model multiple maneuvers as a cooperative agent. It employs a centralized training and distributed execution architecture, optimizes the collision avoidance strategy through a hierarchically defined collision optimization reward function, and outputs maneuver commands. The system visualization module is used to display track status, collision risk, and reward information in real time.

[0024] like Figure 1 , Figure 2 As shown. Specifically: 1. Space target generation module: The simulation primarily includes parameters such as simulation start time, simulation end time, collision start time, collision end time, and the number of debris generated. The generation of the protected space target is based on real space target data, and the debris is generated according to a normal distribution centered on the protected space target's position, with the probability density function shown below. Finally, the data is output to a data file, including the simulation start time, simulation end time, collision time period, orbital parameter identifier (OSC), Kepler elements of the protected space target, and debris data (position and velocity) generated based on the normal distribution of the protected space target's position. (1.1) For the values ​​that a random variable can take, The mean, Standard deviation, Let Variance be the variance.

[0025] 2. Track Environment Simulation Module: Orbit simulation was performed using the MJD2000 time system with a default step size of 1e-6 days. The evolution of the orbits of Earth, space targets, and debris was simulated simultaneously, with and without maneuvers. The simulation results were combined to generate an accurate early warning time window and issue a warning n orbital cycles in advance.

[0026] Given the six roots of the Kepler orbit: semi-major axis eccentricity ,inclination Right ascension of ascending node Perigeal argument True near point angle and the gravitational constant of the central celestial body The position of a space target in the geocentric inertial frame (ECI) and speed Calculated directly using the following formula: (1.2) (1.3) 3. Collision Calculation Module: This module primarily calculates the collision probability between protected space targets and debris space targets using the Chen-Bai method. It calculates the distance and relative motion geometric parameters between the two objects at their closest approach point, and combines this with positional errors (standard deviations of normal, tangential, and trans-orbital directions). The detailed calculation formula for the collision probability is as follows: (1.4) Let be the collision probability. This is the average of the closest points in two perpendicular directions. This represents the error variance in the corresponding direction. The sum of the radii of the two objects. The first half is the two-dimensional normal probability density function. Essentially, it's the probability density of positional overlap. It quantifies the likelihood that the centers of mass of two objects, considered as point masses, will align at their nearest point of contact, under the influence of orbital prediction and measurement errors. Its value is directly determined by the mean (predicted proximity distance) and covariance (error magnitude) of their relative positions, reflecting the risk to trajectory accuracy. The latter part includes... The term is the size adjustment factor, which converts the probability density of the above-mentioned point masses coinciding into the actual probability of a physical collision between objects of a certain size by multiplying it by an "effective collision cross-sectional area" that is proportional to the square of the sum of the radii of the two objects, thus reflecting the size risk of the target.

[0027] 4. Track Corridor Definition Module: The orbital corridor is defined as the range within which a protected space target can move within a debris cluster. This module specifies the range within which a space target can move in a single maneuver. A single maneuver is the minimum of two constraints: constraint one is that the maneuver cannot exceed 50% of the spacecraft's remaining fuel, and constraint two is the allowable distance from debris as a percentage of the fuel's travel distance. The minimum of these two constraints is used as an indicator of the reward function.

[0028] (1.5) For the track corridor threshold, The weight of the spacecraft's remaining fuel. The distance that can be flown with the remaining fuel. To protect space targets from debris at the closest possible distance.

[0029] 5. Collision pre-avoidance algorithm module based on Qmix: The collision pre-avoidance module based on Qmix primarily models the multiple maneuvers of a space target as several collaborative agents. Through a centralized training and distributed execution architecture, it optimizes the collision avoidance strategy, minimizing fuel consumption and trajectory deviation while ensuring the safety of the space target. The collision pre-avoidance algorithm module based on Qmix includes the following sub-modules during the training phase: reward function setting, continuous action discretization, loss function calculation, agent network, hybrid network, and threat pre-avoidance.

[0030] (1) Reward function setting submodule: The overall reward adopts the threshold calculation method, which consists of the collision threshold, fuel consumption threshold and track consumption threshold. Linear reward is used within the threshold, and incremental penalty is used outside the threshold. In the detailed design, this invention sets multiple granularities of calculation, such as ① considering only fuel consumption, ② considering fuel consumption + safety reward + whether it exceeds the track corridor.

[0031] (1.6) , , As a reward weight, As a safety reward, For fuel consumption, To determine whether the orbital corridor has been exceeded. The above formula is the reward function formula for granularity ②, while the reward function formula for ① only has the second term in the above formula.

[0032] (1.7) Where p is the total collision probability, ranging from [0,1]; When the collision probability threshold is hour: (No penalty); when Time: Penalty value That is, the relative proportion that exceeds the threshold.

[0033] (1.8) Where f is the cumulative fuel consumption and fuel_thr is the fuel consumption threshold.

[0034] (1.9) Where d is the fuel consumption, orb_thr is the orbital corridor threshold, which is... .

[0035] In addition, other threshold constraints were considered, such as maximum speed increment, pitch angle range, yaw angle range, maximum collision probability, safe distance, maximum change in semi-major axis, maximum change in tilt angle, and maximum eccentricity.

[0036] (2) Continuous Action Discretization Submodule: Discretizes the velocity increment dimension to level M and the time dimension to level N. The spacecraft can maneuver in the x, y, and z directions, generating all possible combinations of discrete actions. The discrete actions are then processed by an agent to find an optimal solution, and finally the discrete actions are made continuous to obtain the optimal method.

[0037] (1.10) For the action vector that has been executed, The velocity increments in the x-axis, y-axis, and z-axis directions, respectively. The optimal discrete level index is selected for the agent based on the velocity increments in the three directions. For the duration of the maneuver, The optimal discrete level index selected by the agent for the maneuver time.

[0038] (3) Loss function submodule: Loss function using TD error (1.11) For loss function, For expectation operator, For states and joint actions, For rewards and the next state, The total Q value, For the current network parameters, For the target network parameters, This is the discount factor.

[0039] (4) Agent Network: This network consists of multiple agents responsible for calculating the Q-value for each independent maneuver (i.e., each agent) to select the optimal discrete action. Each agent includes an agent_net network module and a target_agent_net network module with identical structures; where, The agent_net network module is used to calculate the current Q-value and select actions during the online learning phase, and is updated based on the backpropagation of the loss function; the agent_net network consists of three fully connected layers. The `target_agent_net` network module, similar to the `agent_net` network module, is primarily used to calculate the target Q-value, which remains constant within the update interval. It only calculates the target Q-value, providing a relatively stable target for `agent_net` updates and contributing to improved training stability.

[0040] The agent_net network module is a three-layer fully connected network, consisting of: an input layer, a hidden layer, a ReLU layer, another hidden layer, a ReLU layer, and an output layer. For example... Figure 3 As shown. The input layer receives the concatenated input vector. The first half of the vector is the global state of the entire environment, and the second half is the one-hot encoded identifier of the current agent, `agent_id`. This design allows each agent to perceive both the global environment and its own identity. The first hidden layer performs a first nonlinear transformation and compression on the high-dimensional input features, extracting preliminary feature representations. `hidden_dim` defaults to 128. The second hidden layer further extracts and combines features at a deeper level, enhancing the network's representational ability to understand the more complex relationship between state and action value. The output layer outputs a vector with dimension `action_dim`. `action_dim` corresponds to the size of the discrete action space. Each value in the vector represents the expected cumulative reward for choosing the corresponding discrete action in the current input state.

[0041] The collision pre-avoidance algorithm module based on QMix includes an input layer and an agent network in the application phase. The input layer is used to preprocess the environmental information generated by the orbital environment inference module and receive the current agent's encoded identifier. The agent network only includes the agent_net network module, which is used to output maneuver commands.

[0042] (4) Hybrid networks: This is used to combine the local Q-values ​​output by each agent network into a global Q-value, and to ensure that the synthesis satisfies the "monotonicity" constraint.

[0043] (5) Threat pre-avoidance threshold setting submodule: This invention provides three different threshold values ​​for the system. The default collision probability value is 1e-4, and the selectable values ​​are 1e-6 and 1e-8, which allows the system to make judgments earlier and perform pre-avoidance.

[0044] 6. System Visualization Module: It provides a concise visual view, including the Earth, the orbit of the protected space target, debris orbits, detailed information, and reward values, offering real-time, multi-dimensional monitoring capabilities. It provides an intuitive presentation of orbital dynamics, orbital maneuvers, and system status.

[0045] It is worth noting that in the embodiments of the above system, the modules included are divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional module are only for easy differentiation and are not used to limit the scope of protection of the present invention.

[0046] Example 2 Embodiment 2 of this invention aims to systematically evaluate the performance of a low-Earth orbit (LEO) target pre-avoidance method based on the QMix algorithm, focusing on addressing the problems of strong suddenness, high fuel cost, and lack of collaborative optimization in traditional collision avoidance methods. The experiment compares the performance of the QMix algorithm with the traditional CE (Cross-Entropy) algorithm under different collision avoidance thresholds and simulation days, verifying the advantages of QMix in pre-avoidance scenarios. Against this backdrop, with the surge in the number of LEO space debris and the increased collision risk, this experiment simulates a real orbital environment to quantify the algorithm's comprehensive effectiveness in fuel consumption, collision avoidance, and orbit maintenance.

[0047] The experimental design is based on multi-parameter configuration, with each configuration generating 10 sets of randomized environments to ensure statistical reliability. Key parameters include: Collision avoidance thresholds: set at two levels, 1e-8 and 1e-4, representing high sensitivity and standard sensitivity collision warning thresholds, respectively. The threshold is used to trigger avoidance maneuvers, and the calculation formula is based on the collision probability. .

[0048] (1.12) It is the average of the nearest approach point in the normal and tangential directions (calculated using the relative motion geometry parameters of the orbit). This represents the variance of the position error in the corresponding direction (originating from orbital uncertainty). The sum of the radii of the two objects represents the risk of overlapping physical envelopes. This represents the collision probability. The formula is simplified from the integral of a two-dimensional normal distribution. First, the relative positional distribution of the two objects at their nearest intersection point is calculated; then, the overlap probability is approximated using area integration, and an object size adjustment factor is incorporated to improve accuracy. In experiments, this formula is used to calculate the collision risk in real time, triggering an avoidance action when the risk exceeds a set threshold.

[0049] Other threshold constraints are shown in Table 1 below: Table 1

[0050] Simulation duration: Set to 14 days, 7 days, and 1 day to simulate pre-avoidance scenarios at different time scales. Longer simulation durations (e.g., 14 days) demonstrate the multi-cycle prediction capability of pre-avoidance, while shorter durations (e.g., 1 day) test emergency response.

[0051] Algorithm used: Comparison of QMix (Multi-Agent Reinforcement Learning) and CE (Conventional Collision Evaluation). QMix employs a centralized training-distributed execution architecture, while CE is based on static probability computation.

[0052] The parameter configuration table is shown in Table 2 below: Table 2

[0053] The environment was generated by the space target generation module. In this experiment, the environment was configured to protect one space target and contain five debris objects. The data for the protected space target was derived from real orbital parameters, and the debris was generated in a normal distribution centered on the protected space target. The orbital environment simulation module used the MJD2000 time system (step size 1e-6 days) to perform orbital evolution, simulating both maneuvering and non-maneuvering scenarios to generate early warning time windows.

[0054] The experimental implementation of the QMix module includes the following detailed steps: Step 1: Action Discretization Definition of continuous motion space: Spacecraft maneuvering motion is a 4-dimensional continuous vector: , representing the velocity increments and maneuver moments in the three directions, respectively.

[0055] Discretization strategy: Velocity increment dimension: The velocity increment range (e.g., [-0.1, 0.1] km / s) in each direction (X, Y, Z) is uniformly discretized into 10 levels. Therefore, there are a total of 10*10*10=1000 combinations of velocity increments in the three directions.

[0056] Time dimension: Discretize the time (or time window) of the executable maneuver into 3 levels.

[0057] Generate a set of discrete actions: Combine the velocity increments with the time levels using a Cartesian product, resulting in 1000 * 3 = 3000 discrete actions. Each discrete action corresponds to a unique integer index.

[0058] Mapping table creation: A mapping table of size 3000*4 is pre-computed and stored in memory. Each row of the table corresponds to a continuous action vector represented by a discrete action index. .

[0059] Step 2: Network Training Data preparation: The agent interacts with the environment to generate experience tuples. And store it in the experience playback buffer D.

[0060] Sampling batch: Randomly sample a small batch of data from buffer D.

[0061] Calculate the current Q value: a. For each state s and each agent i in the batch, concatenate the one-hot encoding of state s and agent i, and input it into the corresponding agent_net_i.

[0062] b. Based on the actual combined actions selected in the batch. Extract the Q-value of the corresponding action index from the output of each agent_net_i to obtain a local Q-value vector. .

[0063] c. Input the Q_i of all agents and the global state s into the hybrid network QMixer to calculate the global Q value. .

[0064] Calculate the target Q value: a. For the next state Let each target_agent_net_i independently output the Q-value of all actions, and select the maximum value. .

[0065] b. These maximum local Q values ​​and states The input is fed into the target hybrid network target_QMixer to obtain the target global Q value. .

[0066] c. Calculate the TD target: .

[0067] Calculate loss and update: a. Calculate the mean squared error loss: .

[0068] b. Perform backpropagation to calculate the gradient of the loss function with respect to the agent network parameters and the hybrid network parameters.

[0069] c. Update the parameters of agent_net and QMixer using the optimizer.

[0070] d. Call the soft_update function to slowly bring all parameters of target_agent_net and target_QMixer closer to the current network parameters.

[0071] Step 3: Offline and Online Learning In the offline policy learning process based on experience playback: Data source: The dataset is constructed using trajectory data, maneuver sequences and corresponding results (collision or not, fuel consumption) recorded in historical missions.

[0072] Training method: The above dataset is used as a fixed experience replay buffer for supervised pre-training of agent_net and QMixer. The goal is to enable the network to initially learn the mapping relationship between state and action value, avoiding the extremely poor performance and training instability caused by random exploration in the early stages of online learning.

[0073] Online phase: using Strategy Balance exploration and utilization, and adapt to changes in debris trajectory in real time.

[0074] The experimental results are shown in Table 3 below, which presents the reward value and average value for 10 experiments under each configuration (negative values ​​represent penalties, and the smaller the value, the better the performance). Table 3

[0075] The QMix pre-avoidance method has an average value of -0.362 over a simulation period of 14 days, which is significantly better than CE's -2.234 over a shorter period of days. This indicates that pre-avoidance can optimize actions several days in advance and avoid the high penalties associated with emergency maneuvers.

[0076] When the collision avoidance threshold is 1e-8 (high sensitivity), the QMix reward is more stable, indicating that early warning can reduce the collision risk; when the threshold is 1e-4, QMix has an average value of -13.245 in 7 days of simulation, which is still better than CE's -17.187, highlighting QMix's collaborative optimization capability.

[0077] Figure 4 The top left graph depicts the trend of training rewards over training episodes, with the blue line representing the single reward value per episode and the red line representing the 20-episode moving average reward. The reward gradually increases and stabilizes after initial sharp fluctuations. The top right graph shows the change in collision rate over training episodes, with the collision rate rapidly decreasing from a high level and eventually approaching zero, indicating that the agent has effectively learned collision avoidance strategies. The bottom left graph presents the change in fuel consumption over training episodes, with fuel consumption showing an overall decreasing trend and smaller fluctuations in the later stages, indicating that the agent is optimizing fuel usage efficiency. The bottom right graph uses a histogram to show the distribution of reward values, with rewards mainly concentrated in the mid-to-high value range, indicating that the agent performs well in most episodes.

[0078] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A low-Earth orbit target pre-avoidance system based on the QMix algorithm, characterized in that, include: The space target generation module is used to generate orbital parameters for protecting space targets and debris that is normally distributed based on the locations of the protected space targets. The orbital environment simulation module is used to analyze the evolution of the orbits of Earth, protected space targets, and debris, and to generate early warning time windows. The collision calculation module is used to calculate the collision probability between protected space targets and debris; The track corridor definition module is used to determine the movable range of protected space targets during a single maneuver; The collision pre-avoidance algorithm module based on QMix is ​​used to model multiple maneuvers as cooperative agents. It adopts a centralized training and distributed execution architecture, optimizes the collision avoidance strategy through a hierarchically defined collision optimization reward function, and outputs maneuver commands. and The system visualization module is used to display track status, collision risk, and reward information in real time.

2. The low-Earth orbit space target pre-avoidance system based on the QMix algorithm according to claim 1, characterized in that, The orbital parameters generated by the space target generation module include: simulation start time, simulation end time, collision time period, orbital parameter identifier, Kepler element of the protected space target, and fragment data that is normally distributed based on the position of the protected space target, wherein the fragment data includes position and velocity.

3. The low-Earth orbit space target pre-avoidance system based on the QMix algorithm according to claim 1, characterized in that, The orbital environment simulation module uses the MJD2000 time system to perform dual simulations with and without maneuvers, and generates an early warning time window based on the simulation results, while issuing a warning n orbital cycles in advance.

4. The low-Earth orbit space target pre-avoidance system based on the QMix algorithm according to claim 1, characterized in that, The collision calculation module employs the Chen-Bai method to calculate the distance and relative motion geometric parameters between the protected space target and debris at their closest approach point. Combined with positional errors, it uses a two-dimensional normal distribution model to determine the collision probability between the protected space target and debris. for: in, The mean of the nearest points in the x and y directions. This represents the error variance in the corresponding direction. To protect the sum of the radii of space targets and debris.

5. The low-Earth orbit space target pre-avoidance system based on the QMix algorithm according to claim 1, characterized in that, The single maneuver is the minimum of two constraints, where the first constraint is that the maneuver does not exceed 50% of the spacecraft's remaining fuel weight, and the second constraint is the percentage of the distance between the protected space target and debris relative to the remaining fuel's navigable range. The minimum value of the two constraints is used as the orbital corridor threshold to determine the movable range of the protected space target in a single maneuver.

6. The low-Earth orbit space target pre-avoidance system based on the QMix algorithm according to claim 1, characterized in that, The collision pre-avoidance algorithm module based on QMix includes, during the training phase: a reward function setting submodule, a continuous action discretization submodule, a loss function calculation submodule, an agent network, a hybrid network, and a threat pre-avoidance submodule. The reward function setting submodule is used to consider at least one of the collision threshold, fuel consumption threshold and orbit consumption threshold, and obtain the overall reward based on the reward weight. Linear reward is used within the threshold and incremental penalty is used outside the threshold. The continuous motion discretization submodule is used to discretize the velocity increment dimension into M levels and the time dimension into N levels, and to perform maneuvers in the x, y, and z directions for the protected space target, resulting in... A combination of discrete actions is selected by an agent to find the optimal solution, and then the discrete actions are made continuous to obtain the optimal method. The loss function calculation module is used to calculate the loss function; An agent network, consisting of multiple agents, is used to calculate the Q-value for each independent maneuver and select the optimal discrete action. Hybrid networks are used to synthesize the Q-values ​​calculated by individual agents into a global Q-value, and ensure that the synthesis satisfies the monotonicity constraint. The threat pre-avoidance submodule is used to output maneuver commands; Each of the aforementioned agents includes an agent_net network module and a target_agent_net network module with identical structures; wherein... The agent_net network module is used to calculate the current Q value and select actions during the online learning phase, and to update the loss function based on backpropagation. The target_agent_net network module is used to calculate the target Q value during the offline policy learning process based on experience playback and provide it to the agent_net network module, which remains unchanged during the update interval.

7. The low-Earth orbit space target pre-avoidance system based on the QMix algorithm according to claim 6, characterized in that, The collision pre-avoidance algorithm module based on QMix includes, in the application phase: an input layer and an agent network; wherein, The input layer is used to preprocess the environmental information generated by the orbital environment simulation module and to receive the encoded identifier of the current intelligent agent. The intelligent agent network includes only the agent_net network module, which is used to output maneuver commands.

8. The low-Earth orbit space target pre-avoidance system based on the QMix algorithm according to claim 6 or 7, characterized in that, The agent_net network module is a three-layer fully connected network, including: input layer, hidden layer, ReLU layer, hidden layer, ReLU layer and output layer.

9. The low-Earth orbit space target pre-avoidance system based on the QMix algorithm according to claim 6, characterized in that, The offline policy learning process based on experience replay includes: sampling historical interaction data including state, action, reward, and next state from the experience replay buffer; updating the parameters of the agent_net network module through temporal difference learning; and using the target_agent_net network module to provide a stable target Q value, and learning the mapping relationship between state and action value. During the online learning phase, the following methods were adopted: The strategy involves the agent_net network module selecting actions based on the current state, interacting with the environment to collect experience, and adapting to changes in the fragment trajectory in real time.

10. The low-Earth orbit space target pre-avoidance system based on the QMix algorithm according to claim 6, characterized in that, The loss function for: in, For expectation operator, For states and joint actions, For rewards and the next state, For the next action, The global Q value, For the current network parameters, For the target network parameters, Here, D represents the experience replay pool, which is the discount factor. Represents a joint action for all agents. Find the maximum value.

Images