Spacecraft behavior prediction method and system
Through reinforcement learning algorithms, multi-dimensional space state and reward functions are constructed, and prediction models are dynamically updated, which solves the problem of insufficient timeliness and accuracy of space vehicle behavior prediction in the existing technology, and realizes high-precision real-time monitoring and prediction of non-cooperative satellites and space waste.
Patent Information
- Application Number
- CN202411970497.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-30
- Publication Date
- 2025-08-22
- Estimated Expiration
- 2044-12-30
AI Technical Summary
The existing prediction methods cannot meet the real-time monitoring and high-precision prediction of space vehicles such as non-cooperative satellites and space waste at the same time, and the timeliness and accuracy are insufficient.
Reinforcement learning algorithm is adopted to build multidimensional spatial states, combine reward functions and Q-learning mechanisms, dynamically update the prediction model, and use real-time data flow for training to optimize the prediction of space vehicle behavior.
It realizes high-precision real-time monitoring and prediction of space vehicle behavior, improves the accuracy and timeliness of prediction, and adapts to complex and changeable aerospace environment.
Smart Images
Figure CN119918398B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of satellite orbit prediction, and in particular to a method and system for predicting the behavior of a spacecraft. Background Art
[0002] With the rapid development of global space activities, particularly the intensification of commercial space exploration and inter-state space exploration, the number of satellites in orbit has exploded. According to the International Space Research Institute, there are currently over 10,000 satellites in orbit, and this number is expected to double over the next decade. The diverse applications of satellites, including communications, remote sensing, weather monitoring, scientific experiments, and navigation and positioning, have greatly promoted the advancement of human science and technology. However, the resulting issues of non-cooperative satellites and space debris have gradually attracted widespread attention.
[0003] Non-cooperative satellites are typically those that do not proactively provide status information, are not properly coordinated, and may pose a potential threat to other satellites or space missions. The behavior of these satellites is unpredictable and may occur due to malfunction, loss of control, abandonment, or intentional malicious action. Predicting their behavior is not only critical to the safety of space missions but also poses significant challenges to ground station operations, the protection of space assets, and national security.
[0004] Reinforcement learning, a key branch of machine learning, is particularly well-suited for solving decision-making problems in dynamic, complex, and uncertain environments. By interactively learning and optimizing strategies with the environment, intelligent agents can gradually improve the success rate of tasks in unknown environments. In the prediction of non-cooperative satellite behavior, reinforcement learning can guide intelligent agents to explore and exploit the complex dynamic behaviors of non-cooperative satellites through reward mechanisms. Therefore, non-cooperative satellite behavior prediction algorithms based on reinforcement learning have gradually become a hot topic of research.
[0005] In recent years, several international space agencies and research organizations have begun exploring reinforcement learning-based technologies for predicting the behavior of non-cooperative satellites. For example, NASA, ESA, and other organizations have conducted a series of related studies aimed at improving their monitoring and prediction capabilities for non-cooperative satellites. Furthermore, academics both domestically and internationally have conducted extensive research in this area, proposing a variety of innovative algorithms based on deep reinforcement learning. These studies not only provide a new theoretical foundation for predicting the behavior of non-cooperative satellites, but also offer a feasible technical approach for practical application.
[0006] Although existing research has made some progress, it still faces many challenges. First, how to effectively construct the state space and action space so that the reinforcement learning algorithm can accurately reflect the complex behavior of non-cooperative satellites still requires in-depth exploration. Second, the design of the reward function is crucial. How to quantify the risks and impacts of non-cooperative satellite behavior and how to guide learning through a reasonable reward mechanism are both urgent issues that need to be addressed. In addition, to meet the demand for real-time prediction of non-cooperative satellite behavior, how to achieve fast and efficient online learning and decision-making is also an important direction for future research.
[0007] In summary, with the increasing number of non-cooperative satellites and the intensifying potential threats they pose, traditional behavior prediction methods are no longer able to effectively cope with the complex and changing space environment. Reinforcement learning-based non-cooperative satellite behavior prediction algorithms, with their adaptability and intelligent decision-making capabilities, demonstrate broad application prospects. In the future, with continued algorithm optimization, dataset construction, and technological maturity, reinforcement learning-based non-cooperative satellite behavior prediction technology will provide strong support for space safety and further promote the development of the space industry.
[0008] Traditional methods for predicting the behavior of non-cooperative satellites rely heavily on physical models and empirical rules. These methods typically assume that satellite motion follows the laws of classical mechanics and rely on precise orbital data and environmental models. However, this approach exhibits significant limitations in the ever-changing space environment. Since the orbital data of non-cooperative satellites may lack real-time updates, and their behavior is often affected by a variety of complex factors, such as interactive collisions, gravitational perturbations, and solar radiation pressure, the effectiveness of the prediction model is reduced. Furthermore, traditional methods struggle to adapt to the complexity and randomness of non-cooperative satellite behavior, resulting in insufficient timeliness and accuracy in predictions, making them unable to meet the demands of modern space missions for refinement and efficiency.
[0009] With the rapid development of machine learning, particularly deep learning, data-driven approaches have provided new insights into the prediction of non-cooperative satellite behavior. Deep learning can automatically extract features from large datasets, capturing the underlying patterns in non-cooperative satellite behavior and significantly improving the adaptability and flexibility of prediction models. However, deep learning algorithms typically require large amounts of labeled data, making constructing training sets a daunting task for the relatively scarce data environment of non-cooperative satellites. Furthermore, the black-box nature of deep learning algorithms makes their decision-making processes difficult to interpret, complicating their application in space missions and preventing them from simultaneously ensuring both timely and accurate predictions.
[0010] Therefore, it is necessary to provide a method and system for predicting spacecraft behavior, which can realize real-time monitoring and high-precision prediction of spacecraft such as non-cooperative satellites and space debris, and overcome the problems of insufficient timeliness and accuracy of prediction.
[0011] The above information disclosed in this Background section is only for enhancement of understanding of the background of the application and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the Invention
[0012] The main purpose of the present invention is to solve the problem that existing prediction methods cannot simultaneously meet the timeliness and accuracy of predictions, and to provide a method and system for predicting spacecraft behavior that can achieve real-time monitoring and high-precision prediction of spacecraft such as non-cooperative satellites and space debris.
[0013] To achieve the above-mentioned object, the present invention provides a first aspect of a method for predicting spacecraft behavior, comprising the following steps:
[0014] S1: Obtain the spatial state of the spacecraft at multiple consecutive moments;
[0015] S2: Initialize the benefit value and select a space state as the current space state;
[0016] S3: Calculate the reward value for executing a flying action in the current space state;
[0017] S4: Obtain the next space state after the current space state completes the flight action;
[0018] S5: Calculate the benefit value of executing all the next flight actions in the next space state;
[0019] S6: Update the benefit value based on the benefit value, reward value, and benefit value of executing all next flight actions in the next space state;
[0020] S7: taking the next spatial state as the current spatial state;
[0021] S8: Repeat steps S4 to S7 until the profit value is greater than the specified profit value;
[0022] S9: Selecting a flight action with the maximum benefit value as the predicted action of the space vehicle's behavior based on the current space state and benefit value of the space vehicle.
[0023] As an exemplary embodiment of the present invention, in step S1, the spatial state includes spatial position, velocity, acceleration and environmental parameters.
[0024] As an example embodiment of the present invention, in step S3, the reward value is calculated using Formula 1:
[0025] R(s,a)=R accuracy +R stability -R penalty +R cooperationFormula 1;
[0026] Among them, R(s, a) represents the reward value, s represents the current space state, a represents a flight action performed in the current space state, and R accuracy represents the prediction error reward, R stability represents the trajectory stability reward, R penalty represents the stability reward, R cooperation Represents the distance bonus between a spacecraft and other satellites.
[0027] As an example embodiment of the present invention, the prediction error reward is calculated using Formula 2:
[0028] R accuracy =-||P actual -P predicted || Formula 2;
[0029] Among them, R accuracy represents the prediction error reward, P actual represents the actual position of the spacecraft, P predicted Represents the predicted position of a space vehicle.
[0030] As an example embodiment of the present invention, the trajectory stability reward is calculated using Formula 3:
[0031] R stability =-||△a|| Formula 3;
[0032] Among them, R stability represents the trajectory stability reward, and △a represents the difference between the current acceleration and the previous acceleration.
[0033] As an example embodiment of the present invention, the difference between the current acceleration and the previous acceleration is equal to the current acceleration minus the previous acceleration.
[0034] As an example embodiment of the present invention, the smoothness reward is calculated using Formula 4:
[0035] R penalty =C penalty ×|△a| Formula 4;
[0036] Among them, R penalty represents the stability reward, C penalty is a constant, C penalty Indicates the penalty intensity, and △a indicates the difference between the current acceleration and the previous acceleration.
[0037] As an example embodiment of the present invention, the distance reward between the spacecraft and other satellites is calculated using Formula 5:
[0038] Rcooperation =C cooperation ×(D threshold -D current ) Formula 5;
[0039] Among them, R cooperation represents the distance reward between the spacecraft and other satellites, C cooperation is a constant that determines the intensity of the distance reward; D threshold represents the ideal distance threshold; D current Indicates the actual distance between the current spacecraft and other satellites.
[0040] As an example embodiment of the present invention, in step S6, the revenue value is updated according to the revenue value, the reward value and the next space state using Formula 6:
[0041]
[0042] Formula 6;
[0043] Among them, Q(s, a) represents the benefit value of executing a flight action in the current spatial state, s represents the current spatial state, a represents a flight action executed in the current spatial state, R(s, a) represents the reward value, s represents the current spatial state, a represents a flight action executed in the current spatial state, Q(s', a') represents the benefit value of executing the next flight action in the next spatial state, s' represents the next spatial state, a' represents the next flight action executed in the next spatial state, It represents the maximum benefit value that can be obtained by executing all the next flight actions in the next space state, α represents the learning rate, α is a constant, γ is the discount factor, and γ is a constant.
[0044] According to a second aspect of the present invention, the present invention provides a space vehicle behavior prediction system, comprising: a space state acquisition module, a benefit value storage module, a benefit value calculation module, and a benefit value output module;
[0045] The space state acquisition module is used to acquire the space state of the spacecraft at multiple consecutive moments;
[0046] The profit value calculation module is connected to the spatial state acquisition module and the profit value storage module, and is used to initialize the profit value, select a spatial state as the current spatial state; calculate the reward value of executing a flight action in the current spatial state; obtain the next spatial state after the current spatial state completes the flight action; calculate the profit value of executing all next flight actions in the next spatial state; update the profit value based on the profit value, the reward value, and the profit value after executing all next flight actions in the next spatial state; set the next spatial state as the current spatial state; and repeatedly update the profit value until the profit value exceeds the specified profit value.
[0047] The benefit value output module is connected to the benefit value storage module and is used to select a flight action with the maximum benefit value as a predicted action of the spacecraft behavior according to the current space state and benefit value of the spacecraft.
[0048] This solution first constructs a multidimensional spatial state, including the trajectory, speed, acceleration, etc. of the spacecraft. By comparing it with historical data, the algorithm uses the strategy evaluation mechanism of reinforcement learning to dynamically update the prediction model, analyze the behavior patterns of spacecraft in different time periods, and combine it with real-time data streams for training. It has adaptive and intelligent decision-making capabilities, achieves high-precision prediction of satellite behavior, overcomes the problems of insufficient timeliness and accuracy of prediction, and shows broad application prospects. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] The above and other objects, features, and advantages of the present application will become more apparent by describing in detail exemplary embodiments thereof with reference to the accompanying drawings. The drawings described below are merely some embodiments of the present application, and it is apparent to those skilled in the art that other drawings can be derived from these drawings without inventive effort.
[0050] Figure 1 A diagram schematically illustrates the steps of a method for predicting the behavior of a spacecraft. DETAILED DESCRIPTION
[0051] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be embodied in many forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art. Like reference numerals in the drawings represent like or similar parts, and thus repetitive description thereof will be omitted.
[0052] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps etc. can be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.
[0053] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.
[0054] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.
[0055] It should be understood that although the terms first, second, third, etc. may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Thus, the first component discussed below could be referred to as the second component without departing from the teachings of the present invention. As used herein, the term "and / or" includes any one and all combinations of one or more of the associated listed items.
[0056] Those skilled in the art will understand that the drawings are merely schematic diagrams of example embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing the present application, and therefore cannot be used to limit the scope of protection of the present application.
[0057] According to a first specific embodiment of the present invention, the present invention provides a space vehicle behavior prediction system, comprising: a space state acquisition module, a benefit value storage module, a benefit value calculation module, and a benefit value output module.
[0058] The space state acquisition module is used to acquire the space state of the spacecraft at multiple consecutive moments.
[0059] The benefit value calculation module is connected to the space state acquisition module and the benefit value storage module, and is used to initialize the benefit value, select a space state as the current space state; calculate the reward value of performing a flight action in the current space state; obtain the next space state after the current space state completes the flight action; calculate the benefit value of performing all next flight actions in the next space state; update the benefit value according to the benefit value, the reward value and the benefit value after performing all next flight actions in the next space state; use the next space state as the current space state; and repeatedly update the benefit value until the benefit value is greater than the specified benefit value.
[0060] The benefit value output module is connected to the benefit value storage module and is used to select a flight action with the maximum benefit value as a predicted action of the spacecraft behavior according to the current space state and benefit value of the spacecraft.
[0061] According to a second embodiment of the present invention, the present invention provides a method for predicting the behavior of a spacecraft, using the prediction system for the behavior of a spacecraft of the first embodiment, such as Figure 1 As shown, the following steps are included:
[0062] S1: Obtain the spatial state of the spacecraft at multiple consecutive moments.
[0063] The spatial state includes spatial position, velocity, acceleration and environmental parameters.
[0064] The space status is obtained as follows:
[0065] 1. Predicted orbit data
[0066] Space vehicle prediction is an important process to ensure the normal operation of space vehicles in orbit. It mainly includes orbit calculation, environmental factor analysis, attitude prediction and mission execution, so as to better ensure the smooth operation of satellites.
[0067] Input parameters:
[0068] TLE (two row elements) data:
[0069] Line 1: 141456U 16025A 24066.14192917.00005348 00000+0 57368-309997;
[0070] The meaning of the first line in order is:
[0071] 1: row number (always 1), the data of the first row;
[0072] 41456: The satellite's international designator (NORAD Catalog Number), used to uniquely identify the satellite;
[0073] U: Satellite status identifier (U means undetermined status).
[0074] 16025A: International designator of the launch, indicating that the satellite is the Ath satellite of the 25th launch in 2016;
[0075] 24066.14192917: The day number and decimal part represent the number of days since January 1, 1950 (24066 represents the 24066th day, and the following number is the partial day of that day);
[0076] .00005348: orbital decay rate, in cycles / day 2 ;
[0077] 00000+0: represents the correction value of orbital attenuation, usually used to supplement the information of orbital attenuation;
[0078] 57368-3: Orbital increments and units, usually expressed to one significant figure;
[0079] 0: the epoch number of the orbit (usually zero);
[0080] 9997: Checksum digits, usually used for data integrity verification;
[0081] Line 2: 241456 98.1738 83.43360022473 0.2493359.873014.88888585419294
[0082] The meanings of the second line in order are:
[0083] 2: row number (always 2);
[0084] 41456: International designator of the satellite, same as the first row;
[0085] 98.1738: orbital inclination, in degrees;
[0086] 83.4336: the longitude of the satellite's ascending node, in degrees;
[0087] 0022473: The satellite's orbital eccentricity, given as a value expressed as a valid figure;
[0088] 0.2493: perigee altitude, which is the altitude relative to the Earth's mean radius;
[0089] 359.8730: Orbital phase, in degrees, indicating the position of the satellite in its orbit;
[0090] 14.88888585419294: The satellite's orbital period, in minutes.
[0091] Start Time: The start time of orbit prediction, in UTC format.
[0092] End Time: The end time of orbit prediction, in UTC format.
[0093] Based on the input parameters, the orbital status information is output, including the changes in position (longitude, latitude, altitude) and velocity (velocity components) within a specified time period. The orbital status information is calculated as follows:
[0094] (1) Initialize the environment: Build the required ground data directory so that the program can access the necessary geophysical data and parameters required for orbit calculations.
[0095] (2) Parsing TLE data:
[0096] Extract satellite orbital elements based on TLE data, including:
[0097] semi-major axis a;
[0098] Eccentricity e;
[0099] orbital inclination i;
[0100] Longitude of ascending node Ω;
[0101] Argument of perigee ω;
[0102] Mean anomaly M.
[0103] (3) Create the initial track:
[0104] Based on the parsed TLE data, the initial orbit is generated according to the extracted orbital elements. The orbit object will be used for subsequent orbit propagation calculations. The Kepler orbit equation is used:
[0105]
[0106] Where E is the eccentric anomaly, p = a(1-e 2 ) is the semi-diameter, and μ is the standard gravitational parameter of the Earth.
[0107] (4) Define the time range:
[0108] Determine the start and end time of the orbit forecast and convert them into the required format for time calculation.
[0109] (5) Setting up track propagator:
[0110] Create an orbit propagator and select the numerical integration method to perform orbit propagation. Set the time step so that the orbit state is evaluated over the entire time range.
[0111]
[0112] Where r is the position vector and v is the velocity vector.
[0113] (6) Execution orbit prediction:
[0114] The calculation is carried out step by step within the time range. At each time point tn, the state of the satellite is calculated using the orbit propagator. For each time point, the state of the current satellite is calculated using the orbit propagator, and its position information and velocity information are obtained as follows:
[0115] r n+1 =r n +V n ·dt
[0116]
[0117] (7) Record and output results:
[0118] The orbital state (position and velocity) at each time point is recorded.
[0119] 2. Environment modeling and state definition
[0120] Environmental modeling is crucial for predicting spacecraft behavior. The state of a spacecraft moving in space can be described by a series of parameters that reflect its position and motion. We define a state space S, where each state s can be represented by a multidimensional vector containing the satellite's spatial position, velocity, and other possible characteristics.
[0121] Input parameters:
[0122] Position coordinates: (x, y, z) (meters)
[0123] Velocity component: (V x , V y , V z )(meters per second)
[0124] Acceleration: (a x , a y , a z )(meters per square second)
[0125] x, y, and z represent the position coordinates of the satellite in three-dimensional space. These coordinates are expressed in the geocentric inertial coordinate system (ECI) with the center of the earth as the origin.
[0126] V x , V y , V z are the components of the satellite's velocity in all directions, measured in meters per second (m / s).
[0127] The specific calculation is as follows:
[0128] (1) Initialize the state vector: Initialize the state vector s (i.e., spatial state) based on the input position information, velocity information, and acceleration information.
[0129] (2) Environmental parameter modeling: Based on specific environmental conditions, a mathematical model of the effects of gravity, air resistance, and solar radiation pressure is established.
[0130] (3) Status update: The state of the satellite is updated and predicted using the mathematical model of environmental parameters, that is, the current space state is updated using the state transfer equation:
[0131]
[0132] Here, Δt is the time interval.
[0133] (4) Integrate the output state vector: Integrate the updated position, velocity, and acceleration information into a new state vector s.
[0134] (5) Output state vector: Output the adjusted spatial state vector s.
[0135] Output: State vector s: a multidimensional vector representing the current spatial state of the spacecraft.
[0136] s=[x,y,z,V x , V y , V z , a x , a y , a z , environmental parameters].
[0137] Environmental parameters include gravity, air resistance, solar radiation pressure and other environmental parameters.
[0138] S2: Initialize the benefit value and select a space state as the current space state.
[0139] The benefit value is the Q value. Initialize the benefit value: initialize the Q value corresponding to each space state and each flight action, usually set to zero.
[0140] S3: Calculate the reward value for executing a flying action in the current space state.
[0141] The reward function plays a crucial role in reinforcement learning, especially in Q-learning frameworks. It not only guides the agent's learning process but also directly influences the final learning outcome. In the case of non-cooperative satellite behavior prediction, we design a reward function to evaluate the quality of satellite behavior.
[0142] The goal of the reward function is to take into account multiple factors in order to provide feedback to the agent (space vehicle) at each time step about the quality of its behavior, thereby optimizing its decision-making process. In particular, the following aspects should be considered:
[0143] Prediction accuracy: The error between the predicted position of the satellite in its current state and its actual position.
[0144] Trajectory stability: The smoothness of satellite trajectory changes to avoid sudden acceleration or deceleration.
[0145] The reward value is calculated using Formula 1:
[0146] R(s,a)=R accuracy +R stability -R penalty +R cooperation Formula 1;
[0147] Among them, R(s, a) represents the reward value, s represents the current space state, a represents a flight action performed in the current space state, and R accuracy represents the prediction error reward, R stability represents the trajectory stability reward, R penalty represents the smoothness reward (i.e., imposes a penalty on sharply changing acceleration), R cooperation Represents the distance bonus between a spacecraft and other satellites.
[0148] The prediction error reward is calculated using Formula 2:
[0149] R accuracy =-||P actual -P predicted || Formula 2;
[0150] Among them, R accuracy represents the prediction error reward, P actual represents the actual position of the spacecraft, P predicted Represents the predicted position of a spacecraft. The actual position of a spacecraft can be obtained historically or from a database of actual spacecraft positions. These can be put together to form a more comprehensive spatial state. That is, the spatial state includes both the actual position and the predicted position.
[0151] The purpose of the prediction error reward is to evaluate the prediction accuracy of the satellite. The smaller the error, the higher the reward value.
[0152] The trajectory stability reward is calculated using Formula 3:
[0153] R stability =-||△a|| Formula 3;
[0154] Among them, R stability represents the trajectory stability reward, △a represents the difference between the current acceleration and the previous acceleration, and ||△a|| represents the norm of the △a vector.
[0155] The difference between the current acceleration and the previous acceleration is equal to the current acceleration minus the previous acceleration. The purpose is to evaluate the change in acceleration and understand the smoothness of the space vehicle's motion.
[0156] Trajectory stability bonus: Smaller acceleration changes mean smoother trajectories, leading to higher rewards.
[0157] The stability reward is calculated using Formula 4:
[0158] R penalty =C penalty ×|△a| Formula 4;
[0159] Among them, R penalty represents the stability reward, C penalty is a constant, C penalty Indicates the penalty intensity, and △a indicates the difference between the current acceleration and the previous acceleration.
[0160] The difference between the current acceleration and the previous acceleration is equal to the current acceleration minus the previous acceleration.
[0161] The purpose of the smoothness bonus is to penalize sharply changing accelerations to avoid jerky motion.
[0162] The distance bonus between a spacecraft and other satellites is calculated using Formula 5:
[0163] R cooperation =C cooperation ×(D threshold -D current ) Formula 5;
[0164] Among them, R cooperation represents the distance reward between the spacecraft and other satellites, C cooperation is a constant that determines the intensity of the distance reward; D threshold represents the ideal distance threshold; D current Indicates the actual distance between the current spacecraft and other satellites.
[0165] R accuracy 、R stability 、R cooperation and R penalty Synthesize the final reward: Substitute the various reward values into the comprehensive calculation formula 1 to obtain the final reward R. Update the Q-value strategy: Based on the reward R, use the Q-learning algorithm to update the spacecraft's strategy to optimize its behavior.
[0166] S4: Obtain the next space state after the current space state completes the flight action.
[0167] Q-learning is a value-based reinforcement learning algorithm. Q (s, a) represents the expected reward for taking action a (a∈A) at a given spatial state s (s∈S). The action that maximizes the reward is then selected based on the Q value. Here, S is the set of spatial states, and A is the set of action.
[0168] S5: Calculate the benefit value of executing all next flight actions in the next space state.
[0169] S6: Update the benefit value based on the benefit value, reward value, and benefit value of executing all next flight actions in the next space state.
[0170] The profit value is updated according to the profit value, reward value and next space state using formula 6:
[0171]
[0172] Formula 6;
[0173] Among them, Q(s, a) represents the benefit value of executing a flight action in the current spatial state, s represents the current spatial state, a represents a flight action executed in the current spatial state, R(s, a) represents the reward value, s represents the current spatial state, a represents a flight action executed in the current spatial state, Q(s', a') represents the benefit value of executing the next flight action in the next spatial state, s' represents the next spatial state, a' represents the next flight action executed in the next spatial state, It represents the maximum benefit value that can be obtained by executing all the next flight actions in the next space state, α represents the learning rate, α is a constant, γ is the discount factor, and γ is a constant.
[0174] The learning rate is used to update the step size of the Q value and determines the weight of the old and new Q values; the discount factor determines the importance of future rewards.
[0175] Updated Q value Q(s,a): represents the expected long-term reward (benefit value) of taking flight action a in space state s.
[0176] S7: Set the next spatial state as the current spatial state.
[0177] S8: Repeat steps S4 to S7 until the profit value is greater than the specified profit value.
[0178] S9: Selecting a flight action with the maximum benefit value as the predicted action of the space vehicle's behavior based on the current space state and benefit value of the space vehicle.
[0179] Update the Q value and continuously perform offline learning to obtain the optimal strategy and select the best trajectory from the calculated predicted trajectory.
[0180] This approach first constructs a multidimensional spatial state, encompassing the spacecraft's trajectory, velocity, acceleration, and other information. By comparing this state with historical data, the algorithm leverages a reinforcement learning strategy evaluation mechanism to dynamically update the prediction model, analyze the spacecraft's behavior patterns over different time periods, and integrate training with real-time data streams. This approach optimizes strategies through interactive learning with the environment, enabling intelligent agents (entities that learn strategies through interaction with the environment to maximize their accumulated rewards) to gradually improve the success rate of missions in unknown environments. This reward mechanism guides the agent's exploration and exploitation, adapting to the complex dynamic behavior of spacecraft. The approach possesses adaptive and intelligent decision-making capabilities, enabling high-precision predictions of satellite behavior, overcoming the challenges of insufficient timeliness and accuracy in predictions, and demonstrating broad application prospects.
[0181] The exemplary embodiments of the present invention are specifically shown and described above. It should be understood that the present invention is not limited to the detailed structure, configuration or implementation described herein; on the contrary, the present invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.
Claims
1. A method for predicting spacecraft behavior, characterized in that: The following steps are involved: S1: Obtain the spatial state of the spacecraft at multiple consecutive moments; The spatial state includes spatial position, velocity, acceleration and environmental parameters; the environmental parameters include gravity, air resistance and solar radiation pressure; S2: Initialize the benefit value and select a space state as the current space state; S3: Calculate the reward value for executing a flying action in the current space state; The reward value is calculated using Formula 1: R(s, a) = R accuracy +R stability -R penalty +R cooperation Equation 1; Among them, R(s, a) represents the reward value, s represents the current space state, a represents a flight action performed in the current space state, and R accuracy represents the prediction error reward, R stability represents the trajectory stability reward, R penalty represents the stability reward, R cooperation Represents the distance reward between the spacecraft and other satellites; S4: Obtain the next space state after the current space state completes the flight action; S5: Calculate the benefit value of executing all the next flight actions in the next space state; S6: Update the benefit value based on the benefit value, reward value, and benefit value of executing all next flight actions in the next space state; S7: taking the next spatial state as the current spatial state; S8: Repeat steps S4 to S7 until the profit value is greater than the specified profit value; S9: Selecting a flight action with the maximum benefit value as the predicted action of the space vehicle's behavior based on the current space state and benefit value of the space vehicle.
2. The method for predicting spacecraft behavior according to claim 1, characterized in that: The prediction error reward is calculated using Formula 2: R accuracy =-||P actual -P predicted || Formula 2; Among them, R accuracy represents the prediction error reward, P actual represents the actual position of the spacecraft, P predicted Represents the predicted position of a space vehicle.
3. The method for predicting spacecraft behavior according to claim 1, wherein: The trajectory stability reward is calculated using Formula 3: R stability =-||△a|| Formula 3; Among them, R stability represents the trajectory stability reward, and △a represents the difference between the current acceleration and the previous acceleration.
4. The method for predicting spacecraft behavior according to claim 3, wherein: The difference between the current acceleration and the previous acceleration is equal to the current acceleration minus the previous acceleration.
5. The method for predicting spacecraft behavior according to claim 1, wherein: The smoothness reward is calculated using Formula 4: R penalty =C penalty ×|△a| Formula 4; Among them, R penalty represents the stability reward, C penalty is a constant, C penalty Indicates the penalty intensity, and △a indicates the difference between the current acceleration and the previous acceleration.
6. The method for predicting spacecraft behavior according to claim 1, wherein: The distance bonus between the spacecraft and other satellites is calculated using Formula 5: R cooperation =C cooperation ×(D threshold -D current ) Formula 5; Among them, R cooperation represents the distance reward between the spacecraft and other satellites, C cooperation is a constant that determines the intensity of the distance reward; D threshold represents the ideal distance threshold; D current Indicates the actual distance between the current spacecraft and other satellites.
7. The method for predicting spacecraft behavior according to claim 1, wherein: In step S6, the profit value is updated according to the profit value, the reward value and the next space state using formula 6: Among them, Q(s, a) represents the benefit value of executing a flight action in the current spatial state, s represents the current spatial state, a represents a flight action executed in the current spatial state, R(s, a) represents the reward value, s represents the current spatial state, a represents a flight action executed in the current spatial state, Q(s', a') represents the benefit value of executing the next flight action in the next spatial state, s' represents the next spatial state, a' represents the next flight action executed in the next spatial state, It represents the maximum benefit value that can be obtained by executing all the next flight actions in the next space state, α represents the learning rate, α is a constant, γ is the discount factor, and γ is a constant.
8. A spacecraft behavior prediction system, characterized in that: include: Space state acquisition module, income value storage module, income value calculation module, income value output module; The space state acquisition module is used to acquire the space state of the spacecraft at multiple consecutive moments; The spatial state includes spatial position, velocity, acceleration and environmental parameters; the environmental parameters include gravity, air resistance and solar radiation pressure; The income value storage module is used to store income values; The benefit value calculation module is connected to the space state acquisition module and the benefit value storage module, and is used to initialize the benefit value and select a space state as the current space state; Calculate the reward value for executing a flight action in the current spatial state; obtain the next spatial state after the current spatial state completes the flight action; calculate the benefit value of executing all next flight actions in the next spatial state; update the benefit value based on the benefit value, the reward value, and the benefit value of executing all next flight actions in the next spatial state; set the next spatial state as the current spatial state; repeatedly update the benefit value until the benefit value is greater than the specified benefit value; The reward value is calculated using Formula 1: R(s, a) = R accuracy +R stability -R penalty +R cooperation Formula 1; Among them, R(s, a) represents the reward value, s represents the current space state, a represents a flight action performed in the current space state, and R accuracy represents the prediction error reward, R stability represents the trajectory stability reward, R penalty represents the stability reward, R cooperation Represents the distance reward between the spacecraft and other satellites; The benefit value output module is connected to the benefit value storage module and is used to select a flight action with the maximum benefit value as a predicted action of the spacecraft behavior according to the current space state and benefit value of the spacecraft.