Transformer neural network systems and methods for enhanced orbital prediction accuracy

Abstract

Transformer neural network model for predicting trajectories of resident space objects is provided. Unlike recurrent neural networks and their subsets, the transformer model simultaneously ingests the input sequence data and employs a self-attention mechanism, to be trained by parallel processing. The developed model can predict RSOs' future trajectory for an extended time horizon. Both state vectors and Keplerian data sets were used for the training. A hybrid SGP-4 interpolation is deployed to transform unsynchronized publicly available data sets to a synchronized data set that is up-sampled to a consistent time of one minute. The transformer ingests the resulting datasets to forecast a subsequent 10-day trajectory of any RSO according to its previous 10-day history input. Then, the model can employ its own forecasted data as a new input to predict RSOs' trajectories to further extend the prediction time horizon.

Description

TRANSFORMER NEURAL NETWORK SYSTEMS AND METHODS FOR ENHANCED ORBITAL PREDICTION ACCURACYTechnical Field

[0001] The embodiments disclosed herein relate to orbital prediction modelling, and, in particular to transformer neural networks, systems and methods for enhanced orbital prediction accuracy.Introduction

[0002] Space is now more congested and contested than ever before. The top priority for both public and private sectors is to ensure satellites operate collision-free. Hence, residence space object (RSO) orbit predictions with high accuracy as one of the challenging areas in space situational awareness (SSA) is critical. Errors accumulating overtime and subsequently distorting the prediction are the most irritating drawbacks of the Simplified General Perturbations Orbit Model 4 (SGP-4) as an industry standard propagator of RSO orbits.

[0003] Ensuring a secure and sustainable space environment has never been as crucial as it is in the present day. As of August 2023, approximately 36,500 fragments of space debris larger than 10cm, around one million pieces ranging from 1cm to 10cm, and roughly 130 million fragments measuring between 1 mm and 1cm are present. These fragments pose significant risks to operational satellites, spacecraft, and the safety of personnel on board. The significance of this situation becomes even more pronounced when considering that only approximately 34,450 of these million debris fragments are actively monitored, leaving the majority of them unaccounted for (“Space debris by the numbers,” European Space Agency, accessible at https: / / www.esa.int / Space_Safety / Space_Debris / Space_debris_by_the_numbers.

[0004] Trying to keep the space safe and sustainable without collision resulted in the appearance of a necessary field of research that SSA is trying to address (P. McCormick, Space situational awareness in Europe: The fractures and the federative aspects of European space efforts., Astropolitics 13, no. 1 (2015): 43-64). Among the SSA requirements, being able to predict the RSOs orbit for a short / long future has been considered so necessary, and of course, challenging.

[0005] In an ideal scenario where no perturbations affect orbiting objects, most orbital elements remain constant. Only the true anomaly undergoes continuous changes over time and can be accurately predicted through analytical equations. However, the real world presents a different and more intricate picture. This complexity leads to interconnected relationships among orbital elements, where each element is directly or indirectly influenced by various factors, including perturbing forces (e.g., atmospheric drag, solar radiation pressure, third-bodies gravity) acting on orbiting objects, resulting in changes to other elements over time. Consequently, an accurate prediction of future trajectories for RSOs is essential to empower satellites and their operators to determine safe and risk-free operational pathways.

[0006] Previously, different approaches have been proposed to forecast the future trajectory of RSOs, from physics-based models (D. Brouwer, Solution of the problem of artificial satellite theory without drag, The Astronomical Journal 64 (1959): 378; Y. Kozai, The motion of a close earth satellite, The Astronomical Journal 64 (1959): 367; C. G. Hilton, J. R. Kuhlman, Mathematical models for the space defence centre, Philco-Ford Publication No. U-3871 17 (1966): 28; M. H. Lane, K. H. Cranford, An improved analytical drag theory for the artificial satellite problem, American Institute of Aeronautics and Astronautics, 1969; K. Cranford, An improved analytical drag theory for the artificial satellite problem, Astrodynamics Conference, p. 925. 1969; R. S. Hujsak, A Restricted Four Body Solution for Resonating Satellite with an Oblate Earth, In American Institute of Aeronautics and Astronautics Conference. 1979; F. R. Hoots, A short efficient analytical satellite theory, Journal of guidance, control, and dynamics 5, no. 2 (1982): 194-199) to data-driven models (H. Peng, X. Bai., Artificial neural network-based machine learning approach to improve orbit prediction accuracy., Journal of Spacecraft and Rockets 55, no. 5 (2018): 1248-1260; H. Peng, X. Bai., Using Artificial Neural Network in Machine Learning Approach to Improve Orbit Prediction Accuracy., 2018 Space Flight Mechanics Meeting, p. 1966. 2018; H. Peng, X. Bai., Exploring capability of support vector machine for improving satellite orbit prediction accuracy., Journal of Aerospace Information Systems 15, no. 6 (2018): 366-381 ; H. Peng, X. Bai., Comparative evaluation of three machine learning algorithms on improving orbit prediction accuracy., Astrodynamics 3 (2019): 325-343; H. Peng, X. Bai., Machine learning approach to improve satellite orbit prediction accuracy using publicly available data., The Journal of the astronautical sciences 67, no. 2 (2020): 762-793; H. Peng, X. Bai., Gaussian processes for improving orbit prediction accuracy., Acta astronautica 161 (2019): 44-56; H. Peng, X. Bai., Obtain Confidence Interval for the Machine Learning Approach to Improve Orbit Prediction Accuracy., In AAS / AIAA Astrodynamics Specialist Conference, 2018, pp. 2131-2147. Univelt Inc., 2018.

[0026] H. Peng, X. Bai, Improving Accuracy and Precision through Machine Learning Fusion using Two- Line Element Sets., p. 0863. 2022; H. Peng, X. Bai., Generalization Capability of Machine Learning Approach among Different Satellites: Validated Using TLE Data., In AAS / AIAA Astrodynamics Specialist Conference, 2018, pp. 2111-2129. Univelt Inc., 2018; H. Peng, X. Bai., A medium-scale study of using machine learning fusion to improve TLE prediction precision without external information., Acta Astronautica 204 (2023): 477-491 ; H. Peng, X. Bai., Limits of Machine Learning Approach on Improving Orbit Prediction Accuracy Using Support Vector Machine., Advanced Maui Optical and Space Surveillance (AMOS) Technologies Conference, p. 15. 2017; H. Peng, X. Bai., Improving orbit prediction accuracy through supervised machine learning., Advances in Space Research 61 , no. 10 (2018): 2628-2646).

[0007] Peng and Bai investigated the limits of machine learning in improving orbit prediction accuracy (H. Peng, X. Bai., Limits of Machine Learning Approach on Improving Orbit Prediction Accuracy Using Support Vector Machine., Advanced Maui Optical and Space Surveillance (AMOS) Technologies Conference, p. 15. 2017). In this regard, they employedavailable data, trained an SVM model, and used it to decrease orbit prediction errors in future epochs. Then, they designed and performed several numerical experiments during which some critical limits of the trained model were demonstrated. According to their investigations, using more data improves the performance of the trained model. However, after providing enough data, there are no further improvements. In addition, they discovered that the SVM model cannot be used for the far future.

[0008] In another study, Peng and Bai presented a technique that combines physicsbased orbit prediction methods with an ML technique (H. Peng, X. Bai., Improving orbit prediction accuracy through supervised machine learning., Advances in Space Research 61 , no. 10 (2018): 2628-2646). For the dynamic model part, they included significant factors with considerable contributions to the orbit prediction error and implemented all of them using Orekit as a low-level space dynamics library (L. Maisonobe, V. Pommier, P. Parraud. Orekit: An Open Source Library for Operational Flight Dynamics Applications., 4th international conference on astrodynamics tools and techniques, pp. 3-6. Paris: European Space Agency, 2010). For the machine learning modelling technique, the SVM was chosen. They used a simulation-based space catalogue environment as their test bed, and their results indicated significant reductions in the prediction error. The same team used historical data to train an ANN model. According to their results, the trained ANN was successful in different situations, specifically for different types of RSOs and future epochs (H. Peng, X. Bai., Artificial neural network-based machine learning approach to improve orbit prediction accuracy., Journal of Spacecraft and Rockets 55, no. 5 (2018): 1248- 1260).

[0009] In another study, Peng et al. explored the use of ANNs to enhance the accuracy of orbit predictions (H. Peng, X. Bai., Using Artificial Neural Network in Machine Learning Approach to Improve Orbit Prediction Accuracy., 2018 Space Flight Mechanics Meeting, p. 1966. 2018). The ANN models for each component of the orbit prediction error were trained using historical data from an RSO in an SSO. The study assessed the generalization capabilities of the trained ANNs in two ways. The results indicate that initialization is crucial for good performance, and the best network structures vary for each error component. Additionally, the results showed that the ANNs can be applied to a relatively broad range of nearby RSOs that were not used for training.

[0010] Also, they further improved the existing ML approach for the orbit prediction accuracy of RSOs by incorporating uncertainty information about the ML-modified orbit prediction error (H. Peng, X. Bai., Gaussian processes for improving orbit prediction accuracy., Acta astronautica 161 (2019): 44-56). This was achieved by using heteroscedastic GPs, which provided an estimation of the distribution of orbit prediction errors, including both means and uncertainties. The performance and capabilities of the trained GP models were first evaluated using a simulation environment. The data from publicly available RSO catalogues was then used to test the models further. The results were mostly positive, showing improvement in the accuracyof individual components and the overall position and velocity. They also addressed a few cases with poor performance and explained that these would be challenging to improve, given the limited information in the dataset.

[0011] Salleh et al. proposed that regression techniques and ANNs could be used to create an orbital propagation model using observed and historical data with limited input information, making it easier to access for any organization or individual (N. Salleh, S. S. Yuhaniz, N. F. M. Azmi, Modeling orbital propagation using regression technique and artificial neural network, International Journal on Advanced Science, Engineering and Information Technology, vol. 12, no. 3, pp. 1279-1286, 2022). Instead of using all features, training a model with just the two most impactful features saves time and reduces computational costs. Their work contributed to developing a NARX model using ANN to learn complex data distributions of the TLE dataset and the ability of the model to learn from historical data with minimal features. The evaluation of the trained model demonstrated its accuracy in orbit propagation. Additionally, as the study highlights that having too many input features can reduce accuracy; some features may fluctuate due to external factors such as perturbation and solar drag, leading to unstable models.

[0012] While there is a vast literature on physics-based models that are able to predict the future trajectories of different space objects around the Earth they have significant limitations, though. These models are mostly unable to take different uncertainties of the real world into their assumptions. If any model can, in rare cases, it would not be able to consider the disturbances roots perfectly. For instance, proposed physics-based models can only engage themselves with an approximation of atmosphere drag and solar activity. While these disturbances import huge uncertainties into the problem, they should be considered with all their details as comprehensively as possible, specifically for LEO objects. The SGP-4 can probably be considered the most famous physics-based model, which is the industry standard framework for RSOs position and velocity prediction (H. Curtis, Orbital Mechanics for Engineering Students. Butterworth-Heinemann, 2013; F. R. Hoots, R. L. Roehrich. Models for propagation of NORAD element sets. Office of Astrodynamics, 1980; D. Vallado, P. Crawford., SGP4 orbit determination., AIAA / AAS Astrodynamics Specialist Conference and Exhibit, p. 6770. 2008; D. Vallado, P. Crawford, R. Hujsak, T. S. Kelso., Revisiting Spacetrack Report# 3., AIAA / AAS Astrodynamics Specialist Conference and Exhibit, p. 6753. 2006). This model works in conjunction with two-line element (TLE) sets (H. Curtis, supra, D. A. Vallado, Fundamentals of astrodynamics and applications. Vol. 12. Springer Science & Business Media, 2001), which are extensively used for satellite tracking and space situation awareness and serve as a convenient and effective means of encoding significant information about the RSOs' orbits. For longer time horizons, the model is unable to forecast the orbit characteristics due to errors accumulation and, subsequently, distortion from the actual values (M. Z. Abnili, N. L. Azad, H. N. Oqab, G. B Dietrich, Space Domain Awareness Using Deep Continual Learning Sequence Predictors, IAC-21-A6.9.5, 72nd International Astronautical Congress, Dubai, UAE, 2021 , 25 - 29 October).

[0013] All the information regarding uncertainties and disturbances with significant limitations in their modelling is embedded in the real-world data. Hence, data-driven models can be considered an effective alternative to physics-based models due to their creation based on real-world system dynamics without noticeable simplification.

[0014] Accordingly, there is a need for a vanilla transformer neural network sequence predictor for forecasting resident space object trajectories that is trained with publicly available data from Space-track.org (https: / / www.space-track.org / auth) and Planet-Labs (https: / / ephemerides.planet-labs.com).Summary

[0015] To surmount SGP-4 imperfections, the study [1] was expanded and implemented the vanilla transformer instead of the long short-term memory (LSTM) and gated recurrent unit (GRU) networks in an orbit prediction model in the Low Earth Orbit (LEO). Unlike recurrent neural networks and their subsets, the transformer simultaneously ingests the input sequence data. Due to the functionality of the self-attention mechanism, it can also be trained by parallel processing, saving a significant amount of time.

[0016] The developed model can predict RSOs’ future trajectory for an extended time horizon. Both state vectors and Keplerian data sets have been used for the training. The hybrid SGP-4 interpolation, developed previously, is deployed to transform unsynchronized and messy publicly available data sets to a synchronized and neat one that is up-sampled to a consistent time of one minute. The transformer ingests the resulting datasets to forecast the subsequent 10- day trajectory of any RSO according to its previous 10-day history input. Then, the model can employ its own forecasted data as a new input to predict RSOs’ trajectories to further extend the prediction time horizon at the cost of losing accuracy.

[0017] Moreover, using an innovative idea to feed the Sine and Cosine of the true anomaly as two new features, instead of the true anomaly itself, into the model during the training, the transformer model results in less error and improved accuracy in the prediction of this variable compared to previous studies. Through benchmarking the developed model against SGP-4 and comparing its accuracy with our available LSTM and GRU networks, the results show a promising performance of the transformer model. The trained model displays less mean absolute percentage error and an improved correlation between actual data and 10-day forecasted results compared to the SGP-4, LSTM, and GRU models.

[0018] According to some embodiments, there is a method of forecasting trajectories of resident space objects (RSO). The method comprises receiving orbital element data for a plurality of RSOs from at least one data source. The orbital element data may be Keplerian data. The method further comprises interpolating additional orbital element data points to generate an RSO dataset with a one sample per minute time-step and generating units of data for each RSO for a set period of days, the units of data including encoder inputs, decoder inputs, and target valuesfor loss function calculation, wherein each of the encoder inputs, the decoder inputs, and the target values represent different groups of days from the set period of days.

[0019] The method further comprises training a transformer model, including an encoder block, a decoder block, and an attention layer, using the encoder inputs, the decoder inputs, the target values, and transformer variables including the Sine and Cosine functions of a true anomaly and generating predictions of RSO trajectories using the trained transformer model. The transformer variables include at least one of magnitude of angular momentum, eccentricity, right ascension, inclination, and argument of perigee.

[0020] The encoder block of the transformer model includes identical stacked layers, wherein each layer includes a multi-head self-attention sub-layer and a fully connected feedforward sub-layer, and wherein the encoder block includes an input embedding module which converts the encoder inputs into vectors. The decoder block of the transformer model receives an output from the encoder block. The attention layer of the transformer model uses a query, keys, and values to map the vectors from the encoder block to output vectors, wherein a compatibility function computes a weight matrix according to a similarity between the query and a corresponding key, and wherein the weight matrix is used to compute the predicted trajectory of the transformer model.

[0021] Performance of the trained transformer model was evaluated using an mean squared error (MSE) loss function for the validation sets to find the best iteration of the trained model, and wherein the performance of the best iteration of the trained model is evaluated using an MSE loss function for the test sets.

[0022] Other aspects and features will become apparent, to those ordinarily skilled in the art, upon review of the following description of some exemplary embodiments.Brief Description of the Drawings

[0023] The drawings included herewith are for illustrating various examples of articles, methods, and apparatuses of the present specification. In the drawings:

[0024] FIG. 1 is a diagram of using a transformer in resident space object orbit prediction;

[0025] FIG. 2 is a diagram of a transformer model for predicting orbit of resident space objects;

[0026] FIG. 3 is range of the values of true anomaly;

[0027] FIG. 4 is range of the sine and cosine of the true anomaly;

[0028] FIG. 5A is true and predicted values for magnitude of angular momentum;

[0029] FIG. 5B is true and predicted values for eccentricity;

[0030] FIG. 5C is true and predicted values for right ascension;

[0031] FIG. 5D is true and predicted values for inclination;

[0032] FIG. 5E is true and predicted values for argument of perigee; and

[0033] FIG. 5F is true and predicted values for true anomaly.Detailed Description

[0034] Various apparatuses or processes will be described below to provide an example of each claimed embodiment. No embodiment described below limits any claimed embodiment and any claimed embodiment may cover processes or apparatuses that differ from those described below. The claimed embodiments are not limited to apparatuses or processes having all of the features of any one apparatus or process described below or to features common to multiple or all of the apparatuses described below.

[0035] One or more systems described herein may be implemented in computer programs executing on programmable computers, each comprising at least one processor, a data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. For example, and without limitation, the programmable computer may be a programmable logic unit, a mainframe computer, server, and personal computer, cloud based program or system, laptop, personal data assistance, cellular telephone, smartphone, or tablet device.

[0036] Each program is preferably implemented in a high level procedural or object oriented programming and / or scripting language to communicate with a computer system. However, the programs can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Each such computer program is preferably stored on a storage media or a device readable by a general or special purpose programmable computer for configuring and operating the computer when the storage media or device is read by the computer to perform the procedures described herein.

[0037] A description of an embodiment with several components in communication with each other does not imply that all such components are required. On the contrary a variety of optional components are described to illustrate the wide variety of possible embodiments of the present invention.

[0038] Further, although process steps, method steps, algorithms or the like may be described (in the disclosure and I or in the claims) in a sequential order, such processes, methods and algorithms may be configured to work in alternate orders. In other words, any sequence or order of steps that may be described does not necessarily indicate a requirement that the steps be performed in that order. The steps of processes described herein may be performed in any order that is practical. Further, some steps may be performed simultaneously.

[0039] When a single device or article is described herein, it will be readily apparent that more than one device I article (whether or not they cooperate) may be used in place of a single device I article. Similarly, where more than one device or article is described herein (whether or not they cooperate), it will be readily apparent that a single device I article may be used in place of the more than one device or article.

[0040] When a tilde (~) is used herein, it signifies a range of ± 10% of the indicated value. For example, ~1.0 means between 0.9 to 1.1.

[0041] Vanilla Transformer.

[0042] FIG. 2 shows a transformer model 200 from A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, t. Kaiser, I. Polosukhin., Attention is all you need., Advances in neural information processing systems 30 (2017). Vaswani et al. describe the implementation of transformer networks, generally, and specifically multi-headed attention mechanisms, positionwise feed forward networks, embeddings and Softmax, positional encoding, optimizer selection, and regularization. Herein, the transformer model 200 is implemented as a vanilla transformer neural network sequence predictor for resident space objects.

[0043] The transformer 200 is a novel deep learning method benefiting from an innovative architecture. It has shown performance in solving sequence-to-sequence problems such as natural language processing (NLP) and computer vision (CV). Unlike RNNs, transformers are designed to take and process the input data all at once. This helps transformers to be faster than RNN-based models such as LSTM and GRU in learning. Moreover, transformers depend on an attention mechanism to find relations between input and output. Thanks to this mechanism, the transformer 200 has capability to look for long-range dependencies in long sequences. More importantly, it may be competent in remembering those dependencies, unlike RNN-based models, which forget long-range and older dependencies.

[0044] The transformer network 200 is composed of two main blocks: encoder 202 and decoder 204. Having an input sequence of ( i, ... ,xn), the encoder 202 maps the input to another sequence of (zi, ... ,zn). Then, the decoder 204 will generate an output sequence (yi, ... ,yn) one at a time.

[0045] Stacked self- attention and pointwise fully connected layers are used in both the encoder 202 and the 204 decoder. Instead of a self-attention layer, one of its variants, multiheaded mechanism will be repeated to the number of heads. As the number of heads increases, the model 200 will discover more accurate input sequence dependencies. However, the computational cost will be increasing and, at some point, prohibitive. Finally, the output from all heads are concatenated into one matrix and passed to the next layer.

[0046] Identical layers are stacked for the encoder block 202. Every layer includes a multihead self-attention sub-layer 212a and a fully connected feed-forward sub-layer 214a. The dmOdei = 512 is the produced output dimension for the model’s all sub-layers. Given the need to convert the input sequence to a vector or a matrix, there is an input embedding module 216 at the beginning of the encoder block 202. The embedding module 216 takes the input, transfers it to an embedding space, and converts it to a vector. Then, the vector is passed to a positional encoding module 218.

[0047] The same layers 212, 214 have been considered for the decoder block 204 as well. Additionally, there is a third multi-head attention sub-layer 220 dealing with the output of the encoder stack 202. Previously, in other attention-based models, only the final, hidden state of the encoding part is sent to the decoder 204, but in the transformer 200, the encoder 202 passes all the hidden states, even the intermediate ones. Besides that, the first multi-head attention sub-layer has been modified to hide the following positions after the current one. By doing so, the model 200 only uses previous and current data to forecast future data without seeing the following available data beforehand.

[0048] The attention layer 212 function uses a query, keys, and values to map these vectors to an output vector. To this end, a compatibility function is utilized to compute the weight matrix according to the similarity between the query and the corresponding key. These weights will be used later to compute the output. Vaswani et al. proposed a specific attention function named scaled dot-product attention, which may also be used herein.

[0049] Data Preparation.

[0050] Raw RSO data was accessed from the Space-Track.org site. According to other embodiments, the raw data is acquired from one or more data sources. As data is in the TLE format, the Keplerian orbital elements will be extracted from those TLEs first. Then, the orbital elements will be converted to state-vectors type, and the mentioned interpolation algorithm is applied to generate a neat dataset with a time-step of one sample per minute. Again, the generated dataset will be converted back to Keplerian orbital elements.

[0051] There may be numerous limitations in getting RSOs’ coordination data around the Earth, specifically for LEO objects. There are not an ample number of observing stations all over the globe and, consequently, in the best-case scenario, there are only 2-3 observations for each RSO, which is still valuable, specifically for those that are rotating the Earth 10-20 times every day. Deep learning models, like the transformer 200, may not be trained well by this amount of data. To generate more meaningful and rational data, the same hybrid SGP4 interpolation method presented in a previous study (Abnili et al., supra) was used.

[0052] To review the method, the initial state which is the Xw(t)=[a,e,i,a»,fl,0] will be propagated forward by utilizing SGP4 until the next available state (Xk+St) in equal time steps (<5t = 60 seconds). In the same way, it will be propagated backwards to its previous timestamp (Xk). Then, using weighted averaging by taking the benefit of a sigmoid function, one hundred percent weight for the initial state (Xk) and zero percent for the next timestamp are considered. Next, both weights will change step by step to zero percent for the initial state and one hundred percent for the next.

[0053] Considering 500 RSOs, one sample was generated per minute for 300 days for the RSOs. These small-time steps create a continuous and smooth trajectory for RSOs and prevent jagged and jumpy behaviours.

[0054] The RSOs are divided into two training and test sets sections with ratios of 0.8 and 0.2, respectively. Additionally, the training dataset is divided into two sections of training and validation set with ratios of 0.8 and 0.2, respectively.

[0055] The performance of the trained model 200 in the training session is evaluated using an MSE loss function on the validation dataset, and the best model is saved. Finally, thebest-saved model's performance is measured using the MSE loss function on the test dataset. The MSE equation is stated i

[0056] In the epoch, a random order of RSOs in the training dataset is chosen, and 300 days of trajectory data of each RSO will be categorized as encoder inputs (10-days units), decoder inputs (10-days units), and the third section, which is going to be used to calculate the loss function (10-days units). The very first 10-day data will be passed to the encoder, the next 10-day will go into the decoder, and eventually, the final 10-day will play the reference role for the 10-day horizon predicted states by the model, which is also used to calculate the loss value during the training.

[0057] Therefore, for the RSO, 10 groups of units of data will be available, including encoder and decoder inputs, as well as the target values for loss function calculation. In each epoch, the whole training set will be utilized. Then, for the next epoch, the RSOs order will be shuffled.

[0058] As most RSOs data come in the Keplerian type, the learning model is provided with the same data type. However, to use the hybrid SGP4 model, necessary conversions to proxy variables may have taken place. In some cases with enough data for each space object, these conversions can be neglected, and all the processes will be in Keplerian elements.

[0059] For training the model, the Keplerian format is chosen. The features of the transformer model are the six variables of the magnitude of angular momentum, eccentricity, right ascension, inclination, argument of perigee, and true anomaly. However, in the systems and methods described herein, the true anomaly is not provided directly as an input to the model.

[0060] Considering its commonly accepted definition and how it is measured and reported relative to the ECI frame, the range of true anomaly value variations will be from 0 radian to 2TT radian, as shown in Fig. 3.

[0061] However, this variation is not smooth, and there is a jump from 2TT radian to 0 in each period of the orbit. As shown in Fig. 3, these jumps force harsh discontinuities to the data and the learning model can barely learn from them and predict future sequences, specifically as different RSOs have separate orbit periods.

[0062] To deal with this problem, the true anomaly is omitted from the features vector and the Sine and Cosine functions of the true anomaly are added as two new features. As Sine and Cosine functions are continuous all the time, the learning model shows better performances in both learning and prediction, as seen in Fig. 4.

[0063] After the prediction, the Arcsine function will be used to find the true anomaly magnitude from its Sine function. Also, the Arccose function is used to resolve the quadrant ambiguity from the cosine of the true anomaly.

[0064] Theory and Calculation.

[0065] Space object orbit prediction is one of the still active research fields in astronautics and SSA. Herein is provided, a creative model for the orbit prediction problem by utilizing the vanilla transformer model 200. As a novel approach in handling time series problems, the transformer 200 ingests an input of the previous 10 days of RSOs trajectory to learn the dependencies and then get enabled to predict the future 10-day trajectories.

[0066] To assess the trained model's performance, Figs. 5A-5F display its predicted output for a randomly selected RSO in a 10-day prediction horizon. In order to alleviate the visual clutter in the Figures, a detailed view of the first 1000 minutes of the prediction horizon is included, to better discern plot details. As the object choice is random, any biases in the selection are irrelevant. Furthermore, the RMSE associated with orbital parameters is detailed in Table 1 for both the model presented in this current study and our previously introduced models. This facilitates a comparison of the model performance. Evidently, the vanilla transformer model 200 surpasses the predictive capabilities of both conventional RNNs and RNNs with continual learning across most scenarios regarding the orbital parameter prediction.Table 1 . RMSE of orbital variables using the transformer model 200 and previous models from Vaswani et al.

[0067] The initial noteworthy observation pertains to the model presented herein, which demonstrates a remarkable ability to retain information related to long-range and historical dependencies. To further clarify, the trained transformer model 200 exhibits no reduction in the RMSE when predicting elements within the defined prediction horizon. This outcome aligns with the transformer's established proficiency in retaining information associated with long-range and historical dependencies. Consequently, it is anticipated that the transformer model’s 200 performance may deteriorate if it were to utilize previously predicted elements as input to extend the prediction horizon and forecast the subsequent 10-day orbit of the RSO, although this aspectwas not thoroughly analyzed. Therefore, one of the noteworthy achievements of this model 200 is an extension of the prediction horizon while preserving accuracy.

[0068] Given the model’s varying performance across different orbital elements, it is essential to thoroughly examine the model’s performance for each element individually.

[0069] Regarding the angular momentum magnitude (FIG. 5A), the transformer model 200 demonstrates an accuracy improvement of at least approximately 21 % compared to the other models. The findings from Figure 5A align with the outcome, as the model’s predictions closely track the actual values. It is important to note that during instances of rapid value fluctuations, some divergence and disparities between the model’s output and the actual values are acceptable.

[0070] Regarding the eccentricity (FIG. 5B), once again, the Transformer model exhibits improved performance to the other models, demonstrating a minimum 14.5% enhancement in predictive accuracy. In Figure 5B, the model tracks the actual values, although, during instances of rapid changes, there are noticeable abrupt shifts and disparities between the predicted and actual values.

[0071] Concerning the inclination (FIG. 5D), the transformer model 200 significantly enhances the predictive precision by orders of magnitude. However, regarding the right ascension (FIG. 5C), the transformer model 200 does not outperform the other models trained in previous studies. The RNN model with continuity maintains an advantage for the right ascension. A similar trend is observed for the argument of perigee (FIG. 5E), where the RNN with continual learning also holds the lead. Besides the direct and indirect effects which all the orbital elements have on each other, to delve into the aforementioned observations, it’s important to note that Earth's oblateness introduces perturbations to both the right ascension and the argument of perigee, with these perturbations being influenced by the inclination (G. R. Hintz, Orbital mechanics and astrodynamics. Berlin: Springer, 2015). The prediction accuracies for the right ascension and the argument of perigee may be lower compared to that of the inclination. The complex interplay between Earth's oblateness, inclination, right ascension, and argument of perigee presents a challenge in attempting to separate their effects.

[0072] The most intriguing outcome emerges in the model’s performance on the true anomaly variable (FIG. 5F). As previously noted, this parameter undergoes nonlinear changes within 0 to 2TT radian range, a behavior comprehensible to humans yet less intuitive for a machine learning model. As anticipated, the approach to addressing this nonlinearity yields results, with the transformer model 200 achieving approximately 76% enhancement in prediction accuracy.

[0073] Regarding the performance of the model 200 on certain orbital variables while still maintaining an acceptable performance on the others, it should be noted that the model may simultaneously process all six orbital parameters and optimizing their weights collectively. The variable with the fastest dynamics, namely the true anomaly, influences the model’s predictions for the other variables. Consequently, the model aims to harmonize the rate of change across allthe variables, leading to noticeable peaks and troughs in predicted values for the slower variables like eccentricity, right ascension, inclination, and argument of perigee. This phenomenon may explain the transformer model’s less satisfactory performance in predicting the right ascension and argument of perigee. This issue arises due to the model’s attempt to apply the rapid dynamics learned from a fast-changing variable, like the true anomaly, to the prediction of a slowly evolving variable. While applying filtering and smoothing methods could alleviate this challenge; the presented results prioritize maintaining the integrity of the predictions in their raw form. These fluctuations may not pose a barrier to the prediction objective, given their nearly negligible magnitude compared to the average values.

[0074] The transformer model 200 may provide an advancement in accuracy when forecasting most orbital elements. This progress is particularly evident compared to prior methodology, which involved continual learning across various dimensions. The model’s potential for predicting RSO orbits and facilitating tracking tasks holds substantial promise within the field. The method may align with and surpass prevailing industry benchmarks regarding prediction accuracy.

[0075] Recognizing the data demands of deep learning models, purposefully expanding the dataset for orbit prediction and tracking objectives emerges. The system’s predictive capacities may be further enhanced by introducing a more extensive set of data. Additionally, as advancements in technology continue to shape data collection tools, the forthcoming improvements may unlock richer and more accurate data streams. The gradual proliferation of data collection nodes across global locations adds another layer of promise, as it offers a higher density of data points. This, in turn, reinforces the reliability of data available for studies like the current one. Consequently, this interplay of enriched data and refined tools introduces a phase of improved outcome validation, potentially elevating the model’s performance for official purposes.

[0076] The predominant issue stemmed from insufficient data to train the model adequately. Mitigating this concern involved interpolating additional data points between the reported measurements in the reference (H. Peng, X. Bai, Improving Accuracy and Precision through Machine Learning Fusion using Two-Line Element Sets., p. 0863. 2022), employing the SGP4 algorithm. A more significant challenge arises from discrepancies evident in publicly available data compared to preceding and subsequent data points, which underscores measurement errors. This issue may hamper the learning process and, consequently, the predictive performance of the trained model.

[0077] To address this issue, the system models may incorporate Kalman Filter techniques to manage errors in data points. The system may include specialized software packages like Orekit may be leveraged to establish a simulation environment that can generate artificial yet realistic data points. This approach offers the advantage of accommodating various perturbing forces, such as atmospheric drag, solar radiation pressure, and the gravitationalinfluence of third bodies like the Sun and Moon. This versatility stands in contrast to the SGP4 propagator, which solely considers Earth's gravity.

[0078] The utilization of Sine and Cosine functions in conjunction with true anomaly, as opposed to true anomaly alone, may provide an enhancement in the predictive capabilities of the trained transformer model 200. This shift may provide an improved performance that outshone the previous models. The potential of this method may extend to the inclination and the argument of perigee in forthcoming investigations. The integration of continual learning alongside the transformer model 200 holds promise for achieving heightened predictive accuracy. Another critical avenue for exploration involves examining the impact of employing distinct neural networks for each individual orbital variable on the overall performance and accuracy of the trained models.

[0079] The model may be transitioned from the basic transformer model to more sophisticated iterations such as the Autoformer (H. Wu, J. Xu, J. Wang, M. Long, Autoformer: Decomposition transformers with auto-correlation for long-term series forecasting, Advances in Neural Information Processing Systems 34 (2021): 22419-22430) can offer advantages when it comes to the prediction of RSOs' orbital trajectories and their tracking applications. Exploring these advanced Transformer variants, like the Autoformer, can unlock new opportunities for further enhancing the state-of-the-art in SSA and space object tracking.

[0080] The systems and methods for enhanced orbital prediction accuracy described herein can be applied to satellite path planning and single-object maneuver planning to avoid collisions between a primary satellite and secondary objects (e.g., space debris). By predicting the trajectories of secondary objects, and the orbital path of the primary satellite, expected collisions between the secondary objects and the primary satellite can be detected days in advance. When an expected collision is detected, an collision avoidance maneuvers for the primary satellite can be planned and performed well in advance of the expected collision to navigate the satellite away from potential collisions while also minimizing disruption to the satellite’s desired orbital path that may occur if last-minute drastic maneuvers are performed.

[0081] According to an embodiment, after predicting RSO trajectories by the systems and methods described herein, it is determined whether any one or more RSO trajectories intersect with the predicted orbital path of a satellite and the expected time of collision is determined. The predicted orbital path of the satellite may itself be predicted in the same manner as the predicted RSO trajectories. Collision avoidance maneuvers can then be planned and executed by the satellite in advance of the expected collision. Collision avoidance maneuvers may be performed incrementally, e.g., over a period of days, instead of all at once, if the expected collision is detected sufficiently early. In addition, after a collision avoidance maneuver is performed, the future trajectories of both the satellite and the RSO can be continually predicted and updated to assess the effectiveness of the collision avoidance maneuver. Furthermore, collision avoidance maneuvers may be performed semi-autonomously and or autonomously by a satellite implementing the systems and methods herein.

[0082] According to an embodiment, a hybrid approach may be implemented integrating physical laws and space domain knowledge into the transformer architecture to form physics- informed transformers for predicting RSO trajectories as well as collision avoidance maneuvers to include at least orbital dynamics, atmospheric drag, perturbations (gravitational effects from the Earth, Moon, and other celestial bodies), space weather (solar activity, atmospheric density, solar wind, coronal mass ejections, geomagnetic storms) and other space environmental factors or the like.

[0083] According to some embodiments, transformer-based models may be used for space debris tracking to support in-space services, including but not limited to tracking and predicting debris movements around the serving satellites to ensure safe proximity operations, enhancing the situational awareness of servicing spacecraft, docking operations during repairing or refueling active spacecraft, support mitigation and remediation of space debris including providing support in autonomous debris removal missions in predicting debris paths, re-entry of spacecraft, optimizing satellite positioning in constellations to avoid interference and minimize fuel consumption for station keeping operations, real time navigation, trajectory optimization and adaptive path planning during rendezvous and or docking operations.

[0084] According to an embodiment, the transformer model architecture may be used to implement a foundational model for orbit prediction and space object behavioural analysis. Transformers may be used to detect anomalies to continuously monitor RSOs using data collected from a plurality of sensors, where anomalous behaviour may be identified such as unexpected changes in velocity or acceleration or changes in RSO trajectories. In other cases, digital twins of RSOs may be used to continuously monitor the trajectories and behaviours of RSO, these digital twins can be used to compare predicted behaviors and observed behaviours using a plurality of sensors to determine operations and potential anomalies.

[0085] Digital twins of RSOs are a virtual representation of the space environment that mirrors the behaviour and movement of the RSOs, these models can be continuously updated using data ingested from various tracking and data sources to create a dynamic map of the RSOs. Furthermore, digital twins can be updated using the transformer models to predict expected trajectories of RSOs over a user-defined time period and incorporating a plurality of data sources observing the RSO for real-time monitoring and management of space assets. Additionally, digital twins can be used to assess the risk of potential collisions and simulate collision avoidance maneuvers, for planning of evasive maneuvers to identifying the optimal trajectories to minimize future collisions. These maneuvers can be simulated in a virtual environment and various scenarios can be tested to support the decision-making processes and emergency responses for space actors to better understand the potential effects of space weather, collisions, satellite fragmentation events, new debris formation, and other anomalies to their space operations.

[0086] According to an embodiment, the transformer model is used to generate data that can train other artificial intelligence (Al) systems, for example the transformer model acts apretrained foundational model which is fined tuned to train or inform other Al models to improve performance or performance other specific tasks that those Al models are designed for such as anomaly detection, monitoring satellite health, differentiating between active satellite and space debris. In other implementations, transformer models may be used to enable transfer learning, where the transformer model is used to facilitate learning in other Al models to improve performance and or perform another specific task. In some implementations learning may include zero-shot learning, representation learning, multi-modal learning, and reinforcement learning or the like.

[0087] According to an embodiment, data streams about RSOs may be produced from multiple data sources including at least one or more of the following: light-curves; inverse synthetic aperture radar; hyperspectral data; infrared sensors, Light Detection Ranging (LIDARS), Microwave Amplification by Stimulated Emission of Radiation (MASERS), a plurality of radar systems on the ground, onboard flight vehicles and / or in space, a plurality of optical telescopes on the ground, onboard flight vehicles and / or in space, a plurality of cameras on the ground, onboard flight vehicles and / or in space, or the like.

[0088] According to an embodiment, the transformer model 200 may receive RSO data from real data sources and / or simulated data sources.

[0089] According to an embodiment, the transformer model 200may receive RSO data from a plurality of nodes in a network, each node including at least a plurality of sensors and a transceiver to communicate with other nodes. Furthermore, the nodes may be situated on Earth and / or in space.

[0090] While the above description provides examples of one or more apparatus, methods, or systems, it will be appreciated that other apparatus, methods, or systems may be within the scope of the claims as interpreted by one of skill in the art.

Claims

Claims:

1. A method of forecasting trajectories of resident space objects (RSO), the method comprising: receiving orbital element data for a plurality of RSOs from at least one data source; interpolating additional orbital element data points to generate an RSO dataset with a one sample per minute time-step; generating units of data for each RSO for a set period of days, the units of data including encoder inputs, decoder inputs, and target values for loss function calculation, wherein each of the encoder inputs, the decoder inputs, and the target values represent different groups of days from the set period of days; training a transformer model, including an encoder block, a decoder block, and an attention layer, using the encoder inputs, the decoder inputs, the target values, and transformer variables including the Sine and Cosine functions of a true anomaly; and generating predictions of RSO trajectories using the trained transformer model.

2. The method of claim 1 , wherein the orbital element data is Keplerian data.

3. The method of claim 2, wherein the transformer variables includes at least one of magnitude of angular momentum, eccentricity, right ascension, inclination, and argument of perigee.

4. The method of claim 1 , wherein the encoder block of the transformer model includes identical stacked layers, wherein each layer includes a multi-head self-attention sub-layer and a fully connected feed-forward sub-layer, and wherein the encoder block includes an input embedding module which converts the encoder inputs into vectors.

5. The method of claim 1 , wherein the decoder block of the transformer model receives an output from the encoder block.

6. The method of claim 1 , wherein the attention layer use a query, keys, and values to map the vectors from the encoder block to output vectors, wherein a compatibility function computes a weight matrix according to a similarity between the query and a corresponding key, and wherein the weight matrix is used to compute the predicted trajectory of the transformer model.

7. The method of claim 1 , wherein interpolating additional orbital element data points is performed by SGP4 interpolation.

8. The method of claim 1 , wherein the RSO orbital element data is divided into two training and test sections, and each training section is divided further into training and validation sets.

9. The method of claim 8, wherein the ratios of the training and test sections are 0.8 and 0.2, respectively.

10. The method of claims 8, wherein the ratios of the training and validation sections are 0.8 and 0.2, respectively.

11. The method of claim 1 , wherein the performance of the trained transformer model is evaluated using an mean squared error (MSE) loss function for the validation sets to find the best iteration of the trained model, and wherein the performance of the best iteration of the trained model is evaluated using an MSE loss function for the test sets.

12. The method of claim 1 , wherein for each RSO a first ten days of data are used as encoder inputs, a second ten days of data are used as decoded inputs, and a third ten days of data are used as target values for predicted states to calculate loss values, and wherein the pattern of ten days for each of the encoder inputs, decoder inputs, and target values continues for the entire set period of days.

13. The method of claim 1 , wherein a hybrid SGP4 model is used and the RSO data is converted to proxy variables.

14. The method of claim 1 , wherein following generation of predictions by the trained transformer model, an Arcsine function is used to find the magnitude of the true anomaly from the Sine function.

15. The method of claim 1 , wherein following generation of predictions by the trained transformer model, an Arccose function is used to resolve quadrant ambiguity from the Cosine function.

16. The method of claim 1 , wherein the RSO orbital element data is filtered using Kalman filter techniques.

17. The method of claim 1 , wherein a simulation environment software is used to interpolate data points.

18. The method of claim 1 , wherein the method if performed by a computing device comprising at least one memory and at least one processor.

19. The method of claim 1 , wherein the at least one data source includes one or more of: observed data; simulated data; light-curves; inverse synthetic aperture radar data; hyperspectral data; infrared sensor data, Light Detection Ranging (LIDARS), Microwave Amplification by Stimulated Emission of Radiation (MASERS), a plurality of radar systems on the ground, onboard flight vehicles and / or in space; a plurality of optical telescopes on the ground, onboard flight vehicles and / or in space; a plurality of cameras on the ground, onboard flight vehicles and or in space.

20. The method of claim 1 , wherein the transformer model is a foundational model for orbit prediction and space object behavioural analysis.

21. The method of claim 20, wherein the transformer model is used to generate digital twins of the plurality of RSOs.

22. The method of claim 1 , wherein the transformer model receives the orbital element data from a plurality of nodes in a network, the nodes being positioned on Earth and / or in Space.

23. The method of claim 1 , wherein the transformer model is used to support in-space services including tracking and predicting debris movements around serving satellites.

24. The method of claim 1 , further comprising using the transformer model to train other artificial intelligence systems to improve performance or perform a specific task.

25. The method of claim 1 , further comprising: determining whether any one or more of the RSO trajectories intersect with a satellite orbital path; and determining an expected time of collision.

26. The method of claim 25, further comprising: planning one or more collision avoidance maneuvers to be preformed in advance of the expected time of collision.

27. The method of claim 26, further comprising performing the collision avoidance maneuvers semi-autonomously or autonomously by a satellite.

Images