Satellite clock error time sequence prediction method and system based on deep reinforcement learning

Abstract

The invention relates to the technical field of satellite navigation and time synchronization, in particular to a satellite clock error time sequence prediction method and system based on deep reinforcement learning. According to the method, a noise gating convolution module is adopted to extract features, and the features comprise a data path, a gating path and a noise sensing path; the noise sensing path is used for extracting local frequency jitter of the historical clock error sequence and performing weighted statistics to obtain background noise intensity; generating a self-adaptive gating coefficient by combining the gating path with the background noise intensity; and the data path is used for performing expansion convolution processing on the historical clock error sequence to obtain convolution features, and multiplying the convolution features by the adaptive gating coefficient to obtain effective features as output of the noise gating convolution module. According to the invention, the background noise intensity of satellite clock error data can be sensed in real time; by dynamically adjusting the gating threshold, false features can be filtered in a strong noise environment, tiny clock difference features can be reserved in a low noise environment, and the anti-noise capability in a non-stationary environment is improved.

Description

Technical Field

[0001] This invention relates to the field of satellite navigation and time synchronization technology, and in particular to a satellite clock bias timing prediction method and system based on deep reinforcement learning. Background Technology

[0002] Satellite clock bias prediction technology is one of the key technologies for accurate satellite navigation and positioning, and its accuracy has a significant impact on satellite navigation and positioning. As the core payload of navigation satellites, the frequency stability and time accuracy of on-orbit atomic clocks directly determine the service performance of the entire navigation system. However, due to the influence of various factors such as the complex space environment, aging, and frequency drift, there is an unavoidable deviation between the physical time output by on-orbit atomic clocks and the ideal standard time, which is known as satellite clock bias.

[0003] Current prediction methods are mainly divided into traditional physical statistical models and emerging deep learning methods. Although deep learning methods, with their powerful nonlinear mapping capabilities, significantly outperform traditional models in capturing the complex characteristics of clock bias variations, they still face the following challenges when dealing with the complex environment of satellite operation in orbit: (1) Existing models mostly use feature extraction methods with fixed parameters, which cannot be adaptively adjusted according to the dynamic changes in environmental noise intensity, making it difficult to balance effective noise reduction and preservation of minute physical features in complex spatial environments.

[0004] (2) Existing methods lack overall constraints on future long-term trajectories, which leads to the continuous amplification of early biases during continuous multi-step prediction, causing the prediction results to diverge and deviate from the true physical evolution trend of atomic clocks, resulting in cumulative errors. Summary of the Invention

[0005] To overcome the shortcomings of existing technologies, such as the easy accumulation of satellite clock error prediction and low accuracy, this invention proposes a satellite clock error timing prediction method based on deep reinforcement learning, which can achieve high-precision satellite clock error prediction.

[0006] This invention proposes a satellite clock error timing prediction method based on deep reinforcement learning. First, a clock error prediction model is trained on a known clock error sequence, which predicts the next clock error based on a historical clock error sequence of length N. Then, the observed clock errors at the most recent N time steps are used as historical clock error sequences and input into the clock error prediction model to obtain the predicted value of the next clock error. The model structure includes a sequentially connected feature extraction layer, an attention enhancement layer, a two-stream evaluation module, and an aggregation layer. The feature extraction layer extracts effective features from the historical clock difference sequence. The attention enhancement layer processes the effective features to obtain attention-enhanced multi-scale temporal features. The two-stream evaluation module processes the multi-scale temporal features and the data features of the historical clock difference sequence to generate different next-step clock difference predictions. The aggregation layer evaluates the Q-value function Q(s,a) of each next-step clock difference prediction generated by the two-stream evaluation module. Then, the next-step clock difference prediction corresponding to the largest Q(s,a) is output as the clock difference prediction result. The feature extraction layer uses a noise-gated convolutional module; the noise-gated convolutional module includes a data path, a gating path, and a noise-sensing path. The noise sensing pathway is used to extract local frequency jitter from the historical clock difference sequence x and perform weighted statistics to obtain the background noise intensity. Gated paths combined with background noise intensity Generate adaptive gating coefficients The data path is used to perform dilated convolution processing on the historical clock difference sequence x to obtain convolutional features, which are then combined with adaptive gating coefficients. The effective features obtained by multiplication are used as the output of the noise-gated convolution module.

[0007] Preferably, the feature extraction layer contains three branches, each of which uses a noise-gated convolution module; the data paths in the three branches are configured with different dilation rates for dilated convolution processing, and the effective data from the three branches constitute the output of the feature extraction layer; The attention enhancement layer generates an attention weight vector based on the concatenation of the effective features output from the three branches of the feature extraction layer. Then, the attention weight vector is used to adaptively weight and fuse the effective features output from the three branches to generate enhanced multi-scale temporal features.

[0008] Preferably, the calculation formula for the noise perception path is as follows: ; Among them, y i and y i+1 These are the i-th and (i+1)-th values ​​in the historical clock difference sequence, respectively. The length of the historical clock difference sequence. The time decay weight for the i-th time series is... , This is the attenuation factor.

[0009] Preferably, the gating path performs convolution processing on the historical clock difference sequence, subtracts the weighted value of the background noise intensity, and then activates to obtain the gating coefficient G.

[0010] Preferably, the clock error prediction model is a reinforcement learning model, where state s includes the historical clock error sequence and its data features, and action a is the next predicted clock error value; The model training process includes the following steps: Two identical clock bias prediction models are constructed, denoted as the online network and the target network, respectively. A fixed-size experience replay buffer is constructed to store the experience samples {s(t), a(t), r(t), s(t+1)} generated by the online network, where t is the time step, a(t) is the action generated by the online network for state s(t), r(t) is the reward function, and s(t+1) is the state at the next time step after executing action a(t) in state s(t). After the experience replay buffer is filled, training samples are extracted from the experience replay buffer. The model loss is calculated by combining the Q-values ​​of the online network and the target network for the training samples and used to update the online network. After the online network is updated a set number of times, the parameters of the online network are copied to the target network. After the loop iterates until convergence, the online network is extracted as the trained clock error prediction model.

[0011] Preferably, the dual-stream evaluation module maps the data features of multi-scale temporal features and historical clock error sequences to obtain the state value function V(s) and the action advantage function A(s,a) for each action a; the action advantage function A(s,a) is the advantage of action a predicted by the model in state s; the aggregation layer is used to decentralize the action advantage function A(s,a) and add it to the state value function V(s) to generate the final Q value function Q(s,a) for each action a; then the action a corresponding to the largest Q(s,a) is used as the clock error prediction result.

[0012] Preferably, the formula for calculating the reward function is: ; in, It is the hyperbolic tangent function. This is the gain coefficient; and All are parameter weights; The immediate basic reward is obtained by weighted summation of multiple model performance evaluation metrics; It is a measure of the frequency stability of predicted values; As a constraint term in the physical model, clock error predictions are used in conjunction with the physical fitting model. deviation, This is a quadratic polynomial trend benchmark obtained by fitting historical clock difference sequences; The trajectory divergence suppression term is obtained by accumulating the deviation between the predicted and actual clock error values ​​over K consecutive steps.

[0013] Preferred, The Allan variance surrogate index is calculated based on the predicted and actual clock error values ​​of the training samples. Physical model constraints The calculation formula is as follows: ; Forward steps, and These are the predicted and actual values ​​at time step t+k, respectively. This is the time decay factor.

[0014] Preferably, during training, the loss function is calculated using the following formula: Y(t) = r(t) + γ × Q target (s(t+1),argmax(Q current (s(t+1),a′))); LOSS=∑ t [Y(t)-Q current (S(t),a(t))] 2 ; Where LOSS is the loss function, Y(t) is the objective function, γ is the discount factor, and Q is the target function. current (s(t+1),a′) represents inputting the state s(t+1) into the online network Q. current We obtain the action distribution and the Q-value corresponding to each action, where a′ is any action in the action distribution; argmax(Q current (s(t+1),a′)) represents Q current The action corresponding to the maximum value in (s(t+1),a′); Q target (s(t+1),argmax(Q current (s(t+1),a′))) represents the output action distribution of argmax(Q) after s(t+1) is input into the target network. current The target Q value corresponding to (s(t+1),a′)); Q current (S(t), a(t)) represents the online network Q current The Q value of the action a(t) generated when the input state is s(t).

[0015] The present invention proposes a satellite clock bias timing prediction system based on deep reinforcement learning, comprising a memory and a processor. The memory stores a computer program, and the processor is connected to the memory. The processor is used to execute the computer program to realize the satellite clock bias timing prediction method based on deep reinforcement learning.

[0016] The advantages of this invention are: 1. The present invention proposes a satellite clock bias time series prediction model based on deep reinforcement learning, which uses a noise-gated convolution module to extract features and can perceive the background noise intensity of satellite clock bias data in real time. By dynamically adjusting the gating threshold, it can filter out false features in a noisy environment and retain small clock bias features in a low-noise environment, thereby improving the noise resistance in non-stationary environments.

[0017] 2. This invention employs a multi-scale feature extraction layer, comprising three parallel branches: short-term, medium-term, and long-term. Each branch consists of a noise-gated convolutional module, configured with different dilation rates to extract features from adjacent time points, periodicity, and long-term time series. Thus, the feature extraction layer extracts multi-scale features, providing richer feature data for subsequent model inference.

[0018] 3. This invention constructs a reward mechanism that conforms to the characteristics of satellite clock bias, integrating single-step accuracy with long-term physical constraints. The trajectory divergence suppression term enables the agent to calculate the error for the next K steps during the training phase, avoiding short-sighted decision-making; the physical model constraint term utilizes the quadratic polynomial properties of atomic clocks to limit the degrees of freedom of the predicted trajectory, preventing non-physical divergence. The multinomial synthesis of the reward function effectively alleviates the problem of accumulated error in long-term time-series prediction in satellite clock bias prediction.

[0019] 4. The reward function of this invention also introduces a frequency stability gating term, forcing the model to consider the stability of frequency changes while pursuing the minimum prediction error. The predicted trajectory generated in this way is not only numerically accurate, but also avoids violent fluctuations that do not conform to physical common sense, making the prediction results highly close to the behavior of real atomic clocks and more in line with physical laws.

[0020] 5. This invention constructs a dedicated training framework based on reinforcement learning and trajectory experience playback, and designs a reward function that conforms to the characteristics of satellite clock bias. Together, they construct a stable and efficient reinforcement learning framework for clock bias time series prediction, providing a new technical approach for satellite clock bias prediction. Attached Figure Description

[0021] Figure 1 This is a block diagram of the clock error prediction model proposed in this invention; Figure 2 This is a structural diagram of the noise-gated convolution module proposed in this invention; Figure 3 This is a flowchart of a satellite clock bias timing prediction method based on deep reinforcement learning proposed in this invention; Figure 4 This is a comparison chart of the actual clock error and the predicted clock error in an example. Detailed Implementation

[0022] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0023] like Figure 1 As shown in the figure, this embodiment proposes a satellite clock bias timing prediction model based on deep reinforcement learning, referred to as the clock bias prediction model, which is based on the historical clock bias sequence x={y i Predict the next clock difference y (1≤i≤N) N+1 N is the length of the historical clock difference sequence.

[0024] The clock error prediction model includes: a feature extraction layer, an attention enhancement layer, a two-stream evaluation module, and an aggregation layer.

[0025] The feature extraction layer contains three branches, each of which uses... Figure 2 The noise-gated convolutional module shown includes a data path, a gating path, and a noise sensing path.

[0026] The noise sensing pathway is used to extract local frequency jitter of the historical clock difference sequence x and perform weighted statistics to characterize the current background noise intensity. The specific calculation formula is as follows: ; Among them, y i and y i+1 These are the i-th and (i+1)-th values ​​in the historical clock difference sequence, respectively. The length of the historical clock difference sequence. The time decay weight for the i-th time series is... , This is the attenuation factor.

[0027] Gated path combined with background noise intensity Generate adaptive gating coefficients The calculation formula is: ; Where x is the historical clock difference sequence, The convolutional output features are obtained by processing x through convolution. is the noise sensitivity coefficient; Sigmoid is the activation function.

[0028] The data path is used to perform dilated convolution processing on the historical clock difference sequence x to obtain convolutional features, and then the convolutional features are combined with adaptive gating coefficients. Multiplying these together yields the effective feature Conv(x)×G.

[0029] The data paths in the three branches are configured with different dilation rates for dilated convolution processing, enabling the three branches to extract features from adjacent time points, periodic features, and long-term time series features respectively. This allows the feature extraction layer to focus on the multi-directional features of the historical clock difference sequence, providing richer feature data for subsequent processing.

[0030] The attention enhancement layer generates an attention weight vector based on the concatenation of effective features from the three branches of the feature extraction layer. Then, it uses this attention weight vector to adaptively weight and fuse the effective features from the three branches, generating enhanced multi-scale temporal features. Specifically, the attention enhancement layer includes a sequentially connected concatenation unit, a fully connected layer, and a weighted fusion unit. The input of the concatenation unit is connected to the output of the feature extraction layer, used to concatenate the three effective features. The fully connected layer calculates the attention weight vector based on the concatenation result output by the concatenation unit, specifically including the weights of each branch in the feature extraction layer. The weighted fusion unit is also connected to the output of the feature extraction layer, used to adaptively weight and fuse the three effective features output by the feature extraction layer using the attention weight vector.

[0031] The dual-stream evaluation module first concatenates the multi-scale temporal features output from the attention enhancement layer with the data features of the historical clock difference sequence. Then, it maps these features through a fully connected layer to obtain the state value function V(s) and the action advantage function A(s,a) for each action a. The action advantage function A(s,a) represents the advantage of action a predicted by the model in state s. Action a is the predicted value of the next clock difference, i.e., the clock difference y. N+1 The prediction is as follows: s represents the state space, which consists of historical clock error sequences and their data features, including temporal variation features, statistical features, frequency domain features, autocorrelation features, and higher-order statistical features. The dual-stream evaluation module generates and outputs multiple possible actions, and calculates the action advantage function A(s,a) for each action.

[0032] The aggregation layer is used to decentralize the action advantage function A(s,a) and add it to the state value function V(s) to generate the final Q-value function Q(s,a) for each action a; then, the action a corresponding to the maximum Q(s,a) is used as y. N+1 Output.

[0033] ; Where A represents the set of all possible actions, |A| represents the total number of actions in A, and a′ represents an action in A; the action advantage function A(s,a′) is the advantage of action a′ predicted by the model in state s.

[0034] In this embodiment, reinforcement learning is used to train the clock error prediction model. During the training process, the state space s includes the historical clock error sequence x={y}i |1≤i≤N} and its data characteristics, action a is the next clock difference y N+1 The reward function consists of four parts: immediate basic reward, frequency stability gating term, physical model constraint term, and trajectory divergence suppression term. The formula is as follows: ; in, , , and These are the immediate basic reward, the frequency stability measure of the predicted value, the physical model constraint term, and the trajectory divergence suppression term, respectively. It is the hyperbolic tangent function. The gain coefficient is used; a nonlinear gain gate is constructed using the hyperbolic tangent function, and the base reward can only be amplified when the frequency stability of the predicted value conforms to the physical characteristics of an atomic clock. and All of these are parameter weights, and can all be set to 0.5.

[0035] Instant basic rewards Specifically, it can be implemented as a weighted sum of several commonly used reward indicators (such as accuracy, trend consistency, smoothness, progress reward, and continuous progress reward).

[0036] The Allan variance surrogate index can be used to measure the frequency stability of predicted values. The calculation formula is as follows: ; in, It is a second-order difference operator. Let x be the standard deviation of the second difference of the historical clock sequence. Let y be the predicted value of the clock bias, and y be the clock bias. The corresponding actual value.

[0037] in For physical model constraints, defined as predicted values Fitting model to physics Deviation: ; It is a quadratic polynomial trend benchmark obtained by fitting historical clock difference sequences. It uses quadratic polynomial fitting of historical trends to penalize predictions that deviate from the physical evolution trajectory. Trajectory divergence suppression term Defined as: ; Forward steps, and These are the predicted and actual values ​​at time step t+k, respectively. This is the time decay factor. By rehearsing the prediction error for the next K steps, the generation of cumulative error is suppressed at its source.

[0038] Reference Figure 3 The training process for the clock error prediction model is as follows: S1. Construct two identical clock error prediction models, denoted as online network Q. current and target network Q target ; Construct and initialize a fixed-size experience playback buffer; S2, make the online network Q current Based on the current state s(t), an action a(t) is generated, and the agent (i.e., the algorithm model corresponding to the satellite) executes the action a(t). Experience samples {s(t), a(t), r(t), s(t+1)} are collected and stored in the experience replay buffer; the initial value of t is t0. s(t)={x(t);X(t)}; x(t)={y(t-N+1),y(t-N+2),…,y(t)}; s(t+1)={x(t+1);X(t+1)};x(t)={y(t-N+2),y(t-N+3),……,y(t), (t+1)}; Where y(t-N+1), y(t-N+2), y(t-N+3), ..., y(t) represent the clock difference observations at times t-N+1, t-N+2, t-N+3 and t, respectively. Let be the predicted clock difference value at time t+1, i.e., action a(t).

[0039] r(t) is the reward function at time t.

[0040] S3. Extract training samples from the experience replay buffer and calculate the objective function Y corresponding to each training sample; Y(t) = r(t) + γ × Q target (s(t+1),argmax(Q current (s(t+1),a′))) Where γ is the discount factor, and in this example, it is set to 0.95; Q current (s(t+1),a′) represents inputting the state s(t+1) into the online network Q. current We obtain the action distribution and the Q-value corresponding to each action, where a′ is any action in the action distribution; argmax(Q current (s(t+1),a′)) represents Q current The action corresponding to the maximum value in (s(t+1),a′); Qtarget (s(t+1),argmax(Q current (s(t+1),a′))) represents the output action distribution of argmax(Q) after s(t+1) is input into the target network. current The target Q value corresponding to (s(t+1),a′)).

[0041] It is worth noting that if time step t is the latest time step in the experience replay buffer, then Y(t) = r(t). To facilitate confirmation that each experience sample has a next time step, in practice, the experience sample can be recorded as a quintuple with the last dimension being the done value (end flag) for boundary handling; that is, if done = 0, then the normal objective function Y(t) = r(t) + γ × Q is used. target (s(t+1),argmax(Q current (s(t+1),a′)));If done=1, then Y(t)=r(t).

[0042] S4. Calculate the model loss on the training samples: LOSS = ∑ t [Y(t)-Q current (S(t),a(t))] 2 The online network Q is updated using the loss function LOSS gradient descent. current ; S5. Determine if the training has converged; in specific implementation, the convergence condition can be set as: the number of times the target network is updated reaches a set value; Yes, then extract the online network Q. current As a clock error prediction model.

[0043] No, proceed to step S6; S6. Determine the distance to the nearest target network Q. target After the update, online network Q current Has the update count reached the set value? No, then return to step S3; If yes, then copy the parameters of the online network to the target network, and then return to step S3.

[0044] It is worth noting that step S3 is executed for the first time after the experience replay buffer is full.

[0045] The following specific embodiments illustrate and verify the above-mentioned satellite clock bias timing prediction method based on deep reinforcement learning.

[0046] In the specific implementation process, the original multi-source satellite observation data from January 19, 2025 to January 25, 2025 was first downloaded from the International GNSS Service Data Center. The specific data included data version, sampling interval, analysis center, reference frame, antenna phase center correction model, and precise clock error observations of multiple systems and satellites such as GPS, GLONASS, Galileo, and BDS.

[0047] Data preprocessing: First, the type and number of the target satellite are selected. This invention selects GPS satellite G02 as the research object. Historical clock bias data for satellite G02 are extracted from the original multi-source satellite observation data. Missing values ​​are checked first, and cubic spline interpolation is used to fill in the missing values. Then, the phase sequence is subjected to first-order differencing, which converts the non-stationary clock bias phase into a clock bias frequency sequence. The frequency sequence is then Z-score standardized, mapping it to a standard normal distribution interval with a mean of 0 and a standard deviation of 1. Finally, a structured time-series dataset containing timestamps, satellite numbers, and standardized frequency difference data is constructed.

[0048] Building a deep reinforcement learning environment: Define state vector This is used to characterize the observation history and sequence features of the current time t. In this embodiment, the history window length N=30. It is a 72-dimensional vector, composed of the following six features: historical clock difference sequence [ , ,..., The data includes: the most recent 30 standardized frequency difference values; time series variation characteristics (12 dimensions), including first-order difference, multi-step difference, short-term / medium-term average rate of change, volatility, etc.; statistical characteristics (10 dimensions, such as mean, standard deviation, median, extreme values, skewness, kurtosis, etc.); frequency domain characteristics (8 dimensions, taking the amplitude of the first 8 major frequency components by performing a fast Fourier transform on the historical 30-point series); autocorrelation characteristics (8 dimensions, calculating the autocorrelation coefficient of the series at several specific lag orders); and higher-order statistical characteristics (4 dimensions, mainly skewness and kurtosis, used to describe the distribution pattern of the series).

[0049] Define an action space A containing 51 discrete actions, where each action... For a standardized prediction residual value that is uniformly distributed in the interval [-2.5, 2.5], after the agent outputs the action index, it is mapped back to the nanosecond-level clock difference prediction value through an anti-standardization operation.

[0050] Reward Function Design: To address accumulated errors and enhance physical interpretability, this embodiment constructs a reward mechanism that conforms to the physical characteristics of satellite clock bias. This reward function... It consists of the following four parts: ; Instant basic rewards It is a weighted sum of accuracy, trend consistency, smoothness, progress reward, and continuous progress reward, where the weights of accuracy, trend consistency, smoothness, progress reward, and continuous progress reward are designed to be 1.2, 0.8, 0.6, 0.9, and 0.5, respectively, to take into account the physical evolution trend and smoothness characteristics of the clock error sequence while prioritizing numerical prediction accuracy; the gain coefficient of the frequency stability gating term is set to... =2.0, when the second-order difference standard deviation of the predicted sequence approaches the true value, the basic reward can be amplified by up to 2 times (1+tanh(2)). 1.96); the coefficients of the physical model constraint terms are set to =0.5, The curve is a quadratic polynomial fitted based on data within the window; the coefficient of the trajectory divergence suppression term is set to... =0.5, look-ahead steps K=5, time decay factor =0.8.

[0051] The above parameters are a set of preferred embodiment parameters determined by optimization on the validation set with the objective of minimizing prediction error, based on the physical characteristics of clock bias, preliminary experiments, and grid search method.

[0052] Building deep reinforcement learning agents: like Figure 1 As shown, the core of the agent is a deep Q-network that integrates multi-scale noise-gated convolutions, where the noise-gated convolution module is as follows: Figure 2 As shown. The agent inputs a 72-dimensional state vector. The time series is separated into a 30-dimensional historical clock difference sequence and a 42-dimensional statistical feature. The temporal part (i.e., the historical clock difference sequence) is input into three parallel noise-gated convolutional modules.

[0053] Data Pathway: This path is responsible for extracting physical features. Historical clock difference sequences are passed through dilated convolutional layers to output convolutional features. To capture clock bias characteristics at different scales, the three branches are configured with differentiated dilation rates: short-term branch: dilation rate d=1, kernel size k=3, focusing on capturing small jitters between adjacent time steps; medium-term branch: dilation rate d=2, kernel size k=3, focusing on capturing local periodic fluctuations; long-term branch: dilation rate d=4, kernel size k=3, with a receptive field covering the entire 30-step window, capturing long-term drift trends.

[0054] Noise perception path: This branch directly operates on the input historical clock difference sequence to calculate background noise intensity in real time. An exponential decay time-weighted mechanism is introduced, first calculating the first difference of the input sequence to characterize local frequency jitter, and then calculating the weighted statistics. The time decay factor is set. =0.95, which gives greater weight to jitter that is closer to the current time.

[0055] Gating Path: This path combines the results from the two paths above to generate the adaptive gating coefficient G. The noise sensitivity coefficient is initialized to... =1.5. This formula uses the calculated noise intensity. By dynamically adjusting the activation threshold of the Sigmoid function, it can adaptively adjust according to the dynamic changes in the intensity of environmental noise, achieving effective noise reduction in noisy environments and preserving subtle physical features in low-noise environments.

[0056] Features of data path output coefficient of the gated path output Element-wise multiplication is performed to output the effective features after noise removal, which are then fed into the next layer of the network.

[0057] Attention Enhancement: The effective features output from the three branches are concatenated and input into a fully connected attention network. Channel attention weight vectors w corresponding to the three branches are calculated. These weights represent the importance of each scale feature's contribution to clock error prediction at the current time step. This weight vector is then used to weight the features of the three branches channel-by-channel, generating an enhanced feature vector. .

[0058] Processing other features: The 42-dimensional non-temporal statistical features are mapped through a fully connected sub-network, which contains one hidden layer and one output layer, to a 64-dimensional high-dimensional feature vector. The above-mentioned enhanced feature vectors Perform a flattening operation to convert the time series matrix into a high-dimensional feature vector, and then combine the flattened enhanced feature vector with... The data are concatenated and nonlinearly mapped through a fusion layer to obtain the final high-level feature vector z.

[0059] The high-level feature vector z is input into two parallel fully connected layers. The first is the value stream layer, which outputs a one-dimensional scalar V(s), representing the intrinsic value of state s. The second is the dominance stream layer, which outputs a 51-dimensional vector A(s, a), representing the dominance of each action relative to the average action. The final Q-value is calculated using the aggregation formula: Q(s, a) = V(s) + [A(s, a) -mean(A(s, a))], outputting a 51-dimensional Q-value vector. The agent selects the action with the highest Q-value as the prediction.

[0060] Training process: This paper employs the Double DQN algorithm (an improved reinforcement learning approach) to address the systematic overestimation of Q-values ​​in standard DQN (reinforcement learning). In the dual-network architecture, the online network is used for action selection and policy optimization, while the target network is dedicated to value evaluation. During each network update, the online network calculates the Q-values ​​of all possible actions for the next state, and the optimal action is selected using the argmax operation. In the value evaluation phase, the target network evaluates the Q-value of the selected optimal action and calculates the target Q-value. By minimizing the mean squared error between the online network output and the target Q-value, the network parameters are updated. This decoupling mechanism between action selection and value evaluation effectively suppresses the systematic overestimation of Q-values, improving training stability and prediction accuracy.

[0061] Simultaneously, an experience replay buffer with a capacity of 10000 is constructed to store the quintuple (s, a, ...). , s', done), where It includes pre-calculated look-ahead rewards. Each time the network is updated, 128 samples are randomly drawn from the experience pool for Double DQN gradient updates.

[0062] Throughout the training process, the system continuously monitors training and adjusts the exploration rate. Training progress and key metrics are output every 20 rounds. When the training loss falls below 0.1, the exploration rate decay is accelerated (ε is updated to 0.95ε), but the minimum exploration rate is capped at 0.05 to ensure the agent maintains a certain level of exploration capability. An early stopping mechanism is also introduced, monitoring the training rewards of the last 20 rounds. If there is no significant improvement, training is terminated early to prevent overfitting.

[0063] To evaluate the effectiveness of the proposed method, a systematic comparative experiment was conducted. The Long Short-Term Memory (LSTM) network, widely used in time series forecasting, was selected as the primary benchmark model. All experiments used the same historical satellite clock bias training dataset, the same test set segmentation, and the same input data preprocessing procedures to ensure fairness in the comparison.

[0064] The following evaluation indicators are used in this invention: Mean Absolute Error (MAE): Measures the degree of average absolute deviation between the predicted value and the actual value. The smaller the value, the higher the prediction accuracy. Root Mean Square Error (RMSE): The square root of the square of the prediction error. It is more sensitive to large errors and can effectively reflect the stability of the prediction.

[0065] Based on the above metrics, the performance comparison between the satellite clock error prediction method based on deep reinforcement learning proposed in this invention and the LSTM benchmark model is shown in Table 1 below.

[0066] Table 1: Comparison of Predictive Performance of Each Model

[0067] Experimental results show that the satellite clock error prediction method based on deep reinforcement learning proposed in this invention outperforms the standard LSTM benchmark model in all key indicators. In the core indicator MAE, the model of this invention reduces the prediction error from 2.76 ns for the standard LSTM model to 2.59 ns, demonstrating a significant improvement in accuracy. The prediction accuracy of 2.59 ns achieved by the model of this invention fully meets and exceeds the standard forecast service accuracy provided by IGS (International GNSS Service), possessing practical engineering value. The prediction results of the model of this invention are as follows: Figure 4 As shown, it exhibits a high degree of consistency with the actual clock difference.

[0068] Meanwhile, ablation experiments were conducted based on the innovations of this invention, with experiments performed on a simple weighted reward function + ordinary gating, a simple weighted reward function + noise-aware gating, and an innovative reward function + noise-aware gating. The models were named Model_A, Model_B, and Model_C, respectively. The comparative results are shown in Table 2.

[0069] Table 2: Comparison of Ablation Experiments

[0070] Experimental results show that the complete model proposed in this invention (Model_C) significantly outperforms the baseline model (Model_A, 6.87 ns) and the model with only noise gating (Model_B, 5.41 ns) with a MAE of 2.59 ns. The performance improvement of Model_B compared to Model_A directly confirms that the noise-aware gating module has independent and effective active denoising capabilities in the feature extraction stage; while the further improvement of Model_C strongly demonstrates the role of the physically enhanced reward function.

[0071] In summary, this embodiment verifies the advancement of the proposed architecture and demonstrates the effectiveness of deep reinforcement learning-based methods in satellite clock error prediction. It provides a novel solution for high-precision time series prediction in complex space environments, showcasing broad application prospects.

[0072] Of course, those skilled in the art will recognize that the present invention is not limited to the details of the exemplary embodiments described above, but also includes the same or similar structures that can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention. No reference numerals in the claims should be construed as limiting the scope of the claims.

[0073] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0074] The technologies, shapes, and structures not described in detail in this invention are all known technologies.

Claims

1. A satellite clock bias timing prediction method based on deep reinforcement learning, characterized in that, The clock difference prediction model is first trained on a known clock difference sequence, which predicts the next step clock difference based on a historical clock difference sequence with a length of N; then the observed clock difference at the last N time steps is input into the clock difference prediction model as the historical clock difference sequence to obtain the next step clock difference prediction value; The model structure includes sequentially connected feature extraction layers, attention enhancement layers, double-flow evaluation modules and aggregation layers; The feature extraction layers are used to extract effective features of the historical clock difference sequence, and the attention enhancement layers are used to process the effective features to obtain attention-enhanced multi-scale time sequence features; the double-flow evaluation modules are used to process the multi-scale time sequence features and the data features of the historical clock difference sequence to generate different next step clock difference prediction values; the aggregation layers are used to evaluate the Q value function Q(s,a) of each next step clock difference prediction value generated by the double-flow evaluation modules; then the next step clock difference prediction value corresponding to the maximum Q(s,a) is output as the clock difference prediction result; The feature extraction layers adopt noise-gated convolution modules; the noise-gated convolution module includes a data path, a gating path and a noise perception path; The noise perception channel is used to extract the local frequency jitter of the historical clock difference sequence x and to obtain the background noise intensity by weighted statistics ; Gating path combined with background noise intensity Generating adaptive gating coefficients The data path is used to perform dilated convolution processing on the historical clock difference sequence x to obtain convolution features, and then the convolution features are multiplied by the adaptive gating coefficients to obtain effective features as the output of the noise gating convolution module.

2. The deep reinforcement learning based satellite clock error timing prediction method of claim 1, wherein, The feature extraction layers include three branches, and each branch adopts a noise-gated convolution module; the data paths in the three branches are configured with different expansion rates for expansion convolution processing, and the effective data of the three branches constitute the output of the feature extraction layers; The attention enhancement layers generate an attention weight vector based on the spliced results of the effective features output by the three branches of the feature extraction layers, and then perform adaptive weighted fusion on the effective features output by the three branches using the attention weight vector to generate enhanced multi-scale time sequence features. 3.The deep reinforcement learning based satellite clock error timing prediction method of claim 1, wherein, The calculation formula of the noise perception path is: where y i and y i+1 are the i-th and i+1-th values of the historical clock difference sequence, respectively, is the length of the historical clock difference sequence, is the time decay weight for the i-th time instance, , is the decay factor.

4. The deep reinforcement learning based satellite clock error timing prediction method of claim 1, wherein, The gating path performs convolution processing on the historical clock difference sequence, and then subtracts the weighted value of the background noise intensity to obtain the gating coefficient G.

5. The deep reinforcement learning based satellite clock error timing prediction method of claim 1, wherein, The clock difference prediction model is a reinforcement learning model, and the state s includes the historical clock difference sequence and its data features, and the action a is the next step clock difference prediction value; The model training process includes the following steps: Two clock difference prediction models that are completely the same are constructed and are denoted as an online network and a target network; a fixed-size experience replay buffer is constructed to store the experience samples {s(t),a(t),r(t),s(t+1)} generated by the online network, t is a time step, a(t) is the action generated by the online network for the state s(t), r(t) is a reward function, and s(t+1) is the next time step state after the action a(t) is executed on the state s(t); After the experience replay buffer is filled, training samples are extracted from the experience replay buffer, and the model loss is calculated by combining the Q values calculated by the online network and the target network for the training samples to update the online network; the online network is updated for a set number of times, and then the online network parameters are copied to the target network; the loop is iterated until convergence, and the online network is extracted as the trained clock difference prediction model.

6. The deep reinforcement learning based satellite clock error timing prediction method of claim 5, wherein, The double-flow evaluation module maps the data features of the multi-scale time sequence features and the historical clock difference sequence to obtain a state value function V(s) and an action advantage function A(s, a) of each action a; the action advantage function A(s, a) is an advantage of the action a predicted by the model for the state s; the aggregation layer is used for adding the action advantage function A(s, a) after decentralized processing to the state value function V(s) to generate a final Q value function Q(s, a) of each action a; and then the action a corresponding to the maximum Q(s, a) is taken as a clock difference prediction result.

7. The deep reinforcement learning based satellite clock error timing prediction method of claim 5, wherein, The calculation formula of the reward function is: wherein, is a hyperbolic tangent function, is a gain coefficient; and are parameter weights; For instant-based rewards, a weighted sum of multiple model performance evaluation indicators is obtained; a frequency stability measure for the predicted value; As a physical model constraint term, the deviation of the clock error prediction value from a physical fitting model is used As a physical model constraint term, the deviation of the clock error prediction value from a physical fitting model is used​ For the trajectory divergence suppression term, the deviation between the clock difference prediction and the true value over the last K steps is accumulated.

8. The deep reinforcement learning based satellite clock error timing prediction method of claim 7, wherein, The Allan variance proxy indicator is obtained based on the clock error prediction value and the true value of the training sample. Physical model constraint term The calculation formula is as follows: is the forward-looking step count, and are the predicted and true values at time step t+k, respectively; is a time decay factor.

9. The deep reinforcement learning based satellite clock error timing prediction method according to any one of claims 1-8, wherein, During the training process, the loss function calculation formula is: Y(t) = r(t) + γ x Q target (s(t+1), argmax(Q current (s(t+1), a′))) LOSS =∑ t [Y(t)-Q current (S(t),a(t))] 2 where LOSS is a loss function, Y(t) is a target function, γ is a discount factor, Q current (s(t+1),a') represents inputting state s(t+1) into the online network Q current to obtain an action distribution and a Q value corresponding to each action, a' is any action in the action distribution; argmax(Q current (s(t+1),a')) represents the action corresponding to the maximum value in Q current (s(t+1),a'); Q target (s(t+1),a') represents the target Q value corresponding to argmax(Q current (s(t+1),a') in the action distribution output by the target network after inputting s(t+1); Q current (s(t+1),a') represents the target Q value corresponding to argmax(Q current (s(t+1),a') in the action distribution output by the target network after inputting s(t+1); Q current (s(t),a(t)) represents the Q value of action a(t) generated when the input state is s(t). 10.A deep reinforcement learning based satellite clock error timing prediction system, characterized in that, The method comprises a memory and a processor, the memory stores a computer program, the processor is connected to the memory, and the processor is used for executing the computer program to realize the satellite clock difference time series prediction method based on deep reinforcement learning according to any one of claims 1-9. The method comprises a memory and a processor, the memory stores a computer program, the processor is connected to the memory, and the processor is used for executing the computer program to realize the satellite clock difference time series prediction method based on deep reinforcement learning according to any one of claims 1-9.

Images