A self-attention-based satellite-borne interference source Doppler observation strategy optimization method

Abstract

The application discloses a kind of based on self-attention satellite interference source Doppler observation strategy optimization method, belong to satellite task scheduling and information perception control field, applicable to the strategy optimization of satellite payload observation path and parameter control in non-cooperative signal positioning.The method models observation task as Markov decision process, trains using improved TD3 reinforcement learning algorithm, suppresses Q value overestimation by double Critic network structure and delay update;Multi-head self-attention mechanism is introduced in Actor network and batch normalization module is added to improve training stability and convergence speed;After training, the strategy is used to dynamically control the observation direction and parameters of satellite payload, and based on the received signal short-time Fourier transform, the spectral principal component is extracted, the Doppler shift is estimated combined with the signal-to-noise ratio, and a reward mechanism that integrates measurement accuracy and coverage efficiency is constructed to improve observation accuracy, convergence performance and adaptability to complex environments.

Description

Technical Field

[0001] This invention relates to the field of spaceborne mission scheduling and information sensing control, and in particular to an optimization method for spaceborne Doppler observation strategy based on self-attention. Background Technology

[0002] In the field of spaceborne wireless communication and non-cooperative signal localization, low-Earth orbit (LEO) satellite platforms have significant advantages in detecting and observing ground-based interference sources due to their flexible orbital coverage capabilities. Localizing non-cooperative interference sources based on Doppler frequency shift information is a key technology in wireless communication interference monitoring and localization applications. In practical applications, the reception quality and frequency observation accuracy of interference source signals are affected by various factors, including the attitude control of the spaceborne payload, the receiving beam pointing, and mission scheduling strategies. To improve spectrum utilization efficiency and interference source observation accuracy, the observation strategy of the spaceborne platform needs to be optimized.

[0003] In airborne IRS (intelligent reflecting surface)-assisted wireless communication networks, the joint design of transmit beamforming and phase shift optimization is a key challenge. In recent years, researchers have proposed various optimization methods. Some researchers have employed block coordinate descent (BCD) and Lagrange dual transformation to jointly optimize base station beamforming and IRS phase shift, aiming to maximize the weighted rate sum under non-ideal oscillations of UAVs. Furthermore, some scholars have proposed using generalized Benders decomposition (GBD) and continuous convex approximation methods to solve the joint optimization problem of beamforming and deployment under multi-IRS collaboration. For resource-constrained situations, researchers have proposed deep reinforcement learning (DRL) strategies, utilizing decision trees or piecewise action design to optimize IRS control and power allocation. For example, in a multi-input single-output (MISO) system integrating an unmanned aerial vehicle (UAV) IRS, the deep deterministic policy gradient (DDPG) algorithm was used to jointly optimize the IRS trajectory and reflection coefficient. Other researchers designed the Piecewise-DRL framework, which avoids getting trapped in local optima by introducing a piecewise action space, thereby improving the joint beamforming effect. These methods provide feasible paths for dealing with high-dimensional states and action spaces.

[0004] Current common methods often rely on heuristic rules, traversal optimization, or simplified models, making it difficult to adapt to policy planning needs in dynamic environments and complex state spaces. Deep reinforcement learning, on the other hand, has potential advantages in complex high-dimensional spaces because it can autonomously learn optimal policies through interaction with the environment. Especially in scenarios with continuous state transitions and differentiable action spaces, policy gradient algorithms such as TD3 (Twin Delayed Deep Deterministic Policy Gradient) have demonstrated good performance.

[0005] However, when faced with the high-dimensional state and action space issues in spaceborne missions, the DRL algorithm often suffers from slow convergence speed and poor policy generalization. To address this problem, this invention proposes a self-attention-based Doppler observation policy optimization method for spaceborne interference sources. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings of the existing technologies by proposing a self-attention-based optimization method for Doppler observation strategies of spaceborne interference sources. By introducing a self-attention mechanism and batch normalization technology, and improving the deep reinforcement learning TD3 algorithm, this method achieves efficient optimization of payload observation strategies in complex state-action spaces, thereby enhancing the observation efficiency and data utilization of spaceborne platforms in non-cooperative target observation missions.

[0007] The technical solution for achieving the objective of this invention is as follows:

[0008] An optimization method for spaceborne interference source Doppler observation strategy based on self-attention includes the following steps:

[0009] Step S1: Establish a Markov decision process model for the spaceborne observation system: Model the observation tasks of the spaceborne payload at different times as a sequential decision problem from state to action to reward;

[0010] Step S2: Adopt the improved TD3 algorithm: Introduce a dual Critic network structure to solve the problem of Q-value overestimation, and enhance training stability by delaying the update of the Actor policy network;

[0011] Step S3: Introduce a self-attention mechanism: Design a multi-head attention layer in the Actor network to model high-order dependencies between state features and enhance the policy network's ability to model complex observation scenarios.

[0012] Step S4: Apply batch normalization: Add batch normalization layers to the Actor and Critic networks to improve the stability of the input distribution during training and accelerate the convergence speed.

[0013] Step S5: The strategy output is used to control the observation behavior of the spaceborne payload in real time, optimize the receiving path of interference sources, and improve the quality of Doppler measurement data.

[0014] Furthermore, step S1 includes the following specific processes: collecting satellite orbital parameters at different times; observing environmental state variables related to the mission; and constructing the state space. The state space includes orbital parameters, historical state information, attitude angles, and receiving parameters, which are then normalized. The observation tasks of the spaceborne payload at different times are modeled as a sequential decision problem from state to action to reward, expressed as:

[0015]

[0016] in, and These are the values ​​of the i-th state variable before and after normalization, respectively. , Let be the mean and standard deviation of the i-th state variable, respectively.

[0017] Furthermore, in step S1, a reward function R needs to be constructed, which weights and combines the signal-to-noise ratio of the received interference signal, the Doppler frequency shift measurement accuracy, and the scheduling coverage efficiency to obtain a comprehensive evaluation index. The reward function R is expressed as:

[0018]

[0019] in , , These are weighting factors used to adjust the impact of each evaluation item on the total reward. The degree of contribution, It can be calculated from the peak steepness of the spectrum. Used to measure the quality of received signals. It is used to reflect the efficiency of resource utilization.

[0020] Furthermore, the TD3 algorithm described in step S2 employs a dual-Critic network structure to construct value functions respectively. The smaller of the two values ​​is used as the target value for updating, as shown below:

[0021]

[0022] in, For the target Q value, The value function of the dual-critic network is given by the policy network, which employs a delayed update mechanism to improve the stability of policy learning.

[0023] Furthermore, in step S3, a multi-head self-attention mechanism is introduced into the Actor network to model the state variables. The attention calculation method is expressed as follows:

[0024]

[0025] in, It is a vector of the current state features after transformation, used for proactive correlation matching. It is another set of feature vectors that are matched against the query to generate attention weights. It is the information itself that needs to be weighted and summed. It is the dimension of the key vector, which serves as a scaling factor to prevent the dot product from becoming too large and causing gradient saturation. The output of each head is fed into subsequent network modules through concatenation and linear mapping.

[0026] Furthermore, the batch normalization module mentioned in step S4 is used to normalize the input data of each hidden layer, improving the distribution stability of each layer during training. Its calculation formula is as follows:

[0027]

[0028] in, and These represent the input and output of the batch normalization layer, respectively. and These represent the mean and variance of the current mini-batch data along the feature dimension, respectively. and These are the learnable scaling and offset parameters, respectively. To prevent small constants from being divided by zero.

[0029] Furthermore, the Doppler frequency shift estimation accuracy in the reward function is obtained by extracting the principal components of the spectrum through short-time Fourier transform, as follows:

[0030]

[0031] in, It is the spectrum data received in the discrete-time domain. The window function extracts a segment of the signal at time offset τ, where N is the number of points in the Fourier transform. It is a discrete frequency index, corresponding to the Doppler frequency shift component in the analysis. At time window position τ, frequency The complex spectrum value at the given location, with the main peak position as the frequency shift observation, is expressed by the following formula for calculating the signal-to-noise ratio:

[0032]

[0033] in, It is the received interference signal component. It is the environmental noise component. Represents variance calculation, used to measure the fluctuation power of a signal or noise.

[0034] Furthermore, the state transition function in the MDP model described in step S1 is constructed based on the orbital dynamics model and can be expressed as:

[0035]

[0036] Where r is the satellite's position vector, The gravitational constant of Earth, It is the perturbation acceleration term, representing the disturbance of the basic orbit by various small external forces. It also considers the orbit update caused by the perturbation of Earth's gravity and the changes in attitude control, so as to simulate the dynamic adjustment of the state transition probability.

[0037] Furthermore, during the training process, the four-tuple of state, action to reward, and next state is stored in the experience replay pool, and the parameters of the policy network and value network are updated through mini-batch random sampling.

[0038] Furthermore, in step S2 of the training process, a target network mechanism is used for stability enhancement. The target Actor network parameter update method is expressed as follows:

[0039]

[0040] in This is the soft update coefficient, with a value ranging from 0 to 1. Attached Figure Description

[0041] Figure 1 is a schematic diagram of the network structure of a spaceborne interference observation system;

[0042] Figure 2 is a schematic diagram of the AB-TD3 reinforcement learning structure based on the self-attention mechanism proposed in this invention;

[0043] Figure 3 shows a comparison of the algorithm performance before and after introducing the attention mechanism under different configurations;

[0044] Figure 4 shows the impact of batch normalization mechanism on network training stability.

[0045] Figure 5 shows the evaluation results of strategy generalization performance under different initial interference source distribution conditions. Detailed Implementation

[0046] To make the technical problems, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. The following embodiments are simulated using MATLAB tools, and this application is compared with other classic algorithms to verify the beneficial effects of this invention. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit this application.

[0047] The core idea of ​​this invention is to model the scheduling process of satellite observation interference sources as a Markov decision process (MDP) and to achieve intelligent optimization of the satellite payload observation strategy through a method that integrates self-attention mechanism and improved TD3 reinforcement learning algorithm.

[0048] like Figure 1 The diagram shown is a network structure diagram of a spaceborne interference observation system. The invention will be described in further detail below.

[0049] Step (1) Establish a Markov Decision Process (MDP) model for the spaceborne observation system, including the following specific steps:

[0050] (1a) Collect satellite orbital parameters at different times, including the three-dimensional position of the onboard platform. and speed ;

[0051] (1b) Environmental state variables related to the observation task, including the estimation of the direction of the disturbance source. Signal strength Historical reception success rate wait;

[0052] (1c) Constructing the state space This includes the orbital parameters, historical states, attitude angles, and receiving parameters, which are then normalized and expressed as follows:

[0053]

[0054] in , The first The mean and standard deviation of each state variable.

[0055] (1d) Define the action space ,in Indicates the beam direction control angle (such as azimuth and elevation). This indicates the frequency reception parameter settings. Indicates the allocation of time for task observation;

[0056] (1e) Construct a reward function R, which is a weighted combination based on the signal-to-noise ratio of the received interference signal, the accuracy of Doppler frequency shift measurement, and the scheduling coverage efficiency. The reward function R can be expressed as:

[0057]

[0058] in , , These are weighting factors used to adjust the impact of each evaluation item on the total reward. The degree of contribution, It can be calculated from the peak steepness of the spectrum. To measure the quality of the received signal, It reflects the efficiency of resource utilization (the higher the value, the greater the energy consumption).

[0059] Step (2) employs an improved TD3 algorithm: A dual-Critic network structure is introduced to address the Q-value overestimation problem, and the Actor policy network is updated with a delay to enhance training stability; this includes the following specific steps:

[0060] (2a) Construct a dual Critic network for estimating separately Value, decrease The strategy bias caused by overestimation is represented as:

[0061]

[0062] Employing a minimum update strategy to alleviate The value is overestimated, which is expressed as:

[0063]

[0064] (2b) Setting The network delayed update mechanism updates once every n steps after the Critic update. The strategy to enhance training stability is expressed as:

[0065]

[0066] (2c) Introducing a target policy network and a soft update mechanism to mitigate drastic policy fluctuations, expressed as:

[0067]

[0068] like Figure 3As shown, we compare the spectral efficiency performance (training reward) of the proposed AB-TD3 scheme with the aforementioned baseline methods over a 30,000-step training process, with parameters set to M=8, N=32, and K=8. Simulation results show that the AB-TD3 algorithm achieves the highest total spectral efficiency over the entire time span, increasing from approximately 2 bps / Hz to nearly 15 bps / Hz. The AB-TD3 algorithm follows closely behind, with its spectral efficiency increasing from approximately 2 bps / Hz to nearly 13 bps / Hz. In contrast, due to the lack of batch normalization, the TD3 and DDPG algorithms struggle to converge during training, exhibiting significant fluctuations in spectral efficiency throughout the entire time span.

[0069] Step (3), introduce a self-attention mechanism: Design a multi-head attention layer in the Actor network to model higher-order dependencies between state features and enhance the policy network's ability to model complex observation scenarios; including the following specific steps:

[0070] (3a), in The network input feature encoding layer introduces multi-head attention to model the intrinsic dependencies between state elements, thereby enhancing the feature representation capability of the actor network and accelerating the training process. Specifically, the attention output of each head is:

[0071]

[0072] in, The code representing the current state (query). and These represent different levels of state characteristics (keys and values). This is a scaling factor used to prevent gradient vanishing.

[0073] (3b), obtained through linear mapping The matrix is ​​used to calculate attention weights by scaling the dot product, and is represented as:

[0074]

[0075] (3c) Multi-head attention performs the above process through H parallel attention heads, then concatenates the results, and integrates them through a linear layer, as shown below:

[0076]

[0077] Each attention head is represented as follows:

[0078]

[0079] For trainable weight matrix, This represents a vector concatenation operation. Each attention head captures information from different modalities, and the integrated result outputs an action through a multilayer perceptron (MLP).

[0080] Figure 4 The superiority of the AB-TD3 algorithm (M=8, N=64, K=8) was further verified. Its overall spectral efficiency consistently outperformed other methods across the entire time range and stably converged to nearly 12 bps / Hz. The performance of the B-TD3 algorithm remained relatively stable, with its spectral efficiency eventually approaching 10 bps / Hz. This further demonstrates the effectiveness of our introduced self-attention mechanism. In contrast, the TD3 and DDPG algorithms consistently performed the worst, failing to learn the improved strategy.

[0081] Step (4), applying batch normalization: Adding batch normalization layers to the Actor and Critic networks improves the stability of the input distribution during training and accelerates the convergence speed; including the following specific steps:

[0082] (4a) Due to the complex structures of the actor and cirtic networks, the TD3 algorithm may face internal covariate shift problems during training. Therefore, batch normalization is introduced between the hidden layers of the network, expressed as:

[0083]

[0084] in, and These represent the input and output of the batch normalization layer, respectively. and These represent the mean and variance of the current mini-batch data along the feature dimension, respectively. and These are the learnable scaling and offset parameters, respectively. To prevent small constants from being divided by zero, batch normalization effectively stabilizes the input distribution, thereby improving training efficiency and stability.

[0085] (4b) By standardizing the input distribution, gradient vanishing and gradient explosion are avoided, thus improving the numerical stability of deep networks in the observation strategy modeling process;

[0086] (4c) Combining the ReLU activation function and Dropout technique, a regularization term is added during the training phase to improve generalization ability. The loss function is expressed as:

[0087]

[0088] in express size, For regularization weights.

[0089] Figure 5 The algorithm performance is demonstrated with configurations of M=12, N=32, and K=12. The spectral efficiency of AB-TD3 continuously improves over time, eventually reaching its maximum value. However, the AB-TD3 algorithm exhibits significant volatility during training. This phenomenon is attributed to the increase in the number of base station (BS) antennas and user equipment (UEs), which exacerbates the curse of dimensionality problem in the action space. Nevertheless, the algorithm eventually converges to a value close to 8. The B-TD3 algorithm exhibits relatively stable performance, but it failed to learn an improved action policy within a limited number of training steps.

[0090] Step (5) involves outputting a strategy to control the observation behavior of the spaceborne payload in real time, optimizing the receiving path of interference sources, and improving the quality of Doppler measurement data. This includes the following specific steps:

[0091] (5a) Generate observation policy actions based on the current state input policy network. ;

[0092] (5b) Map the motion to control parameters, including beam angle. Frequency configuration and the duration of observation ;

[0093] (5c) Execute the strategy and collect spectrum data The short-time Fourier transform (STFT) is calculated as follows:

[0094]

[0095] in For window functions; by Extracting the dominant frequency component to obtain Doppler frequency shift estimation ;

[0096] (5d), the signal-to-noise ratio (SNR) is calculated synchronously and expressed as:

[0097]

[0098] (5e) The collected feedback results form a playback buffer pool. This is used for the next round of training updates and is represented as:

[0099]

[0100] in, This represents the soft update learning rate of the target network.

[0101] Figure 2 illustrates the comparison of reward convergence curves between the AB-TD3 algorithm and the traditional TD3 algorithm across multiple training iterations. It is evident that the proposed AB-TD3 framework outperforms the traditional TD3 method in both convergence speed and stability, validating the effectiveness of introducing self-attention mechanisms and batch normalization strategies in policy optimization.

[0102] Figure 3 illustrates the performance comparison of the algorithm before and after introducing the attention mechanism under different configurations, with the Doppler shift estimation error as the evaluation metric. Simulation results show that the self-attention mechanism can significantly improve the policy network's ability to model state features, ultimately improving observation accuracy.

[0103] Figure 4 illustrates the impact of batch normalization on network training stability. Experiments demonstrate that adding a batch normalization module to the Actor and Critic networks significantly alleviates gradient oscillations, improves convergence speed, and enhances the smoothness of training curves.

[0104] Figure 5 shows the evaluation results of the strategy generalization performance under different initial interference source distributions. The strategy proposed in this invention maintains good observation accuracy and robustness under various interference source distributions, indicating that the designed strategy has strong adaptability and generalization ability.

[0105] Based on the description of this invention, those skilled in the art should readily recognize that this invention improves reinforcement learning strategies by introducing attention mechanisms and batch normalization techniques, effectively enhancing the training stability, strategy convergence speed, and observation accuracy of spaceborne platforms in Doppler observation missions. The proposed AB-TD3 framework possesses excellent generalization capabilities and is suitable for non-cooperative signal localization missions in complex dynamic environments, providing a new technical path for optimizing spaceborne interference source observation strategies.

[0106] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. However, these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for optimizing spaceborne Doppler observation strategies based on self-attention, characterized in that, Includes the following steps: Step S1: Establish a Markov decision process model for the spaceborne observation system: Model the observation tasks of the spaceborne payload at different times as a sequential decision problem from state to action to reward; Step S2: Adopt the improved TD3 algorithm: Introduce a dual Critic network structure to solve the problem of Q-value overestimation, and enhance training stability by delaying the update of the Actor policy network; Step S3: Introduce a self-attention mechanism: Design a multi-head attention layer in the Actor network to model high-order dependencies between state features and enhance the policy network's ability to model complex observation scenarios. Step S4: Apply batch normalization: Add batch normalization layers to the Actor and Critic networks to improve the stability of the input distribution during training and accelerate the convergence speed. Step S5: The strategy output is used to control the observation behavior of the spaceborne payload in real time, optimize the receiving path of interference sources, and improve the quality of Doppler measurement data.

2. The method for optimizing spaceborne interference source Doppler observation strategy based on self-attention as described in claim 1, characterized in that, Step S1 includes the following specific processes: collecting satellite orbital parameters at different times; observing environmental state variables related to the mission; and constructing the state space. The state space includes orbital parameters, historical state information, attitude angles, and receiving parameters, which are then normalized. The observation tasks of the spaceborne payload at different times are modeled as a sequential decision problem from state to action to reward, expressed as: in, and These are the values ​​of the i-th state variable before and after normalization, respectively. , Let be the mean and standard deviation of the i-th state variable, respectively.

3. The method for optimizing spaceborne interference source Doppler observation strategy based on self-attention as described in claim 1, characterized in that, In step S1, a reward function R needs to be constructed. This function weights and combines the signal-to-noise ratio of the received interference signal, the Doppler frequency shift measurement accuracy, and the scheduling coverage efficiency to obtain a comprehensive evaluation index. The reward function R is expressed as follows: in , , These are weighting factors used to adjust the impact of each evaluation item on the total reward. The degree of contribution, It can be calculated from the peak steepness of the spectrum. Used to measure the quality of received signals. It is used to reflect the efficiency of resource utilization.

4. The method for optimizing spaceborne interference source Doppler observation strategy based on self-attention as described in claim 1, characterized in that, The TD3 algorithm described in step S2 employs a dual-Critic network structure to construct value functions respectively. The smaller of the two values ​​is used as the target value for updating, as shown below: in, For the target Q value, The value function of the dual-critic network is given by the policy network, which employs a delayed update mechanism to improve the stability of policy learning.

5. The method for optimizing spaceborne interference source Doppler observation strategy based on self-attention as described in claim 1, characterized in that, In step S3, a multi-head self-attention mechanism is introduced into the Actor network to model the state variables. The attention calculation method is expressed as follows: in, It is a vector of the current state features after transformation, used for proactive correlation matching. It is another set of feature vectors that are matched against the query to generate attention weights. It is the information itself that needs to be weighted and summed. It is the dimension of the key vector, which serves as a scaling factor to prevent the dot product from becoming too large and causing gradient saturation. The output of each head is fed into subsequent network modules through concatenation and linear mapping.

6. The method for optimizing spaceborne interference source Doppler observation strategy based on self-attention as described in claim 1, characterized in that, The batch normalization module mentioned in step S4 is used to normalize the input data of each hidden layer, thereby improving the distribution stability of each layer during training. Its calculation formula is as follows: in, and These represent the input and output of the batch normalization layer, respectively. and These represent the mean and variance of the current mini-batch data along the feature dimension, respectively. and These are the learnable scaling and offset parameters, respectively. To prevent small constants from being divided by zero.

7. The method for optimizing spaceborne interference source Doppler observation strategy based on self-attention as described in claim 1, characterized in that, The Doppler frequency shift estimation accuracy in the reward function is obtained by extracting the principal components of the spectrum using short-time Fourier transform, and is expressed as follows: in, It is the spectrum data received in the discrete-time domain. The window function extracts a segment of the signal at time offset τ, where N is the number of points in the Fourier transform. It is a discrete frequency index, corresponding to the Doppler frequency shift component in the analysis. At time window position τ, frequency The complex spectrum value at the given location, with the main peak position as the frequency shift observation, is expressed by the following formula for calculating the signal-to-noise ratio: in, It is the received interference signal component. It is the environmental noise component. Represents variance calculation, used to measure the fluctuation power of a signal or noise.

8. The method for optimizing spaceborne interference source Doppler observation strategy based on self-attention as described in claim 1, characterized in that, The state transition function in the MDP model described in step S1 is constructed based on the orbital dynamics model and can be expressed as: Where r is the satellite's position vector, The gravitational constant of Earth, It is the perturbation acceleration term, representing the disturbance of the basic orbit by various small external forces. It also considers the orbit update caused by the perturbation of Earth's gravity and the changes in attitude control, so as to simulate the dynamic adjustment of the state transition probability.

9. The method for optimizing spaceborne interference source Doppler observation strategy based on self-attention as described in claim 1, characterized in that, During the training process, the four-tuple of state, action to reward, and next state is stored in the experience replay pool, and the parameters of the policy network and value network are updated by mini-batch random sampling.

10. The method for optimizing spaceborne interference source Doppler observation strategy based on self-attention according to claim 1, characterized in that, In step S2 of the training process, a target network mechanism is used for stability enhancement. The target Actor network parameter update method is expressed as follows: in This is the soft update coefficient, with a value ranging from 0 to 1.

Images