Frequency domain feature perception graph reinforcement learning single Beidou satellite positioning enhancement method

Abstract

The invention discloses a frequency domain feature perception graph reinforcement learning single Beidou satellite positioning enhancement method. The method comprises the following steps: building a reinforcement learning environment based on a single Beidou multi-frequency satellite graph; constructing and integrating a graph feature processor, a frequency domain state sequence prediction module and an actor-judge network to obtain a positioning correction model; training the positioning correction model by using a large amount of state sequence data generated in the interaction process of the reinforcement learning agent and the environment; and carrying out positioning correction by using the trained positioning correction model. By using the method, satellite measurement errors with potential regularity can be effectively modeled, and satellite dynamic positioning correction is realized. The method can be widely applied to the field of satellite positioning.

Description

Technical Field

[0001] The present invention relates to the field of satellite positioning, and in particular to a frequency domain feature-aware graph reinforcement learning single Beidou satellite positioning enhancement method. Background Art

[0002] Nowadays, the Global Navigation Satellite System (GNSS) is the most widely used method among various positioning technologies. All walks of life are applying the global satellite navigation and positioning system in complex multipath environments, so the demand for high-precision positioning of satellite navigation systems has increased dramatically. Among the "3+16" key areas, the application of single Beidou is particularly prominent. These application areas include housing and construction planning, transportation and logistics, etc., which are important directions for national development. Single Beidou allows devices to receive signals from the Beidou satellite navigation system completely independently, supports pure Beidou satellite navigation system signal solution, and performs positioning and timing. The uniqueness of this technology is that it does not rely on other satellite navigation systems, but is completely based on the Beidou satellite navigation system, thus ensuring the accuracy and reliability of positioning. However, in complex urban environments, including urban canyons and under urban overpasses, the multipath effect caused by the refraction and reflection of GNSS signals by obstacles, the signal attenuation caused by buildings and trees, the decrease in the number of visible satellites, and the deterioration of geometric distribution, etc., cause satellite positioning to deviate by more than ten meters, which cannot meet the accuracy requirements of lane-level navigation. In recent years, with the rapid development of artificial intelligence technology, applying artificial intelligence methods to positioning correction in urban environments has become a mainstream solution. However, the trained models have poor generalization performance and low positioning accuracy, and cannot be widely used in practice. One of the key technical problems is that satellite observation features are time-related, and there are certain potential change patterns over a period of time, such as those from satellite orbit periodic errors, clock drift, atmospheric interference, etc. The resulting satellite observations show certain regular fluctuations. However, the potential patterns of the data are often not clearly reflected in the time domain. Artificial intelligence models that only rely on satellite features in the time domain often lack the ability to capture the long-term trends and short-term disturbances of satellite signal features. Summary of the invention

[0003] In view of this, in order to solve the technical problem that most of the existing positioning methods only rely on satellite feature input in time series and cannot capture the potential changes of data features, which leads to low positioning accuracy, the present invention proposes a frequency domain feature-aware graph reinforcement learning single Beidou satellite positioning enhancement method, which uses discrete time Fourier transform to extract potential regularity information from state sequence signals, and assists in learning the optimal strategy to improve positioning accuracy. First, the partially observable Markov decision process (POMDP) ​​is introduced to model the environment and construct a reinforcement learning environment for single Beidou satellite measurement. Secondly, a positioning correction model based on frequency domain state sequence prediction is constructed, which is divided into three modules: 1) Construct a frequency domain state sequence prediction model to extract and utilize the long-term trend and underlying structure of the state sequence. 2) Construct a graph feature processor to extract the topological structure features of Beidou satellites and fully describe the state of the vehicle intelligent body. 3) Construct a cyclic actor-judge network to combine historical information and frequency domain information for positioning correction.

[0004] The frequency domain feature-aware graph reinforcement learning single Beidou satellite positioning enhancement method specifically includes the following steps:

[0005] Construction of a reinforcement learning environment based on single Beidou multi-frequency satellite images;

[0006] Build and integrate the graph feature processor, frequency domain state sequence prediction module and actor-judge network to obtain the positioning correction model;

[0007] The positioning correction model is trained using a large amount of state sequence data generated during the interaction between the reinforcement learning agent and the environment;

[0008] The trained positioning correction model is used to perform positioning correction.

[0009] Based on the above scheme, the present invention provides a frequency domain feature-aware graph reinforcement learning single Beidou satellite positioning enhancement method, which uses discrete time Fourier transform (DTFT) to convert the state sequence of state reinforcement learning to the frequency domain, thereby extracting its potential long-term trend and short-term disturbance information to participate in the learning of the optimal strategy. First, in order to extract the spatial topological information between satellites, a satellite image observation based on single Beidou multi-frequency is constructed, with the characteristics of each satellite as a node, and the edges between nodes are constructed according to the correlation of satellite measurements, thereby constructing the spatial relationship between satellites. Secondly, in order to capture the potential regular changes of satellite features, the input state sequence is converted to the frequency domain by DTFT, and then an auxiliary task model is constructed. By predicting the frequency domain state sequence for representation learning, the feature extractor of reinforcement learning is trained to capture the potential change law of the state sequence. Finally, a reinforcement learning model with a graph neural network structure and frequency domain state representation learning is constructed. Combined with the satellite airspace information extracted by the graph neural network and the satellite frequency domain information encoded by the auxiliary task model, it can effectively model satellite measurement errors with potential regularity and realize satellite dynamic positioning correction. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flowchart of the steps of a frequency domain feature-aware graph reinforcement learning single Beidou satellite positioning enhancement method of the present invention;

[0011] Figure 2 It is a data flow diagram of a frequency domain feature-aware graph reinforcement learning single Beidou satellite positioning enhancement method of the present invention;

[0012] Figure 3 It is a structural block diagram of a frequency domain state sequence prediction module according to a specific embodiment of the present invention;

[0013] Figure 4 It is a structural block diagram of an online encoder and a target encoder according to a specific embodiment of the present invention. DETAILED DESCRIPTION

[0014] In addition to the positioning correction method using artificial intelligence mentioned in the background technology, there is also a positioning correction method based on hardware equipment. However, this method relies too much on strict prior assumptions about sensor or model parameters, and is costly and difficult to promote on a large scale.

[0015] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0016] It should be noted that, for the convenience of description, only the parts related to the relevant invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other.

[0017] It should be understood that the "system", "device", "unit" and / or "module" used in this application is a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the word can be replaced by other expressions.

[0018] As shown in this application and claims, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of clearly identified steps and elements, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements. The elements defined by the sentence "includes a..." do not exclude the existence of other identical elements in the process, method, commodity or device that includes the elements.

[0019] In addition, flow charts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or following operations are not necessarily performed accurately in order. On the contrary, the various steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or a certain step or several steps of operations may be removed from these processes.

[0020] Reference Figure 1 , which is a flow chart of an optional example of a frequency domain feature-aware graph reinforcement learning single Beidou satellite positioning enhancement method proposed in the present invention. The method can be applied to a computer device. The positioning enhancement method proposed in this embodiment may include but is not limited to the following steps:

[0021] Step S1, building a reinforcement learning environment based on a single Beidou multi-frequency satellite image;

[0022] Step S2: constructing a positioning correction model, the positioning correction model including a graph feature processor, a frequency domain state sequence prediction module and an actor-judge network;

[0023] Step S3: training the positioning correction model based on the reinforcement learning environment to obtain a trained positioning correction model;

[0024] Step S4, obtaining the current Beidou satellite measurement signal;

[0025] Step S5, generating an initial position according to the current Beidou satellite measurement signal;

[0026] Step S6, inputting the current Beidou satellite measurement signal into the trained positioning correction model, and outputting correction data;

[0027] Step S7: correct the initial position according to the correction data to obtain a corrected positioning.

[0028] Satellite observation characteristics have certain potential change patterns over a period of time, such as those caused by satellite orbit periodic errors, clock drift, atmospheric interference, etc. The resulting satellite observation errors show certain regular fluctuations, but this potential pattern is often difficult to reflect in the time domain.

[0029] In the frequency domain state sequence prediction module, the input observation features of reinforcement learning are converted to the frequency domain through discrete Fourier transform, so as to capture the potential regular disturbances of satellite measurement data. The disturbances of various error sources will appear as fluctuations or peaks at specific frequencies in the frequency domain. In the frequency domain, low-frequency components usually correspond to long-term change trends, and components with frequencies close to zero correspond to the average value of the signal or the trend of slow changes; the high-frequency part can reveal the interference from short-term changes or noise in the signal, and the high-frequency part helps to distinguish the interference of fast changes in the signal. Therefore, it can help the reinforcement learning model better capture the potential regular changes of the signal, thereby improving the performance of the model.

[0030] The overall data flow of this method refers to Figure 2 .

[0031] In some feasible embodiments, the step S1 specifically includes:

[0032] Action Space a t Settings:

[0033] In order to avoid the difficulty of learning caused by the processing of a large number of discrete actions, this paper sets the action space as a continuous action space. And the action is defined as First, from a mean of μ and variance of σ 2 Sampling from Gaussian distribution, the positioning correction action is defined as “:=” indicates the definition. Among them, is the mean of the Gaussian distribution in the x direction at time t, is the variance in the x direction. Similarly, are the mean and variance in the y direction of the Gaussian distribution, are the mean and variance in the z direction respectively. Next, the position correction value Δx is sampled from the Gaussian distribution in the three directions respectively. t ,Δy t ,Δz t , correct the initial position and output the corrected position

[0034]

[0035] Where u represents the scale factor of the positioning correction action.

[0036] Reward function r t Settings:

[0037] In order to guide the reinforcement learning agent to learn the optimal positioning correction strategy, the initial position and correct position Calculate the corrected advantage error as the reward function of the environment:

[0038]

[0039] The coordinates of the reference position can be obtained through a high-precision but expensive integrated navigation system.

[0040] In some feasible embodiments, the step S2 specifically includes:

[0041] In order to make full use of the large amount of state sequence data generated during the interaction between the reinforcement learning agent and the environment, a model for predicting long-term state sequences in the frequency domain is constructed to assist representation learning, thereby extracting long-term trends and short-term disturbance information of the state sequence to assist in the search for the best strategy. Finally, a recurrent actor-judge network is used to learn the best positioning correction strategy.

[0042] 1) Construct frequency domain state sequence prediction model:

[0043] There are certain potential changes in star observation characteristics over a period of time, such as those from satellite orbit periodic errors, clock drift, atmospheric interference, etc. The resulting satellite observation errors show certain regular fluctuations, but this potential law is often difficult to reflect in the time domain. By using discrete Fourier transform to convert the state sequence to the frequency domain, the potential regular disturbances in the satellite measurement data can be captured. In the frequency domain, low-frequency components usually correspond to long-term change trends, and components with frequencies close to zero correspond to the average value or slow-changing trends of the signal; the high-frequency part can reveal interference from short-term changes or noise in the signal, and the disturbances from various error sources will appear in the frequency domain as fluctuations or peaks at specific frequencies. The high-frequency part helps to distinguish rapidly changing interference in the signal. Therefore, the state sequence information in the frequency domain can help the reinforcement learning model better capture the potential regular changes of the signal, thereby improving the performance of the model. The reinforcement learning model based on frequency domain state sequence prediction proposed in this patent is as follows: Figure 3 shown.

[0044] First, construct an auxiliary self-supervision task, F Re (s t ,at ), F Im (s t ,a t ) are the real and imaginary parts of the Fourier transform value of the predicted state sequence, s t For the state of reinforcement learning, we can use observation o t This state sequence prediction model includes an online network and a target network. The two networks use the same structure of encoder, predictor and projector. The loss function is calculated and the parameters are updated by back propagation along the dotted line. In this structure, the structure of the online encoder (Online Encoder) and the target encoder (Target Encoder) is the same, which is Figure 4 As shown:

[0045] The output of encoder φ is The input is the state s at time t t . (s t ,a t ,s t+1 ,a t+1 ) is a quadruple sampled in the experience replay area. This paper does not use the loss of reinforcement learning to update the encoder, but uses the loss gradient of the state sequence Fourier transform prediction model to update the real-time encoder, but does not update the target encoder.

[0046] The method of the present invention performs auxiliary self-supervision tasks by training a predictor F(·) to predict the discrete time Fourier transform (DTFT) of an infinite step-length state sequence to capture the structural information in the state sequence for representation learning. Due to the conjugate symmetry of the Fourier transform of the real state signal, this paper only predicts half a cycle, and then obtains a cycle transformation through symmetry to save storage space.

[0047] Modeling steps:

[0048] In s t ,a t For a given input, the expectation of the future state sequence set on the infinite horizon of the n-th step state is defined as

[0049]

[0050] γ∈[0,1) is a discount factor, E represents the expected operation, and the following is done The discrete Fourier transform of :

[0051]

[0052] Where ω is the frequency variable. Because the state sequence is a discrete signal, the corresponding discrete-time Fourier transform has a 2π periodicity in frequency. Therefore, L samples of equal distance after Fourier transform are taken in the interval [0,π], and a matrix of size L*D is predicted, where D is the dimension of the state space. In continuous time, the DTFTs at each moment are recursively related to each other, and the following relationship exists:

[0053]

[0054] in: and Γ is defined as

[0055]

[0056] Because the prediction result is complex value, we divide the last layer of the predictor F into two parts: the real part F Re and the imaginary part F Im The loss function of the prediction model is defined as:

[0057]

[0058] where π represents the actor network, and represents the representation of the observation at the corresponding time. In training, we train the encoder φ, predictor F, and projector ψ to minimize the auxiliary loss L pred (φ,F,ψ), and then use the trained encoder φ to alternately update the reinforcement learning actor-judge model.

[0059] 2) Graph feature processor:

[0060] This embodiment uses GNSS to measure M t The calculated pseudorange residual RES and LOS vector are used as positioning correction features, and then the observation of the reinforcement learning environment is defined. As node features of graph neural networks.

[0061]

[0062] and is the pseudorange residual and LOS vector of the jth satellite.

[0063] Next, we construct the edges between nodes in the BeiDou satellite map. We determine whether to establish an edge between nodes by judging the similarity between the two satellites in terms of signal frequency and pseudorange residual. In summary, this paper defines the observation of the reinforcement learning environment as containing BeiDou satellite node features. and the adjacency matrix The matrix:

[0064]

[0065] For the processing of graph structure:

[0066] Define the set of n nodes at time t as V t , define F t is the feature matrix of the node, and assuming that each node has m-dimensional features, the adjacency matrix of the graph is A t ∈D n×n , with a dimension of n×n. Next, define the jth satellite as a node at time t It is expressed as:

[0067]

[0068] The measurements of the reinforcement learning environment include GNSS satellite features and adjacency matrix. Define the graph G at time t t , to represent the information between satellites at the current moment. It is defined as:

[0069]

[0070] is the adjacency matrix, is the node feature. The core formula of the graph convolutional network algorithm is:

[0071]

[0072] in and are the hidden features of the (l+1) and lth layers at time t. The input of the network is the observation of the vehicle agent, that is, The feature propagation law of the lth layer of GNN is:

[0073]

[0074] in I is the identity matrix, yes The diagonal node degree matrix of is a trainable weight, with n (0) =4. The activation function σ in this method uses the ELU function. β is a parameter that controls the saturation range of the negative part in ELU. We use an L-layer GNN to calculate G t The final node embedding, GNN parameter set θ g = {W (1) ,…,W (L)}, in order to avoid satellite information loss, this embodiment uses pooling operation to aggregate multi-satellite features and connects the node outputs of different satellites to form a flattened node embedding b tAs a topological belief state:

[0075]

[0076] for , that is, the feature row of the j-th satellite in the L-th layer of the graph neural network.

[0077] 3) Circular Actor-Judge Network:

[0078] A policy output model recursive actor-judge network is constructed and trained using the proximal policy optimization algorithm (PPO). is the value function of the judges, and the corresponding parameter is θ c , the estimated value of the judge network’s output value can be defined as:

[0079]

[0080] The loss function of the recurrent actor network obtained from the objective function of the PPO algorithm is:

[0081]

[0082] are the updated and pre-updated parameters of the actor network, respectively. E(.) represents the computational expectation, and ∈ is a hyperparameter related to the clipping range. is the probability ratio, which varies in the range [1-∈, 1+∈], and by doing so, the distance between the new strategy and the old strategy is limited. is the generalized advantage estimator (GAE), which combines the discounted reward method to balance variables and biases in reinforcement learning. γ∈[0,1] is a discount factor used to adjust the policy estimate.

[0083] To help the judges network estimate the value more accurately, we define the TD error as:

[0084]

[0085] The goal of the reinforcement learning agent is to maximize the cumulative reward, so the judge network should accurately estimate the value of the belief state to make the estimate closer to the actual cumulative reward. Therefore, the mean squared return error (MSRE) is used to define the judge's loss function:

[0086]

[0087] In order to ensure that the agent fully interacts with the environment, we define the entropy loss function:

[0088]

[0089] Then define the total target loss function:

[0090]

[0091] Among them, β1 and β2 are coefficients.

[0092] In some feasible embodiments, the step S4 specifically includes:

[0093] This embodiment uses multi-frequency Beidou satellite measurement information as observation. t , that is, using multiple frequency points of satellite signals from the four systems of GPS, BDS, GLONASS and Galileo. We set the GNSS measurement set related to the jth Beidou satellite at time t to:

[0094]

[0095] in Expressed as N is the total number of currently visible satellites, is the set of satellite positions, is the set of pseudoranges received by the receiver, and j is the number of the jth satellite.

[0096] In some feasible embodiments, the step S5 specifically includes:

[0097] Through model-based approach Such as least squares (Weight Least Squares, WLS), Kalman filtering (Kalman Filtering, KF), with M t As input, we get a rough estimate of the initial position.

[0098]

[0099] In some feasible embodiments, the training process of the positioning correction model is specifically as follows:

[0100] Step 1: Use the receiver to collect positioning data in a real urban scene, for example, receive measurement signals from multiple satellites at time t to obtain a set of pseudo-range measurements The GNSS measurement set associated with the BeiDou satellite is: Next, we use model-based methods such as Weight Least Squares (WLS) and Kalman Filtering (KF) to calculate the M tAs input, get the estimated initial position

[0101] Step 2: Initialize an experience replay pool, and the agent continues to interact with the environment to collect experience. t Next, we get action a through the policy network t , discount rate γ, the number of steps to accumulate rewards T, and the learning rate lr.

[0102] Step 3: Next, define the initial parameters, including step size N, batch size t, and parameter set θ of the graph neural network g , the parameter set θ of the actor network a , the parameter set θ of the judge network c , the parameters of the real-time encoder (online encoder) and the target encoder (target encoder) of the state sequence prediction model. The parameters of the online encoder include the parameters of the corresponding predictor F and projector ψ, defined as θ aux ; The parameters of the target encoder include the corresponding predictor Projector The parameters are defined as Define the smoothing coefficient τ and update interval K of the target network.

[0103] Step 4: Use the experience in the experience replay pool to train the reinforcement learning model through the PPO algorithm, and use the stochastic gradient descent method to update the parameters of the graph network, actor network and judge network, i.e., θ g ,θ a ,θ c , the loss function is: Simultaneously update the online network of the state sequence prediction model: represents the gradient operation; and the parameters of the target network of the state sequence prediction model are updated after every k steps:

[0104] Step 5: After reaching the maximum number of training iterations, fix the parameters of the trained reinforcement learning model and deploy the multi-agent strategy network to the satellite positioning chip for testing in a real environment.

[0105] This overall training step is not much different from the process of reinforcement learning.

[0106] A frequency domain feature-aware graph reinforcement learning single Beidou satellite positioning enhancement device:

[0107] at least one processor;

[0108] at least one memory for storing at least one program;

[0109] When the at least one program is executed by the at least one processor, the at least one processor implements the frequency domain feature-aware graph reinforcement learning single Beidou satellite positioning enhancement method as described above.

[0110] The contents of the above method embodiments are all applicable to the present device embodiments. The functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0111] A storage medium stores processor-executable instructions, which are used to implement the above-mentioned frequency domain feature-aware graph reinforcement learning single Beidou satellite positioning enhancement method when executed by the processor.

[0112] The contents of the above method embodiments are all applicable to the present storage medium embodiments. The functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0113] The above is a specific description of the preferred implementation of the present invention, but the invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A frequency domain feature-aware graph reinforcement learning single Beidou satellite positioning enhancement method, characterized in that: The following steps are involved: Build a reinforcement learning environment based on single Beidou multi-frequency satellite images; Constructing a positioning correction model, the positioning correction model comprising a graph feature processor, a frequency domain state sequence prediction module and an actor-judge network; Training the positioning correction model based on the reinforcement learning environment to obtain a trained positioning correction model; Get the current Beidou satellite measurement signal; Generate an initial position according to the current Beidou satellite measurement signal; Inputting the current Beidou satellite measurement signal into the trained positioning correction model; Converting the current Beidou satellite measurement signal into a satellite image by means of the image feature processor; The state sequence is converted into the frequency domain by the frequency domain state sequence prediction module, and the regular disturbance in the Beidou satellite measurement signal is captured to obtain the predicted frequency domain information; Combining the predicted frequency domain information with historical information, generating correction data through an actor-judge network; The initial position is corrected according to the correction data to obtain a corrected positioning.

2. According to the frequency domain feature perception graph reinforcement learning single Beidou satellite positioning enhancement method of claim 1, it is characterized in that: The reinforcement learning environment specifically includes: Set the action space to a continuous action space and define the correction action as: in, is the mean of the Gaussian distribution in the x direction at time t, is the variance of the Gaussian distribution in the x direction at time t. Similarly, are the mean and variance of the Gaussian distribution in the y direction at time t, are the mean and variance of the Gaussian distribution in the z direction at time t; The reward function is set as follows: in, represents the initial position, Indicates the corrected position. Indicates the reference position.

3. According to claim 2, a frequency domain feature-aware graph reinforcement learning single Beidou satellite positioning enhancement method is characterized in that: The satellite image is constructed as follows: Define observation as a node of Beidou satellite map, as follows: in, and is the j-th Beidou satellite pseudorange residual and LOS vector; Based on the similarity of the signal frequency and pseudorange residuals measured by the two satellites, it is determined whether to establish an edge between the nodes.

4. According to claim 2, a frequency domain feature-aware graph reinforcement learning single Beidou satellite positioning enhancement method is characterized in that: The frequency domain state sequence prediction module includes an online network and a target network, wherein: The online network includes an online encoder, a real-time predictor and a real-time projector; The target network includes a target encoder, a target predictor and a target projector; Updating the real-time predictor by loss gradient; The real-time predictor and the target predictor are used to predict the discrete-time Fourier transform of an infinite-step state sequence to perform an auxiliary self-supervision task.

5. According to claim 4, a frequency domain feature-aware graph reinforcement learning single Beidou satellite positioning enhancement method is characterized in that: The last layer of the real-time predictor is divided into the real part F Re and the imaginary part F Im , define the loss function L of the real-time prediction period pred as follows: Among them, φ represents the real-time encoder, F represents the real-time predictor, and represents the representation of the observation at the corresponding time, π represents the actor network, represents the Fourier transform of the observation vector, and d represents the cosine similarity.

6. According to claim 5, a frequency domain feature-aware graph reinforcement learning single Beidou satellite positioning enhancement method is characterized in that: The overall objective loss function of the actor-judge network is expressed as follows: Among them, β1, β2 represent the corresponding coefficients, E(.) represents the calculation expectation, represents the loss function of the actor network, represents the loss function of the judge network, represents the policy entropy of the actor network, are the updated and pre-updated parameters of the actor network, respectively; ∈ is a hyperparameter related to the clipping range; is the probability ratio, is the generalized advantage estimate, and γ represents the discount factor.

7. A frequency domain feature-aware graph reinforcement learning single Beidou satellite positioning enhancement device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements a frequency domain feature-aware graph reinforcement learning single Beidou satellite positioning enhancement method as described in any one of claims 1-6.

Images