An Autonomous Driving Localization Method and System Based on LSTM Deep Reinforcement Learning

Abstract

This invention discloses an autonomous driving localization method and system based on LSTM deep reinforcement learning. The method includes: performing normalization preprocessing on the complex environment of autonomous driving based on partially observable Markov decisions to obtain real-time dynamic localization results; inputting the real-time dynamic localization results into an LSTM-based deep reinforcement learning model for correction processing to obtain a position correction action evaluation value; and correcting the position of the autonomous vehicle based on the position correction action evaluation value. The system includes a prediction module, a correction module, and an application module. By using this invention, more accurate satellite positioning can be generated by considering the high dynamics, temporal sequence, and complexity of autonomous driving in complex environments. This invention, as an autonomous driving localization method and system based on LSTM deep reinforcement learning, can be widely applied in the field of autonomous driving satellite positioning technology.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving satellite positioning technology, and in particular to an autonomous driving positioning method and system based on LSTM deep reinforcement learning. Background Technology

[0002] Autonomous driving is an emerging technology in the context of a new round of technological revolution, representing the strategic high ground of future automotive technology. Satellite positioning technology is the technology that allows cars to know their exact location, and it is the prerequisite and foundation for realizing autonomous driving navigation functions. Satellite positioning will provide autonomous vehicles with absolute position information. In addition to accurately perceiving its own position, high absolute positioning accuracy can also improve the search efficiency of map matching and provide a certain backup function when sensors are blocked. Therefore, accurate satellite navigation and positioning technology is an indispensable core technology for autonomous driving. However, the positioning accuracy of global satellite navigation and positioning systems is affected by various environmental factors, including buildings, trees, etc. In dynamic scenarios of autonomous driving, the vehicle's surrounding environment changes rapidly, causing dynamic interference to the satellite signals received by the positioning equipment, leading to inaccurate positioning. Existing research technologies include hardware-based methods such as auxiliary equipment and infrastructure to improve satellite positioning accuracy, as well as software-based methods. For example, multi-GNSS (GPS, BeiDou, GLONASS) models combine collected information for mutual correction, improving the positioning accuracy of smartphones. Similarly, sensor fusion techniques are used to improve GPS positioning accuracy; for instance, an adaptive Kalman filter model is constructed to adapt to dynamically changing driving environments, but this relies on the handling of GPS observation noise and motion models. Strict assumptions were made, and later scholars proposed a deep neural network (DNN) model for positioning correction, which transformed the position estimation problem solved by the traditional GNSS positioning algorithm into the problem of estimating the initial position correction. DNN was used to learn the function mapping from GNSS measurements to position correction. However, this method is a static learning method that does not consider the correlation of historical data and only relies on the initial position information for correction. It is difficult to adapt to the dynamic changes in positioning correction for autonomous driving in complex urban environments. Therefore, the proposed model cannot adapt to the dynamically changing environment and relies on strict prior assumptions about sensor or model parameters. In dynamic driving situations, the vehicle's surrounding environment changes rapidly. Using the deployment of reference stations and map matching to improve the vehicle's positioning accuracy has high deployment costs and is difficult to promote. Summary of the Invention

[0003] To address the aforementioned technical problems, the present invention aims to provide an autonomous driving localization method and system based on LSTM deep reinforcement learning, which can generate more accurate satellite positioning locations by considering the high dynamics, temporal sequence, and complexity of autonomous driving in complex environments.

[0004] The first technical solution adopted in this invention is: an autonomous driving localization method based on LSTM deep reinforcement learning, comprising the following steps:

[0005] Based on partially observable Markov decision-making, normalized preprocessing is performed on the complex environment of autonomous driving to obtain real-time dynamic positioning results.

[0006] The real-time dynamic positioning results are input into an LSTM-based deep reinforcement learning model for correction processing to obtain the position correction action evaluation value.

[0007] The autonomous vehicle's position is corrected based on the position correction action evaluation value.

[0008] Furthermore, the step of performing normalization preprocessing on the complex environment of autonomous driving based on partially observable Markov decisions to obtain the initial result of real-time dynamic localization specifically includes:

[0009] Real-time dynamic positioning technology is used to obtain complex environmental information and agent positioning information for autonomous driving.

[0010] Based on partially observable Markov decision-making, a complex environment model for autonomous driving is obtained.

[0011] The complex environmental information and agent localization information of autonomous driving are represented by an environmental model to obtain the corresponding belief state distribution.

[0012] The POMDP programming method is used to solve the corresponding belief state distribution to obtain real-time dynamic positioning results.

[0013] Furthermore, the step of solving the corresponding belief state distribution using the POMDP planning method to obtain the real-time dynamic positioning result specifically includes:

[0014] Based on the POMDP planning method, the corresponding action space, observation state space, belief state space and reward function are defined.

[0015] The agent's localization information is corrected by processing the action space to obtain the corrected agent localization information;

[0016] Based on the observation state space, predictive processing is performed on the agent's localization information to obtain the predicted agent's localization information.

[0017] Based on the belief state space, the predicted agent location information is replaced with the agent location information and the belief state space is updated.

[0018] By negativeening the sum of squared errors between the corrected and predicted agent localization information using the reward function, real-time dynamic localization results are obtained.

[0019] Furthermore, the formula for negating the sum of squared errors between the corrected agent localization information and the predicted agent localization information is as follows:

[0020]

[0021]

[0022] In the above formula, η represents the reward regularization factor, and r t This represents the model's corrected localization information at time t (x). t y t Matching location information with the map The negative Euclidean distance, γ t-1 R represents the reward discount factor. t This represents the cumulative reward at time t.

[0023] Furthermore, the step of inputting the real-time dynamic positioning results into an LSTM-based deep reinforcement learning model for correction processing to obtain a position correction action evaluation value specifically includes:

[0024] Construct an LSTM-based deep reinforcement learning model and input real-time dynamic localization results. The LSTM-based deep reinforcement learning model includes a policy network and a value network.

[0025] The real-time dynamic positioning results are corrected according to the preset action range based on the policy network, and the positioning observation results with latitude and longitude geographic location information are output.

[0026] Based on the value network, the positioning observation results with latitude and longitude geographic location information are optimized by minimizing the temporal difference error to obtain the position correction action evaluation value, which is used to evaluate the merits of the positioning correction strategy of the strategy network.

[0027] Furthermore, the step of correcting the input real-time dynamic positioning results based on the policy network according to a preset action range and outputting positioning observation results with latitude and longitude geographic location information specifically includes:

[0028] The real-time dynamic positioning results are input into the policy network, which includes an input layer, an LSTM network layer, a hidden layer, and an output layer.

[0029] The real-time dynamic positioning results are normalized based on the input layer to obtain normalized positioning information;

[0030] The LSTM network layers are used to characterize the interrelationships and influences between normalized and dynamic positioning information, so as to output positioning information with changing trends in the time domain for subsequent hidden layers and output layers.

[0031] Based on the hidden layer and the output layer, the represented positioning information is multiplied by the sigmoid function and the tanh function to output the positioning observation results with latitude and longitude geographic location information.

[0032] Furthermore, the step of characterizing the correlations and influences between the normalized location information based on the LSTM network layer to obtain the characterized location information specifically includes:

[0033] The normalized positioning information is input into the LSTM network layer, which includes an input gate, a forget gate, and an output gate.

[0034] The normalized positioning information is updated based on the input gate to obtain the updated positioning information.

[0035] The updated location information is removed based on the forget gate to obtain the removed location information.

[0036] The removed location information is rearranged through the output gate to obtain the represented location information.

[0037] Furthermore, the calculation formula for the forget gate is as follows:

[0038] f t =σ(W f ·g[h t-1 x t ]+b f )

[0039] In the above formula, W f Let b be the weight matrix. f x is the bias value. t The input to the current network is given by g, which represents vector concatenation, and h is the input to the network. t-1 f represents the input to the forget gate. t σ represents the forget gate, and σ represents the activation function sigmoid.

[0040] The expression for minimizing the timing difference error is as follows:

[0041] δ=r t +γV(e t+1 ,w)-V(e t w)

[0042] In the above formula, γ represents the discount rate, and e tLet r be the belief state at time t. t Let V represent the reward value at time t, V represent the value function, w represent the parameters of the value network, and δ represent the temporal difference error.

[0043] The second technical solution adopted in this invention is: an autonomous driving localization system based on LSTM deep reinforcement learning, comprising:

[0044] The prediction module performs normalized preprocessing on the complex environment of autonomous driving based on partially observable Markov decisions to obtain real-time dynamic positioning results.

[0045] The correction module is used to input the real-time dynamic positioning results into the LSTM-based deep reinforcement learning model for correction processing, and obtain the position correction action evaluation value.

[0046] The application module corrects the position of autonomous vehicles based on the position correction action evaluation value.

[0047] The beneficial effects of the method and system of this invention are as follows: This invention models the complex environment of autonomous driving by introducing a partially observable Markov decision process (POMDP), constructs an interactive environment for accurate satellite positioning in complex autonomous driving environments, uses the prediction information of past agents as confidence states, and the time-series signal of the confidence state at the previous moment can determine the next state. Furthermore, a deep reinforcement learning model based on a long short-term memory network is constructed. By introducing a long short-term memory network, features are extracted from the complex environment of autonomous driving to ensure that the model has strong decision-making and perception capabilities, establishes a method to improve the adaptive capability of dynamic environments, and realizes a high-precision satellite positioning method for autonomous driving in complex urban environments. Attached Figure Description

[0048] Figure 1 This is a flowchart of the steps of the autonomous driving localization method based on LSTM deep reinforcement learning in this invention;

[0049] Figure 2 This is a block diagram of the autonomous driving positioning system based on LSTM deep reinforcement learning according to the present invention.

[0050] Figure 3 This is a schematic diagram of the network structure of the deep reinforcement learning model based on the long short-term memory network of this invention;

[0051] Figure 4 This is a schematic diagram of the process of training a model in a complex environment of autonomous driving based on deep reinforcement learning with a long short-term memory network structure, according to the present invention.

[0052] Figure 5 This is a schematic diagram of the localization results on a real road surface based on deep reinforcement learning using a long short-term memory network structure, according to the present invention. Detailed Implementation

[0053] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0054] Reference Figure 1 This invention provides an autonomous driving localization method based on LSTM deep reinforcement learning, which includes the following steps:

[0055] S1. Environment setup for precise satellite positioning in complex autonomous driving environments;

[0056] Specifically, refer to Figure 4 The data in this invention originates from the Beidou PNT Technology Laboratory of the Guangdong-Hong Kong-Macao Greater Bay Area Joint Laboratory. It utilizes location data acquired by a ZED-F9P high-precision centimeter-level RTK positioning module satellite receiver. Considering that the agent's current state information cannot determine its state information at the next moment, thus failing to satisfy a Markov decision process, a Partially Observable Markov Decision Process (POMDP) ​​is introduced to model the environment. By using past predictions from the agent as confidence states, a corresponding action space A is defined. t Observation space O t Belief state space b t The reward function is R(s, a);

[0057] Action Space A t {a t a2, ..., a m*n The model's control strategy requires outputting corresponding positioning correction actions, and the object of the correction is the positioning result broadcast by the positioning device. Therefore, the action space can be defined as a scalar matrix. m and n represent the correction step size in latitude and longitude, respectively, with the correction step size being {-10, -9, ..., 0, ..., 9, 10}, i.e., 21. 2 = 441 correction steps;

[0058] Observation state space O t {e t-N-1 , ..., e t-1 P t The model corrects N-1 points based on historical predictions within the time interval, and the location broadcast result P. n Define the observation state space;

[0059] Belief state space b t {e t-N-1 , ..., e tCompared to the observed state space points, after the model makes prediction corrections, the current prediction correction value e is used. t To replace P t Update the belief state space;

[0060] Reward function R(s, a): The reward function is defined based on the Euclidean distance between the model-corrected positioning information and the map-matched positioning information. That is, it is the negative of the sum of the squared errors of the difference between the model-corrected positioning result and the map-matched result. Since the positioning results corrected by the model at different times t have temporal sequence in the autonomous driving process, it is necessary to accumulate the positioning reward of the entire route according to the time series. The calculation formula for negativeing ​​the sum of the squared errors of the difference between the corrected agent positioning information and the predicted agent positioning information is as follows.

[0061]

[0062]

[0063] In the above formula, η represents the reward regularization factor, and r t This represents the model's corrected localization information at time t (x). t y t Matching location information with the map The negative Euclidean distance is used by the model to seek the optimal correction policy by maximizing the reward function, γt. -1 R represents the reward discount factor. t This represents the cumulative reward at time t.

[0064] S2. Construction of a deep reinforcement learning model based on long short-term memory network;

[0065] Specifically, considering the high dynamics, temporal sequence, and complexity of autonomous driving in complex environments, the use of fully connected neural networks with the PPO algorithm to approximate policy and value functions in deep reinforcement learning models is no longer sufficient. This invention constructs a deep reinforcement learning model using an LSTM network architecture for both the policy and value networks. First, an LSTM network is introduced to extract temporal signal features from the autonomous driving environment, outputting useful perception information and enhancing the learning ability of sequential sample data. Second, the policy network learns a corrected policy based on the current output of the LSTM network, outputting a position correction action evaluation value and generating a precise satellite positioning location. The value network evaluates the merits of the corrected policy. Finally, through continuous interaction between the deep reinforcement learning model and the complex autonomous driving environment, the output value of the policy network continuously approaches the optimal policy. The output value of the value network continuously approximates the true value function V. * .

[0066] S21, Long Short-Term Memory Network;

[0067] Long Short-Term Memory (LSTM) networks are mainly used in scenarios where the input data is in sequence. In complex environments such as autonomous driving and satellite positioning, the data received by deep reinforcement learning models is temporal. By processing the original data through LSTM networks, the learning ability of sequential sample data can be enhanced, and the processed state information can be encoded into a fixed sequence and used for training in the policy network and value network.

[0068] Reference Figure 3 First, the sequence sample data is normalized. The processed data is then passed through the forget gate of the Long Short-Term Memory network to remove some useless information in the sequence sample information. Second, the effective information is updated through the input gate, and the effective information is output by the output gate and saved to the hidden state. The processed state information is then encoded into a fixed sequence. Finally, the processed sequence is further output to the policy network and the value network for training.

[0069] Specifically, firstly, in order to reduce the error of the deep reinforcement learning model caused by the large difference in magnitude between the input feature data, it is necessary to normalize the input feature parameters. For the longitude and latitude information of the positioning signal, a transformation function is used to convert the original feature information into information with values ​​in the range [0, 1]. The transformation function is shown below:

[0070]

[0071] In the above formula, x t This represents RTK positioning latitude feature data. This represents the normalized RTK positioning latitude feature data. Similarly, yt and This indicates that the information is converted into a value range of [0, 1] using a conversion function;

[0072] Secondly, the normalized data is input into the forget gate of the LSTM network to remove some useless information in the sequence sample information, such as... Figure 3 As shown, the forget gate mainly uses the sigmoid function to determine the network output h of the previous time step. t-1 The cell state C of the network at the previous time step t-1 Does it continue to exist in the current network cell state C? t In Chinese, the formula for calculating the forget gate is as follows:

[0073] fx=σ(W f ·g[h t-1 x t ]+b f )

[0074] In the above formula, W f Let b be the weight matrix. f x is the bias value. t The input to the current network is given by g, which represents vector concatenation, and h is the input to the network. t-1 f represents the input to the forget gate. t This represents the forget gate, and σ represents the activation function, the sigmoid function.

[0075] Then, the input gate updates the valid information processed by the forget gate, and the information output by the sigmoid function is multiplied by the information output by the tanh function to determine how much of the current input xx needs to be transmitted to the cell state C. t In this context, the input gate calculation formula is as follows:

[0076] i t =σ(W i ·g[h t-1 x1]+b i )·tanh(W c ·g[h t x1, x t ]+b c )

[0077] In the above formula, W i W c Let b represent the weight matrix. i b c Indicates the bias amount;

[0078] Finally, the processed state information is encoded into a fixed sequence by the output gate and output to the policy network and value network for training. The output gate also uses the multiplication of the information output by the sigmoid function and the tanh function to determine the unit state C. t How much information can be passed to the current output h? t In this context, the formula for calculating the output gate is as follows:

[0079] h t =σ(W0·g[h t-1 x t ]+b0)·tanh(C t )

[0080] In the above formula, C t h represents the state of the unit. t This represents the current output of the output gate, W0 represents the weight matrix, and b0 represents the bias.

[0081] S22, Deep reinforcement learning model based on long short-term memory network;

[0082] Specifically, considering that the difficulty of accurate satellite positioning in the complex environment of autonomous driving is the choice of step size, and that the large difference between the old and new policies during training is not conducive to the learning of the agent, resulting in slow convergence speed and inability to meet real-time requirements during model training, the Proximal Policy Optimization (PPO) algorithm continues the step size selection mechanism of the confidence region policy optimization algorithm, ensuring that its policy keeps increasing, that is, the value function remains monotonically constant during training, which solves the above problems well. Therefore, this invention uses the PPO algorithm to create a deep reinforcement learning model with a policy-value structure, and introduces a long short-term memory network to extract features from the complex environment of autonomous driving to ensure that the model has strong decision-making and perception capabilities. The deep reinforcement learning model based on the long short-term memory network consists of two deep neural networks.

[0083] The first network is the policy network, whose output is the action policy π. θ (a|s), where θ represents the weight parameter matrix of the policy network, which is used to update the agent's behavioral policy by finding an optimal action policy π. θ To maximize the expected reward value R(s, a) using (a|s), since the number of neurons in the input layer of the deep reinforcement learning policy network should be the same as the dimension of the input state, and the number of neurons in the output layer should be the same as the dimension of the correction, the policy in the current state is used as the localization information of RTK. The reward value is obtained after each action is executed, and the policy network parameters are updated through the policy gradient, thereby training the policy network.

[0084] The policy network structure mainly consists of four neural network layers. The first layer is a Long Short-Term Memory (LSTM) network layer, containing 50 hidden LSTM memory units. It encodes the correlation features of the sequence data using the forget gate and input gate, outputting the output as the agent's input. This allows the agent to consider the temporal localization data of the environmental state during decision-making. Based on the LSTM network layer, it represents the interrelationships and influences between normalized, dynamic localization information, providing subsequent hidden layers and output layers with localization information showing changing trends in the time domain. The second and third layers are hidden layers of fully connected layers. Because the hidden layers of a fully connected network can encode the correlation features of the sequence data using the forget gate and input gate, it outputs localization information with changing trends in the time domain. The features of the data are abstracted into another dimension. The number of neurons in the hidden layer has a significant impact on the network's fitting ability. Insufficient neurons will reduce the network's fitting ability and fail to achieve the expected correction accuracy, while too many neurons will cause the network to overfit or increase the training time unnecessarily. Therefore, based on the positioning accuracy requirements, the number of neurons in the second and third layers of the network is set to 128 and 64 respectively, and the ReLU nonlinear function is used as the activation function for better feature extraction. The fourth layer is the output layer of the fully connected layer, containing 441 hidden neuron nodes, which outputs the corrected satellite positioning longitude and latitude information.

[0085] The second network is the value network, whose output is the state value V(s, w), where w represents the weight parameter matrix of the value network. The value network estimates the state value V(s, w) and provides feedback to the policy network. By minimizing the temporal difference error (TD-error) formed by the state value V(s, w) and the reward value R(s, a), the value network parameter w is updated, thereby outputting a more accurate evaluation value of the correction action, which is used to evaluate the quality of the policy network's localization correction strategy.

[0086] Since the input features of the policy network and the value network are similar, their network structure designs are also almost identical. The value network of the deep reinforcement learning model based on the long short-term memory network mainly consists of four layers of neural networks. The first layer is a long short-term memory network layer, which contains 50 hidden LSTM memory units to extract time series reasoning information from the original sequence data and outputs it to the fully connected layer for further processing. The second and third layers are fully connected layers with 64 and 32 hidden neuron nodes, respectively, and use the ReLU nonlinear function as the activation function for better feature extraction. The fourth layer is the output layer of the fully connected layer, with one output neuron, which is used to evaluate the value of the policy network's corrective actions.

[0087] S3. Model training in complex autonomous driving environments using deep reinforcement learning based on long short-term memory network structures;

[0088] Specifically, the training process of deep reinforcement learning based on long short-term memory network structures in complex autonomous driving environments mainly includes the interaction between the model and the partially observable Markov decision-making (POMDP) ​​environment to obtain a sliding window observation buffer Z. t The policy network and value network parameters in the model are updated, and the policy network with corrected actions is obtained after the model training is completed, resulting in higher accuracy localization results.

[0089] First, the model interacts with the complex environment of autonomous driving to obtain the sliding window observation buffer Z. t Since the model's observations of the current state cannot determine the state at the next moment, failing to satisfy the Markov decision process, it is necessary to use past agent predictions as confidence states. Therefore, a partially observable Markov decision process (POMDP) ​​is introduced to model the environment. The model receives a new RTK positioning observation, e. t The sliding window moves forward one step to form a new observation vector of constant size N, where e t The last element consists of N-1 belief states e. t-1 Composition Z tIn the remainder, when the RTK satellite receiver reports a new observation point et, the deep reinforcement learning model is immediately trained and moved to the next observation buffer. That is, in each training step, the input state of the reinforcement learning model contains the observation point reported by RTK and a series of historical predictions.

[0090] The framework behaves similarly to a signal filter, taking the raw longitude and latitude geographic locations reported by RTK and the predicted location information from the deep reinforcement learning model as input states. The deep reinforcement learning policy network then selects small or large correction actions based on existing policies. The output longitude and latitude geographic locations, together with the previous RTK positioning observations, constitute a sliding window observation buffer Z. t ;

[0091] Then, the policy network and value network in the model utilize a sliding window observation buffer Z. t For data-driven parameter updates, the goal of a value network is to minimize the temporal difference error to update the network parameters w, thereby continuously approximating the output value to the true value function V. * By inputting new RTK positioning observations e t Given a value network, the expression for the time-difference error can be obtained using the temporal difference algorithm, as shown below:

[0092] δ=r t +γV(e t+1 ,w)-V(e t w)

[0093] In the above formula, γ represents the discount rate, and e t Let r be the belief state at time t. t This is represented as the reward value at time t;

[0094] By minimizing the expected value of the temporal difference error, the objective function of the value network can be obtained, as shown below:

[0095]

[0096] In the above formula, Represent the expected function;

[0097] For the policy network, the observation buffer Z is determined by the sliding window of the input. t Advantage function calculated from data and value networks The strategy π is updated every t steps. θ The goal of the policy network is to update the policy network parameters θ using the policy gradient ascent method, so that the output value continuously approaches the optimal policy. Through the advantage function The objective function for constructing the policy network is shown below:

[0098]

[0099] in, The dominant function can be replaced by temporal difference error; ∈ represents hyperparameters that need to be manually adjusted; clip(·) represents the pruning function, which modifies the objective function of the policy network by pruning the probability ratio, eliminating ρ t (θ) is the part outside the interval [1-∈, 1+∈], ρ t (θ) represents the probability ratio between the old and new policy networks, and its expression is shown below:

[0100]

[0101] Since the objective functions of both the value network and the policy network are smooth convex functions, the stochastic gradient descent algorithm is directly used to update the weight parameters of the value network and the policy network. Specifically, the weight parameter matrix w of the value network is updated iteratively using stochastic gradient descent, and the weight parameter matrix θ of the policy network is updated iteratively using stochastic gradient ascent. The update formulas are as follows:

[0102]

[0103]

[0104] In the above formula, α > 0 represents the learning rate of the value network, and β > 0 represents the learning rate of the policy network.

[0105] Finally, the model continuously interacts with the complex environment of autonomous driving to obtain the optimal strategy that maximizes long-term benefits. In other words, it uses a deep reinforcement learning policy network to process the input signal to extract features, outputs a position correction action evaluation value, and generates a more accurate satellite positioning position, thereby solving the problem of dynamic interference caused by environmental changes in autonomous driving scenarios.

[0106] Reference Figure 5 The initial results from RTK localization are input into a deep reinforcement learning model based on a long short-term memory network structure for correction. The corrected location information is then output to the real road surface in a time sequence.

[0107] Reference Figure 2 An autonomous driving localization system based on LSTM deep reinforcement learning includes:

[0108] The prediction module performs normalized preprocessing on the complex environment of autonomous driving based on partially observable Markov decisions to obtain real-time dynamic positioning results.

[0109] The correction module is used to input the real-time dynamic positioning results into the LSTM-based deep reinforcement learning model for correction processing, and obtain the position correction action evaluation value.

[0110] The application module corrects the position of autonomous vehicles based on the position correction action evaluation value.

[0111] In summary, referring to Figure 5 This invention uses deep reinforcement learning to correct errors caused by dynamic observation noise in autonomous driving positioning, thereby improving positioning accuracy. First, it constructs an interactive environment for precise satellite positioning in complex autonomous driving environments. Simultaneously, it builds a deep reinforcement learning model based on a long short-term memory (LSTM) network. Since the agent's current state information cannot determine the next state information, failing to satisfy a Markov decision process, a partially observable Markov decision process (POMDP) ​​is introduced to model the environment. Past agent predictions are used as confidence states, and the temporal signal of the previous confidence state can determine the next state. The deep reinforcement learning model based on the LTM network is constructed, including the design and parameter selection of the policy network and value network structures. Second, the model is trained in a complex autonomous driving environment, including the agent-environment interaction process and the parameter update process of the policy and value networks. This yields a policy controller for corrective actions, resulting in higher-precision positioning results. Finally, the trained model is placed in a real, complex autonomous driving environment and corrected by inputting the initial results of RTK positioning received from the real world. The corrected position information is then output to the real road surface in a time sequence.

[0112] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0113] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. An automatic driving positioning method based on LSTM deep reinforcement learning, characterized in that, The method comprises the following steps: Based on the partially observable Markov decision, the complex environment of automatic driving is normalized and pretreated, and the real-time dynamic positioning result is obtained; The real-time dynamic positioning result is input into the deep reinforcement learning model based on LSTM for correction processing, and the position correction action evaluation value is obtained; Based on the position correction action evaluation value, the position of the automatic driving vehicle is corrected; The step of predicting the complex environment of automatic driving based on the partially observable Markov decision to obtain the initial result of real-time dynamic positioning specifically comprises: Obtain the complex environment information and agent positioning information of automatic driving through real-time dynamic positioning technology; Model the complex environment of automatic driving based on the partially observable Markov decision to obtain the environment model; Represent the complex environment information and agent positioning information of automatic driving through the environment model to obtain the corresponding belief state distribution; Solve the corresponding belief state distribution through the POMDP planning method to obtain the real-time dynamic positioning result; The step of solving the corresponding belief state distribution through the POMDP planning method to obtain the real-time dynamic positioning result specifically comprises: Based on the POMDP planning method, define the corresponding action space, observation state space, belief state space and reward function; Correct the agent positioning information through the action space to obtain the corrected agent positioning information; Predict the agent positioning information based on the observation state space to obtain the predicted agent positioning information; Based on the belief state space, replace the predicted agent positioning information with the agent positioning information and update the belief state space; Take the negative of the sum of squared errors between the corrected agent positioning information and the predicted agent positioning information through the reward function to obtain the real-time dynamic positioning result; The calculation formula of taking the negative of the sum of squared errors between the corrected agent positioning information and the predicted agent positioning information is as follows: In the above formula, denotes a reward regularization factor, denotes a model in the information of the model correction positioning at the moment the Euclidean distance of the map matching positioning information the negative number, denotes a reward discount factor, denotes the cumulative reward return at the moment. 2.The automatic driving positioning method based on LSTM deep reinforcement learning according to claim 1, wherein, The step of inputting the real-time dynamic positioning result into the deep reinforcement learning model based on LSTM for correction processing to obtain the position correction action evaluation value specifically comprises: Build a deep reinforcement learning model based on LSTM and input the real-time dynamic positioning result, wherein the deep reinforcement learning model based on LSTM comprises a policy network and a value network; Based on the policy network, correct the input real-time dynamic positioning result according to the preset action amplitude, and output the positioning observation result with latitude and longitude geographic position information; Based on the value network, optimize the positioning observation result with latitude and longitude geographic position information by minimizing the time difference error to obtain the position correction action evaluation value, which is used to evaluate the pros and cons of the positioning correction strategy of the policy network. 3.The automatic driving positioning method based on LSTM deep reinforcement learning according to claim 2, characterized in that, The step of correcting the input real-time dynamic positioning result according to the preset action amplitude based on the policy network to output the positioning observation result with latitude and longitude geographic position information specifically comprises: Input the real-time dynamic positioning result into the policy network, wherein the policy network comprises an input layer, an LSTM network layer, a hidden layer and an output layer; The input layer is used to normalize the real-time dynamic positioning result to obtain normalized positioning information; The LSTM network layer is used to represent the correlation and influence between the normalized dynamic positioning information, and output positioning information with a change trend in the time domain for the subsequent hidden layer and output layer; Based on the hidden layer and the output layer, the positioning information after representation is multiplied by the sigmoid function and the tanh function, and the positioning observation results with latitude and longitude geographic position information are output. Based on the hidden layer and the output layer, the positioning information after representation is multiplied by the sigmoid function and the tanh function, and the positioning observation results with latitude and longitude geographic position information are output. 4.The automatic driving positioning method based on LSTM deep reinforcement learning according to claim 3, wherein, The step of representing the correlation and influence between the normalized positioning information by the LSTM network layer to obtain the represented positioning information specifically includes: The normalized positioning information is input into the LSTM network layer, which includes an input gate, a forget gate and an output gate; The input gate is used to update the normalized positioning information to obtain updated positioning information; The forget gate is used to eliminate the updated positioning information to obtain eliminated positioning information; The output gate is used to perform sequence rearrangement on the eliminated positioning information to obtain the represented positioning information. 5.The automatic driving positioning method based on LSTM deep reinforcement learning according to claim 4, characterized in that, The calculation formula of the forget gate is as follows: In the above formula, is a weight matrix, is a bias quantity, is the input of the current network, denotes vector splicing, denotes the input quantity of the forget gate, denotes the forget gate, denotes the activation function sigmoid. 6.The automatic driving positioning method based on LSTM deep reinforcement learning according to claim 5, wherein, The expression of the minimized time series difference error is as follows: In the above formula, denotes the discount rate, denoted as belief state at time t, denoted as reward value at time t, denotes the value function, denotes the parameters of the value network, denotes the time-difference error.

7. An automatic driving positioning system based on LSTM deep reinforcement learning, characterized in that, The method for performing the LSTM deep reinforcement learning-based automatic driving positioning method according to claim 1 includes the following modules: A prediction module is used to normalize the complex environment of the automatic driving vehicle based on a partially observable Markov decision to obtain real-time dynamic positioning results; A correction module is used to input the real-time dynamic positioning results into the LSTM-based deep reinforcement learning model for correction to obtain a position correction action evaluation value; An application module is used to correct the position of the automatic driving vehicle based on the position correction action evaluation value.

Images