A combined navigation method, device and computer medium based on Q-learning reinforcement learning

Abstract

The present invention proposes a combined navigation method based on Q-learning reinforcement learning. The combined navigation method constructs an adaptive Kalman filter algorithm based on Q-learning reinforcement learning, and applies the adaptive Kalman filter algorithm to the SINS / GNSS / DVL combined navigation system to adaptively make an optimal estimate of the speed measurement noise. The advantage of the present invention is that the adaptive Kalman filter algorithm based on Q-learning reinforcement learning can adaptively make an optimal estimate of the speed measurement noise to better adapt to the situation where the DVL speed measurement error increases due to the complex seabed environment, thereby improving the accuracy of the combined navigation. In addition, the present invention can adaptively adjust the speed measurement noise value according to the external environment, with faster convergence speed, smaller speed error and position error, and greater stability.

Description

Technical Field

[0001] The present invention relates to the technical field of integrated navigation, and in particular to an integrated navigation method, device and computer medium based on Q-learning reinforcement learning. Background Art

[0002] In ocean navigation, SINS / GNSS / DVL integrated navigation systems are widely used in various ship navigation solutions. Kalman filtering (KF) plays a crucial role in the information fusion of SINS / GNSS / DVL integrated navigation systems. Typically, the KF measurement noise covariance matrix is ​​primarily calibrated by the instrument manufacturer and set as a constant matrix during use. Affected by complex ocean environments, DVL velocity measurement errors can increase, leading to decreased navigation accuracy and even error divergence. The varying environments in different oceans make it difficult to establish a precise model to describe DVL velocity measurement errors.

[0003] There are two main solutions to this problem in the existing technology. One is to use a kinematic model of the carrier instead of the DVL output velocity observation when DVL accuracy decreases. The other is to use an adaptive Kalman filter. Compared to traditional Kalman filters, adaptive Kalman filters introduce a forgetting factor to calculate weighting coefficients and re-estimate the observation noise covariance.

[0004] The measurement noise covariance matrix of a conventional Kalman filter is a constant matrix, making it less adaptable to complex seafloor environments. When DVL accuracy decreases, using the carrier's kinematic model instead of the DVL output velocity observations can only maintain integrated navigation accuracy for a short period of time and increases algorithmic complexity. The adaptive Kalman filter calculates the measurement noise covariance matrix based on new information. This matrix incorporates information from all observations, resulting in high matrix computational complexity and the potential for error coupling, leading to slower convergence. If only adaptive adjustment of velocity measurement noise is desired, the adaptive Kalman filter approach will increase the computational complexity.

[0005] In summary, there is an urgent need for a combined navigation method for complex unknown sea areas to solve the problems existing in the existing technology. Summary of the Invention

[0006] The present invention aims to provide a combined navigation method, device and computer medium based on Q-learning reinforcement learning. The specific technical solutions are as follows:

[0007] A combined navigation method based on Q-learning reinforcement learning, the steps are as follows:

[0008] Step S1: defining a coordinate system, specifically, the coordinate system includes an inertial coordinate system, an earth coordinate system, a geographic coordinate system, a navigation coordinate system and a carrier coordinate system;

[0009] Step S2: establishing an inertial navigation system error model. Specifically, establishing an inertial navigation system error model based on the coordinate system in step S1. The inertial navigation system error model includes an attitude error model, a velocity error model, a position error model, and an accelerometer and gyroscope error model.

[0010] Step S3: Establishing a Kalman filter, specifically, fusing the inertial navigation system error model in step S2 with the speed meter information and satellite positioning information to establish an integrated navigation system, and establishing its Kalman filter, the Kalman filter including a state equation, a measurement equation, and a Kalman filter equation;

[0011] Step S4: Obtain the optimal speed measurement noise and perform navigation solution. Specifically, use the Q-learning reinforcement learning algorithm to adaptively update the speed measurement noise in the integrated navigation system to obtain the optimal speed measurement noise. The position measurement noise remains unchanged. The optimal speed measurement noise and position measurement noise are substituted into the Kalman filter in step S3 to perform navigation solution to obtain navigation information.

[0012] Specifically, in step S1, coordinate systems are transformed between different coordinate systems through multiple basic rotations, and the transformation matrix between the coordinate systems is obtained by multiplying the rotation matrices. The transformation matrix is ​​obtained in the rotation order of zxy.

[0013] Specifically, in step S2, the speed meter information is collected using a Doppler speed meter, and the satellite positioning information is collected using a global navigation satellite system and a pseudo-range single point positioning method. The integrated navigation system is a strapdown inertial navigation system / global navigation satellite system / Doppler speed meter integrated navigation system.

[0014] Specifically, in step S4, the Q-learning reinforcement learning algorithm is set as follows:

[0015] The reinforcement learning environment is set as a strapdown inertial navigation system / global navigation satellite system / Doppler velocimeter integrated navigation system, the agent is Q-learning, the state is the DVL measurement noise, the action is the change in the DVL measurement noise, and the reward value is the inverse of the absolute value of the position error.

[0016] Specifically, set the reinforcement learning step size and reinforcement learning parameters as follows:

[0017] The reinforcement learning state space S is the set of DVL speed measurement noise V; the action space A is the set of DVL speed measurement noise change δV; the reward value r is the inverse of the absolute value of the speed error, that is, r = 1 / |δp|; the Q value table Q(S,A) is established; the action value function Q(s t ,at ), where a t represents the action at time t, S t Represents the state at time t.

[0018] Specifically, in the Q-learning reinforcement learning algorithm in step S4, the action is updated as follows:

[0019]

[0020] Among them, a t represents the action at time t+1 after the update, and datasample(A) represents a random action selected from the action space; Indicates selecting the action corresponding to the maximum Q value; rand t is a random number at time t, ranging from [0,1]; ε is the greedy coefficient.

[0021] Specifically, in the Q-learning reinforcement learning algorithm of step S4, the action value function Q(s t ,a t ) is updated as follows:

[0022]

[0023] Among them, α is the learning rate; γ is the discount factor; r t+1 For the new action a t+1 The reward value obtained after .

[0024] In addition, the present invention also proposes a combined navigation device, comprising:

[0025] memory for storing computer programs;

[0026] The processor is configured to implement the above-mentioned combined navigation method when executing the computer program.

[0027] In addition, the present invention also proposes a computer medium having a computer program stored thereon, and the computer program implements the above-mentioned combined navigation method when executed by a processor.

[0028] The application of the technical solution of the present invention has the following beneficial effects:

[0029] The integrated navigation method proposed in this paper utilizes an adaptive Kalman filter (QKF) algorithm based on Q-learning reinforcement learning for a SINS / GNSS / DVL integrated navigation system. This method adaptively optimizes velocity measurement noise estimation to better adapt to the increased DVL velocity measurement error caused by complex seabed environments, thereby improving the accuracy of integrated navigation. The method can adaptively adjust the velocity measurement noise value based on the external environment, resulting in faster convergence, smaller velocity and position errors, and greater stability.

[0030] In addition to the above-described objects, features and advantages, the present invention has other objects, features and advantages. The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are intended to explain the present invention and do not constitute an undue limitation of the present invention. In the accompanying drawings:

[0032] Figure 1 It is a flowchart of the combined navigation method;

[0033] Figure 2 It is a schematic diagram of coordinate rotation transformation;

[0034] Figure 3 This is a workflow diagram of the combined navigation system applying the preferred combined navigation method of this embodiment;

[0035] Figure 4 This is a schematic diagram of the results of DVL measurement error;

[0036] Figure 5a This is a comparative diagram of the pitch angle errors solved by KF, AKF, and QKF navigation.

[0037] Figure 5b This is a comparative diagram of the roll angle errors solved by KF, AKF, and QKF navigation.

[0038] Figure 5c This is a comparative diagram of the heading angle errors (Heading angle errors) solved by KF, AKF, and QKF navigation;

[0039] Figure 6a This is a comparative diagram of the East velocity errors calculated by KF, AKF, and QKF navigation solutions;

[0040] Figure 6b This is a comparative diagram of the North velocity errors calculated by KF, AKF, and QKF navigation solutions;

[0041] Figure 7a This is a comparative diagram of the East position errors calculated by KF, AKF, and QKF navigation solutions;

[0042] Figure 7b This is a comparative diagram of the North position errors calculated by KF, AKF, and QKF navigation solutions. DETAILED DESCRIPTION

[0043] The embodiments of the present invention are described in detail below with reference to the accompanying drawings. However, the present invention can be implemented in many different ways as defined and covered by the claims.

[0044] Example 1:

[0045] See also Figure 1 This embodiment discloses a combined navigation method based on Q-learning reinforcement learning, the steps are as follows:

[0046] Step S1: defining a coordinate system, specifically, the coordinate system includes an inertial coordinate system, an earth coordinate system, a geographic coordinate system, a navigation coordinate system and a carrier coordinate system;

[0047] Step S2: establishing an inertial navigation system error model. Specifically, establishing an inertial navigation system error model based on the coordinate system in step S1. The inertial navigation system error model includes an attitude error model, a velocity error model, a position error model, and an accelerometer and gyroscope error model.

[0048] Step S3: Establishing a Kalman filter, specifically, fusing the inertial navigation system error model in step S2 with the speed meter information and satellite positioning information to establish an integrated navigation system, and establishing its Kalman filter, the Kalman filter including a state equation, a measurement equation, and a Kalman filter equation;

[0049] Step S4: Obtain optimal velocity measurement noise and perform navigation solution. Specifically, use the Q-learning reinforcement learning algorithm to adaptively update the velocity measurement noise in the integrated navigation system to obtain the optimal velocity measurement noise. The position measurement noise remains unchanged. The optimal velocity measurement noise and position measurement noise are substituted into the Kalman filter in step S3 to perform navigation solution to obtain navigation information, which includes attitude information, velocity information, and position information.

[0050] Specifically, the preferred inertial navigation system of the present embodiment is a strapdown inertial navigation system (SINS), the velocimeter information adopts Doppler velocimeter (DVL) to collect, and the satellite positioning information adopts global navigation satellite system (GNSS) and pseudo-range single point positioning method to obtain, and the integrated navigation system is a strapdown inertial navigation system / global navigation satellite system / Doppler velocimeter integrated navigation system.

[0051] Specifically, in step S1, the definitions of each coordinate system are as follows:

[0052] 1. Inertial coordinate system (i system)

[0053] The origin of the inertial coordinate system is set to the center of the earth, x i The axis points to the vernal equinox, z i Axis along the Earth's rotation axis, y i Axis and x i Axis, z i The axes form a right-handed rectangular coordinate system.

[0054] 2. Earth coordinate system (e system)

[0055] The origin of the Earth coordinate system is set to the center of the Earth, x e axis and y e The axis lies in the equatorial plane, x e The direction of the axis points to the prime meridian, y e The axis passes through the intersection of the 90° east meridian and the equator, z e The axis points toward the Earth's North Pole, along its axis of rotation.

[0056] 3. Geographic coordinate system (G system)

[0057] The geographic coordinate system is also called the Northeast Celestial Coordinate System. Its origin is located at the center of mass of the carrier, x g axis, y g Axis, z g The axes are along the local east, north and celestial directions respectively.

[0058] 4. Navigation coordinate system (n system)

[0059] The navigation coordinate system is an artificially defined reference coordinate system used for navigation solution. During the navigation solution process, the physical quantities of other coordinate systems are projected into the n-system before calculation.

[0060] 5. Carrier coordinate system (b system)

[0061] The origin of the carrier coordinate system is located at the center of mass of the carrier, the x-axis points to the right of the carrier, the y-axis points to the front of the carrier, and the z-axis points to the top of the carrier.

[0062] Furthermore, coordinate systems can be transformed by performing multiple basic rotations. Multiplying the rotation matrices yields a transformation matrix between the coordinate systems. Due to the non-commutativity of finite rotations, different rotation orders yield different transformation matrices. In this embodiment, all transformation matrices between coordinate systems are obtained using the zxy rotation order.

[0063] For example, the n system can be converted to the b system by the following rotation sequence, as Figure 2 As shown:

[0064]

[0065] The n system is transformed into the b system through three basic rotations. The corresponding three transformation matrices are as follows:

[0066]

[0067]

[0068]

[0069] Finally, the posture matrix can be obtained for:

[0070]

[0071]

[0072] Among them, ψ is the heading angle; θ is the pitch angle; γ is the roll angle.

[0073] Furthermore, in step S2, the posture error model is as follows:

[0074]

[0075] Where Φ is the attitude error angle, is the rotation of the navigation coordinate system relative to the inertial coordinate system; Calculate the error for the navigation coordinate system, which consists of two parts: the rotation of the navigation coordinate system caused by the rotation of the earth and the rotation of the navigation coordinate system caused by the curvature of the earth's surface when the system moves on the earth's surface That is ε n is the gyro measurement error.

[0076] Furthermore, in step S2, the speed error model is as follows:

[0077]

[0078] Among them, δv nis the velocity error in the navigation coordinate system, f n Represents the projection of the accelerometer output in the navigation coordinate system, is the accelerometer measurement error.

[0079] Furthermore, in step S2, the position error model is as follows:

[0080]

[0081]

[0082]

[0083] in, Indicates the latitude, longitude, and altitude errors in the navigation coordinate system; L, λ, and h represent latitude, longitude, and altitude, respectively; v e 、v n 、v u Represent the speed in the east, north and sky directions respectively, R M 、R N They are the local meridian curvature radius and the meridian curvature radius respectively.

[0084] Furthermore, in step S2, the accelerometer and gyroscope error models are as follows:

[0085]

[0086]

[0087] Among them, ε b and Represent the bias errors of the gyroscope and accelerometer respectively, which are random constants; w g represents the gyroscope angle random walk error, which is zero-mean Gaussian white noise; w a Represents the accelerometer velocity random walk error, which is zero-mean Gaussian white noise.

[0088] Furthermore, in this embodiment, the state vector X of the integrated navigation system is defined as follows:

[0089]

[0090] Among them, δv e ,δv n ,δv u are respectively the eastward, northward and celestial velocity errors in the navigation coordinate system; Φ x , Φ y , Φ z are the pitch, roll and heading misalignment angles respectively; δL, δλ and δh represent the latitude, longitude and altitude errors respectively; They are the gyro constant drift in the carrier coordinate system, are the accelerometer zero bias in the carrier coordinate system.

[0091] In step S3, according to the error model of the SINS and the error model of the gyro accelerometer in step S2, the state equation The expression can be:

[0092]

[0093] Where F is the state transfer matrix, which is updated by SINS in real time; G is the noise distribution matrix; and W is the process noise.

[0094] Furthermore, assuming that the noise is zero-mean Gaussian white noise, its covariance matrix E(WW T ) is expressed as Q value.

[0095] Furthermore, the specific expression of F is as follows:

[0096]

[0097] in,

[0098]

[0099]

[0100]

[0101]

[0102]

[0103]

[0104]

[0105]

[0106]

[0107] Furthermore, in step S3, the expression of the measurement equation is as follows:

[0108]

[0109] Among them, Z is the observed quantity; v dvl Indicates the speed measurement result of the Doppler odometer; v ins is the speed calculated by the strapdown inertial navigation system; L insand λ ins are the latitude and longitude information settled by the strapdown inertial navigation system; L gnss and λ gnss are the latitude and longitude information provided by the global navigation satellite system. H is the measurement matrix; V represents the measurement noise, which is zero-mean Gaussian white noise and consists of velocity measurement noise and position measurement noise. The velocity measurement noise can be represented by the velocity measurement accuracy of DVL, and the position measurement noise can be represented by the positioning accuracy of GNSS. The covariance matrix E(VV T ) is represented as R.

[0110] Furthermore, H can be written as:

[0111]

[0112] Where I represents the identity matrix.

[0113] In addition, in order to facilitate processing on a computer, this embodiment discretizes the continuous Kalman filter equation. The discretized state equation is as follows:

[0114] X k =ΦX k-1 +W k-1 ;

[0115] Among them, X k-1 is the state at time k-1, X k is the state at time k, Φ is the one-step transfer matrix, T is the sampling time, and this embodiment preferably uses a third-order expansion for calculation.

[0116] The basic equation of the discrete Kalman filter is as follows:

[0117]

[0118] P k|k-1 =ΦP k-1 Φ T +Q k-1

[0119] K k =P k|k-1 H T (HP k|k-1 H T +R k ) -1 ;

[0120]

[0121] P k =(IK k H)P k|k-1

[0122] in, is the optimal estimate of the state at time k-1, is the best one-step prediction at time k, P k-1 is the estimated mean square error matrix at time k-1, Φ T is the transposed matrix of the one-step transfer matrix, Q k-1 is the process noise variance matrix at time k-1, P k|k-1 is the state one-step prediction mean square error matrix, H T is the transpose of the measurement matrix, R k K is the measurement noise variance matrix at time k, k is the filter gain, Z k is the observed quantity at time k, P k is the estimated mean square error matrix at time k.

[0123] Furthermore, in step S4, the steps of the Q-learning reinforcement learning algorithm are as follows:

[0124] Step a: Establish a Q-value table. Specifically, the Q-value table is Q(S,A), where S is the state space, constructed by DVL velocity measurement noise; A is the action space, constructed by the change in DVL velocity measurement noise;

[0125] Step b: Select action a t And execute, specifically, select a change in DVL velocity measurement noise from the action space in step 1 as action a t , perform the action a t ;

[0126] Step c: Obtaining a reward value. Specifically, the reward value is calculated based on action a in step 2. The preferred reward value in this embodiment is the inverse of the absolute value of the position error.

[0127] Step d: Update the Q value table. Specifically, update the Q value table based on the reward value in step 3.

[0128] Furthermore, the preferred Q-learning reinforcement learning algorithm of this embodiment is set as follows: the reinforcement learning environment is set to the SINS / GNSS / DVL integrated navigation system, the agent is Q-learning, the state is the DVL measurement noise, the action is the change in the DVL measurement noise, and the reward value is the inverse of the absolute value of the position error.

[0129] Furthermore, the Q-learning reinforcement learning algorithm is set as follows:

[0130] The reinforcement learning environment is set as a strapdown inertial navigation system / global navigation satellite system / Doppler velocimeter integrated navigation system, the agent is Q-learning, the state is the DVL measurement noise, the action is the change in the DVL measurement noise, and the reward value is the inverse of the absolute value of the position error.

[0131] Furthermore, according to the characteristic that the Kalman filter information fusion of the integrated navigation system requires a certain convergence time, the reinforcement learning step size and reinforcement learning parameters are set as follows:

[0132] The reinforcement learning state space S is the set of DVL velocity measurement noise V; the action space A is the set of DVL velocity measurement noise change δV; the reward value r is the inverse of the absolute value of the position error, that is, r = 1 / |δp|; the Q value table Q(S,A) is established; the action value function Q(s t ,a t ), where a t represents the action at time t, S t Represents the state at time t.

[0133] Furthermore, in the preferred Q-learning reinforcement learning algorithm of this embodiment, the action is updated as follows:

[0134]

[0135] Among them, a t represents the action at time t+1 after the update, and datasample(A) represents a random action selected from the action space; Indicates selecting the action corresponding to the maximum Q value; rand t is a random number at time t, ranging from [0,1]; ε is the greedy coefficient.

[0136] Based on the updating method of the above actions, step b in this embodiment can be implemented.

[0137] Furthermore, in the preferred Q-learning reinforcement learning algorithm of this embodiment, the action value function Q(s t ,a t ) is updated as follows:

[0138]

[0139] Among them, α is the learning rate; γ is the discount factor; r t+1 For the new action a t+1 The reward value obtained after .

[0140] Based on the above-mentioned updating method of the action-value function, step d in this embodiment can be implemented.

[0141] Aiming at the problem that the speed measurement error of the Doppler velocimeter increases in unknown ocean environments, which leads to a decrease in navigation accuracy, the Q-learning reinforcement learning algorithm is used to adaptively update the speed measurement noise of the DVL in the Kalman filter to improve the navigation accuracy of the integrated navigation system in unknown sea areas.

[0142] In addition, this embodiment provides a combined navigation device, including:

[0143] memory for storing computer programs;

[0144] The processor is configured to implement the above-mentioned combined navigation method when executing the computer program.

[0145] In addition, this embodiment provides a computer medium having a computer program stored thereon. When the computer program is executed by a processor, the combined navigation method described above is implemented.

[0146] like Figure 3 As shown, the working principle of applying this embodiment to improve the navigation accuracy of the integrated navigation system in unknown waters is as follows:

[0147] In the first step, the inertial measurement unit (IMU) collects the ship's speed and angle information at a relatively fast sampling frequency and solves the collected inertial navigation information;

[0148] In the second step, when the DVL sampling time or the GNSS sampling time is reached (depending on whichever sampling time is longer, if the DVL sampling time is longer, the DVL sampling time point is selected), the DVL velocity information, GNSS positioning information, and inertial navigation solution information are fused through Kalman filtering.

[0149] In the third step, when the reinforcement learning step is reached, the position error at the final solution moment is calculated using GNSS positioning information as a reference. The reinforcement learning agent then receives a reward from the environment. The agent then updates its state based on the action strategy, adjusts the velocity measurement noise in the Kalman filter, and updates the state-action value function.

[0150] In the fourth step, if the reinforcement learning state values ​​before and after the update are different, the Kalman filter estimation covariance matrix is ​​initialized and the adjusted measurement noise is substituted into the Kalman filter for navigation solution.

[0151] Repeating the third step can obtain the optimal measurement noise after adaptive adjustment, and then use the optimal measurement noise to perform navigation solution in the fourth step, thereby improving navigation accuracy.

[0152] To better illustrate the advantages and purposes of this embodiment, the navigation effects of the Kalman filter (KF), the Sage-Husa adaptive Kalman filter (AKF), and the Kalman filter algorithm (QKF) disclosed in this embodiment will be compared based on simulation results.

[0153] Assume that the ship's speed is 5m / s, the DVL scale factor error is 0.2%, and the measurement noise is 0.002m / s. Due to the influence of the external environment, the DVL speed measurement result deviates from the true speed by 1m / s, and the measurement noise remains unchanged. The simulation time is 1 hour, the instrument simulation parameter settings are shown in Table 1, and the navigation simulation parameters are shown in Table 2. The DVL speed measurement error results are shown in Figure 4 As shown, the vertical axis represents speed error and the horizontal axis represents time.

[0154] Table 1 Simulation parameter settings

[0155]

[0156] Table 2 Navigation simulation parameter settings

[0157] East Northbound Tian Xiang Initial attitude error 3’ 3’ 5’ Initial position error 0.1m / s 0.1m / s 0.1m / s Initial position error 0m 0m 0m longitude latitude high Initial position 112.28° 28.12° 0m

[0158] See Figure 5a 、 Figure 5b as well as Figure 5c , where the vertical axis represents angle error and the horizontal axis represents time. As can be seen from the figure, the pitch angle errors calculated by the three algorithms are similar. Compared to the KF, the AKF and QKF suppress the roll and azimuth errors. In practice, azimuth error is more important, and the AKF and QKF can effectively suppress azimuth divergence.

[0159] See Figure 6a and Figure 6b , where the vertical axis represents the velocity error and the horizontal axis represents the time; see Figure 7a and Figure 7b , where the vertical axis represents position error and the horizontal axis represents time. Figure 6a and Figure 6b It can be seen that compared with ordinary KF, the biased velocity error of AKF and QKF is significantly reduced after the algorithm converges. Figure 7a and Figure 7b In the figure, we can see that the position error of the KF algorithm diverges due to the influence of velocity error. This phenomenon can be understood from the SINS error model: velocity error directly contributes to position error. Velocity error is primarily due to bias error, which causes linear divergence of the position error. The AKF and QKF algorithms can converge the position error.

[0160] Looking at the QKF curve, we can see that in the first few minutes, the QKF agent is constantly exploring, and its action strategy tends to be more random, resulting in large error fluctuations. Once the reward value of the environment feedback reaches the expected set requirements, the agent's action strategy tends to be more exploitative, and the error is smaller and relatively stable. Compared with QKF and AKF, QKF converges faster and has smaller velocity and position errors.

[0161] To more intuitively compare the navigation performance of different algorithms, we quantitatively analyzed the speed and position errors. Table 3 shows the root mean square error (RMSE) between the speed and position solutions calculated by different algorithms and the true values ​​after 10 minutes of navigation.

[0162] Table 3 Root mean square error comparison

[0163]

[0164] As shown in Table 3, the velocity RMS errors of QKF and AKF are 50 and 30 times higher, respectively, than those of KF. The position RMS error of QKF is approximately 3 times higher than that of AKF. Therefore, QKF can effectively calculate velocity and position information despite external interference, improving navigation accuracy.

[0165] This embodiment discloses an integrated navigation method based on Q-learning reinforcement learning. This method implements an adaptive Kalman filter algorithm for integrated navigation based on Q-learning reinforcement learning theory. This algorithm adaptively adjusts velocity measurement noise based on the external environment, improving integrated navigation speed and position accuracy. Simulation results show that the QKF algorithm effectively reduces integrated navigation speed and position errors compared to conventional KF and AKF algorithms.

[0166] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0167] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0168] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A combined navigation method based on Q-learning reinforcement learning, characterized in that: Here are the steps: Step S1: defining a coordinate system, specifically, the coordinate system includes an inertial coordinate system, an earth coordinate system, a geographic coordinate system, a navigation coordinate system and a carrier coordinate system; Step S2: establishing an inertial navigation system error model, specifically: establishing an inertial navigation system error model based on the coordinate system in step S1, the inertial navigation system error model including an attitude error model, a velocity error model, a position error model, and an accelerometer and gyroscope error model; Step S3: Establishing a Kalman filter, specifically: fusing the inertial navigation system error model in step S2 with the speed meter information and satellite positioning information to establish an integrated navigation system, and establishing its Kalman filter, the Kalman filter including a state equation, a measurement equation, and a Kalman filter equation; Step S4: Obtain the optimal speed measurement noise and perform navigation solution. Specifically, the speed measurement noise in the integrated navigation system is adaptively updated using the Q-learning reinforcement learning algorithm to obtain the optimal speed measurement noise. The position measurement noise remains unchanged. The optimal speed measurement noise and position measurement noise are substituted into the Kalman filter in step S3 for navigation solution to obtain navigation information.

2. The integrated navigation method according to claim 1, wherein: In step S1, coordinate systems are transformed between different coordinate systems through multiple basic rotations, and the transformation matrix between the coordinate systems is obtained by multiplying the rotation matrices. The transformation matrix is ​​obtained in the rotation order of zxy.

3. The integrated navigation method according to claim 2, characterized in that: In step S2, the inertial navigation system is a strapdown inertial navigation system; in step S3, the speed meter information is collected using a Doppler speed meter, and the satellite positioning information is collected using a global navigation satellite system and a pseudo-range single point positioning method, and the integrated navigation system is a strapdown inertial navigation system / global navigation satellite system / Doppler speed meter integrated navigation system.

4. The integrated navigation method according to claim 3, characterized in that: In step S4, the Q-learning reinforcement learning algorithm is set as follows: The reinforcement learning environment is set as a strapdown inertial navigation system / global navigation satellite system / Doppler velocimeter integrated navigation system, the agent is Q-learning, the state is the DVL measurement noise, the action is the change in the DVL measurement noise, and the reward value is the inverse of the absolute value of the position error.

5. The integrated navigation method according to claim 4, characterized in that: Set the reinforcement learning step size and parameters as follows: The reinforcement learning state space S is the set of DVL speed measurement noise V; the action space A is the set of DVL speed measurement noise change δV; the reward value r is the inverse of the absolute value of the speed error, that is, r = 1 / |δp|; the Q value table Q(S,A) is established; the action value function Q(s t ,a t ), where a t represents the action at time t, S t Represents the state at time t.

6. The integrated navigation method according to claim 5, characterized in that: In the Q-learning reinforcement learning algorithm in step S4, the action is updated as follows: Among them, a t+1 represents the action at time t+1 after the update, and datasample(A) represents a random action selected from the action space; Indicates selecting the action corresponding to the maximum Q value; rand t is a random number at time t, ranging from [0,1]; ε is the greedy coefficient.

7. The integrated navigation method according to claim 6, characterized in that: In the Q-learning reinforcement learning algorithm of step S4, the action value function Q(s t ,a t ) is updated as follows: Among them, α is the learning rate; γ is the discount factor; r t+1 For the new action a t+1 The reward value obtained after .

8. A combined navigation device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the combined navigation method according to any one of claims 1 to 7 when executing the computer program.

9. A computer medium, characterized in that The computer medium stores a computer program, and when the computer program is executed by a processor, the combined navigation method according to any one of claims 1 to 7 is implemented.

Images