An underwater robot pose estimation method fusing ddpg and ckf

Abstract

The application discloses a kind of underwater robot posture estimation methods of fusion DDPG and CKF, this method first with gyroscope data, four-element state equation is built, accelerometer and magnetometer data build measurement equation;Again, the volume point is obtained using three-order spherical path volume rule, the state and measurement are predicted, and the state and covariance are updated using Kalman gain.Next, CKF passes the innovation, innovation covariance, state and observation cross-covariance to the Actor network of DDPG, and outputs the required Q and R of CKF;Q and R are applied to underwater simulation environment, and the obtained data is stored in experience pool.Subsequently, the Q value is obtained by sampling and training Critic network, the current network is updated by gradient method, the target network is soft-updated, and finally the adaptive Q and R are fed back to CKF, the quaternion is updated, and the attitude angle is calculated, to realize high-precision attitude estimation.

Description

Technical Field

[0001] This invention relates to the field of underwater robots, and in particular to an underwater robot attitude estimation method that integrates DDPG and CKF. Background Technology

[0002] As human exploration of marine resources deepens, autonomous underwater vehicles (AUVs) are increasingly being used in complex marine environments. Attitude calculation is one of the key technologies for AUV's precise navigation and control. Currently, Kalman filter-based algorithms (such as Extended Kalman Filter (EKF), Unscented Kalman Filter (UKF), and Volumetric Kalman Filter (CKF)) are the mainstream methods for attitude calculation.

[0003] However, these methods have an inherent limitation: the process noise covariance matrix (Q) and the measurement noise covariance matrix (R) are usually set to fixed constant values. These values ​​rely on prior knowledge or tedious manual parameter tuning. The operating environment of AUVs is highly uncertain, time-varying, and subject to strong disturbances (such as turbulence and robotic arm motion interference). Fixed noise parameters cannot accurately reflect the dynamically changing process noise and measurement noise characteristics of the system. This leads to a decrease in estimation accuracy, slower convergence speed, and even potential divergence of traditional Kalman filters during sudden environmental changes, severely restricting the autonomy and reliability of AUVs in complex underwater environments. Summary of the Invention

[0004] To address the aforementioned shortcomings, the present invention aims to propose an AUV adaptive attitude estimation method based on the Deep Deterministic Policy Gradient (DDPG) algorithm fused with Cumulative Kalman Filter (CKF), which can improve the accuracy and robustness of attitude calculation and meet the demand for high-precision attitude estimation in complex environments.

[0005] To achieve this objective, the present invention adopts the following technical solution:

[0006] An underwater robot attitude estimation method integrating DDPG and CKF includes the following steps:

[0007] S100. Establish state equations using gyroscope data and quaternions, and establish measurement equations using acceleration and magnetometer data.

[0008] S200: The third-order spherical phase diameter volume rule is used to obtain the volume point and predict the state and measurement.

[0009] S300, Calculate the Kalman gain and update the state and covariance;

[0010] S400 provides information through a capacitive Kalman filter algorithm. New information covariance and the cross-covariance between state and observation Given the current actor network and the target actor network in the deep deterministic policy gradient algorithm, output actions respectively. and Here, the actions are Q and R of CKF, the volumetric Kalman filter algorithm is CKF, and the depth deterministic policy gradient algorithm is DDPG;

[0011] S500, output action The current state is obtained by acting on the underwater simulation environment. Reward value And the state in the next moment. These values ​​will be stored in the experience pool;

[0012] S600, sample a small portion of the data from the experience pool, and The input is fed into the current critic network for training to obtain the Q-value. and Input into the target critic network to obtain ;

[0013] S700: Update the current commentator network using gradient descent and update the current actor network using gradient ascent.

[0014] S800: Use a soft update method to slowly update the target actor network and the target commentator network;

[0015] S900: The adaptive process noise covariance matrix Q and measurement noise covariance matrix R obtained from training are fed back to CKF to update the state. The updated quaternion is extracted from the state vector and then the attitude is calculated to obtain the attitude angle.

[0016] Preferably, in step S100:

[0017] The attitude quaternion of the underwater robot and the three-axis random drift error of the gyroscope are selected as the state variables of the state equation, which are expressed as follows:

[0018] Equation (1)

[0019] Let the unit quaternion be . The three-axis random drift error of the gyroscope is = , , ;

[0020] Then, according to the Runge-Kutta method, the quaternion equation is constructed as follows:

[0021] Equation (2)

[0022] Where I is the identity matrix, This indicates the time step, also known as the sampling time, which is the time interval between two consecutive time points. It is the angular velocity matrix of the body coordinate system b relative to the carrier coordinate system n, used for the conversion between angular velocity and quaternions;

[0023] Based on the state variables and quaternion equations, the state equations of the system can be constructed as follows:

[0024] Equation (3)

[0025] in The driving matrix for system noise, Let the process noise of the system be... The covariance matrix is ​​Q;

[0026] Using data from the Accelerometer and Magnetometer as observations, the system's measurement equations are established, and the angular velocity measured by the Gyro gyroscope is corrected. The measurement equations are as follows:

[0027] Equation (4)

[0028] Where h is a measurement function that describes the state. The relationship between and the ideal measured value (noise-free) Z. For the measurement noise of the system, The covariance matrix is ​​R;

[0029] Then, the CKF algorithm is used for updates, including state updates, measurement updates, error covariance matrix updates, and noise covariance matrix updates.

[0030] Furthermore, in step S200, the steps for predicting the state and measurement are as follows:

[0031] CKF uses the third-order spherical phase diameter volume rule to obtain 2n volume points. The CKF algorithm flow is as follows:

[0032] S210, Initialization parameters:

[0033] Equation (5)

[0034] In the formula, These are the initial values ​​of the state variables that were manually set at the beginning. E is the expected value, but initially it was assumed that the AUV was placed horizontally and that the gyroscope did not have zero bias. Set as , It is the initial error covariance, representing the initial guess. The degree of uncertainty; it is a diagonal matrix, where the values ​​on the diagonal represent the variance of each state variable; let... As shown below:

[0035]

[0036] in, = , =0.1;

[0037] S220 calculates the volume point of the state equation:

[0038] Given the state at time k The statistical properties of N( ; , Find the volume point:

[0039] = Equation (6)

[0040] In the formula, m = 2n. It is the average value from the start time to time k. covariance matrix It is the transformed volume point. It is a square root matrix used to generate volume points. It is a unit spherical point, selected according to the third-order spherical-phase volume rule in the CKF algorithm, which can ensure that it conforms to a normal distribution after nonlinearization;

[0041] The volume point is transferred through the state nonlinear function as follows:

[0042] Equation (7)

[0043] In the formula, f is the state transfer function, used to transfer the volume point Pass it to the next time step. These are newly generated volume points;

[0044] Step S200 also includes S230, which calculates the estimated value and estimated covariance of the one-step state prediction:

[0045] Equation (8)

[0046] In the formula, It is an estimate of the state prediction in one step;

[0047] Equation (9)

[0048] In the formula, It is a one-step estimation of covariance, where Q is the process noise;

[0049] S240, Calculate the volume point of the measurement equation:

[0050] Estimating the variance of state prediction By performing Cholesky matrix decomposition, we can obtain: = The corresponding volume points can be calculated as follows:

[0051] + , Equation (10)

[0052] In the formula, This is the newly generated volume point for measuring the equation;

[0053] The newly generated volume point is obtained through a nonlinear measurement function:

[0054] Equation (11)

[0055] In the formula, h is the measurement transfer function. These are newly generated volume points;

[0056] S250 calculates the one-step measurement prediction, prediction variance, and cross-covariance:

[0057] Equation (12)

[0058] +R formula (13)

[0059] Equation (14)

[0060] In the formula, It is a measurement-based prediction. It is the new information covariance. It is the cross-variance between state and prediction.

[0061] Further, step S300 includes step S310, which calculates the Kalman gain:

[0062] Equation (15)

[0063] in, It is the Kalman filter gain;

[0064] It also includes step S320, which updates the status and cooperative defense difference;

[0065] Equation (16)

[0066] Equation (17)

[0067] In the formula, It is the optimal estimate at the current moment. It is the current covariance matrix. It's new information.

[0068] Further, step S400 includes step S410, obtaining the action:

[0069] Step 410, the new information provided in step S300 New information covariance and the cross-covariance between state and observation Inputs are fed into the current actor network and the target actor network to obtain actions. and Here, the actions are Q and R in CKF. The current actor network is the current actor network, which adopts a recurrent neural network architecture, i.e., RNN. The model takes sequence data as input, and then passes it through a hidden layer. The sequence data is preprocessed through a fully connected layer. The fully connected layer performs linear transformations and non-linear mappings on the features of each time step in the input sequence, thereby projecting them into a higher-dimensional feature space to extract more abstract feature representations.

[0070] Subsequently, the feature sequence is input into the Long Short-Term Memory (LSTM) unit. The LSTM implements a gating mechanism through the tanh and sigmoid activation functions. To further enhance the network's expressive power, the ReLU activation function is introduced after its output.

[0071] The processed features are then fed into a second fully connected layer, which integrates and abstracts the temporal features extracted by the L unit, further fusing high-level semantic information to prepare for the final output. The last fully connected network uses the sigmoid activation function to map the high-dimensional features to the target output dimension, and finally generates continuous action value outputs through a linear output layer.

[0072] Preferably, step 500 includes step 510;

[0073] The output action in step 400 The current state is obtained by acting on the underwater simulation environment. Reward value And the state in the next moment. These values ​​are then stored in the experience pool.

[0074] Ideally, the Q-value prediction Q( , ) and Q-value estimation ( , ):

[0075] A portion of samples is taken from the experience pool and input into the critic network for training. The steps in 400 are then... and The input is fed into the current critic network to obtain the Q-value prediction Q( , ), in step 400 The input is fed into the target critic network to obtain the Q-value estimate. ( , ), in calculating the target Q value:

[0076] ( , Equation (18)

[0077] In the formula It is the reward value in step 500. It is a discount factor. ( , ) is the Q-value estimate; α ranges from 0 to 1, with an optimal value of 0.98;

[0078] The critic network employs a two-branch input structure, receiving state sequences and action sequences as inputs respectively;

[0079] First, each input undergoes feature transformation and information fusion through a fully connected layer. This process uses the ReLU activation function to introduce nonlinearity, thereby enhancing the network's ability to express complex feature patterns.

[0080] Subsequently, the fused feature sequence is input into the Long Short-Term Memory (LSTM) network layer, which can effectively capture long-term dependencies in sequence data; LSTM maintains the memory of historical information through its gating mechanism and uses contextual information to enhance the representation of the current state;

[0081] Finally, the network generates an estimate of the state-action value function, i.e., the Q-function, through the output layer. The output is a scalar value used to evaluate the expected cumulative reward that can be obtained after performing the corresponding action in a given state.

[0082] Furthermore, update the current actor network and critic network:

[0083] First, we use the method of minimizing the mean square error to minimize the mean square error between the current Q value and the target Q value. The formula is:

[0084] Equation (19)

[0085] In the formula, N is the number of samples taken from the empirical replay pool. It is the target Q value. This is the current state. This is the current action. These are the parameters of the current CRT network.

[0086] Then, the gradient descent algorithm is used to minimize this loss function L to update the parameters of the current commentator network. Finally, the gradient ascent method is used to update the current actor network parameters. ,

[0087] Equation (20)

[0088] In the formula, N is the number of samples taken from the empirical replay pool. It is the gradient of the Q-value with respect to the action. This is the current state. This is the current action. These are the parameters of the current CRT network. It is the gradient of the action with respect to the network parameters of the actor. These are the parameters of the current actor network.

[0089] Furthermore, soft update target actor networks and target commentator networks:

[0090] The target actor network and the target commentator network are updated regularly using soft updates.

[0091] Equation (21)

[0092] Equation (22)

[0093] In the formula It is a soft update coefficient. These are the current critic network parameters. These are the parameters of the target critic network. These are the current actor network parameters. These are the network parameters of the target actor.

[0094] Furthermore, step S900 specifically includes:

[0095] Finally, the adaptive process noise covariance matrix Q and measurement noise covariance matrix R obtained from training are fed back to CKF to update the quaternions, thus obtaining the accurate attitude angles. The specific formula is as follows:

[0096] Equation (23)

[0097] In the formula, It is the pitch angle. It's the roll angle. It is the yaw angle.

[0098] One of the above technical solutions includes the following beneficial effects: a closed-loop interactive system between DDPG and CKF is constructed, wherein the output of CKF serves as the state input of DDPG, and the output of DDPG (adjusted Q, R) is fed back to CKF for the next filtering prediction and update, thereby forming a high-precision and highly robust attitude calculation scheme that can adapt to environmental changes. Attached Figure Description

[0099] Figure 1 This is a schematic diagram of the overall framework of the present invention;

[0100] Figure 2 This is an execution flowchart of the Deep Deterministic Policy Gradient Algorithm (DDPG) of this invention;

[0101] Figure 3 This is a schematic diagram of the actor structure of the Deep Deterministic Policy Gradient Algorithm (DDPG) of this invention;

[0102] Figure 4 This is a schematic diagram of the structure of the Deep Deterministic Policy Gradient Algorithm (DDPG) of this invention. Detailed Implementation

[0103] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0104] like Figure 1 As shown, an underwater robot attitude estimation method integrating DDPG and CKF includes the following steps:

[0105] S100. Establish state equations using gyroscope data and quaternions, and establish measurement equations using acceleration and magnetometer data.

[0106] S200: The third-order spherical phase diameter volume rule is used to obtain the volume point and predict the state and measurement.

[0107] S300, Calculate the Kalman gain and update the state and covariance;

[0108] S400 provides information through a capacitive Kalman filter algorithm. New information covariance and the cross-covariance between state and observation Given the current actor network and the target actor network in the deep deterministic policy gradient algorithm, output actions respectively. and Here, the actions are Q and R of CKF, the volumetric Kalman filter algorithm is CKF, and the depth deterministic policy gradient algorithm is DDPG;

[0109] S500, output action The current state is obtained by acting on the underwater simulation environment. Reward value And the state in the next moment. These values ​​will be stored in the experience pool;

[0110] S600, sample a small portion of the data from the experience pool, and The input is fed into the current critic network for training to obtain the Q-value. and Input into the target critic network to obtain ;

[0111] S700: Update the current commentator network using gradient descent and update the current actor network using gradient ascent.

[0112] S800: Use a soft update method to slowly update the target actor network and the target commentator network;

[0113] S900: The adaptive process noise covariance matrix Q and measurement noise covariance matrix R obtained from training are fed back to CKF to update the state. The updated quaternion is extracted from the state vector and then the attitude is calculated to obtain the attitude angle.

[0114] Beneficial effects: By utilizing the DDPG reinforcement learning algorithm, the process noise covariance matrix Q and measurement noise covariance matrix R in CKF are adjusted online autonomously based on information such as real-time innovation, innovation covariance, and state-observation cross-covariance during the filtering process, without the need for manually setting fixed values. The values ​​of Q and R are adjusted autonomously once in each filtering cycle. Q has a dimension of 7 and a value range of greater than 0, while R has a dimension of 6 and a value range of greater than 0.

[0115] A closed-loop interactive system between DDPG and CKF was constructed, in which the output of CKF serves as the state input of DDPG, and the output of DDPG (adjusted Q, R) is fed back to CKF for the next filtering prediction and update, thus forming a high-precision and robust attitude calculation scheme that can adapt to environmental changes.

[0116] This paper presents an adaptive state estimation algorithm that integrates DDPG (Distributed Data Generation Programming) with Cumulative Kalman Filtering (CKF). By constructing a DDPG network, the state estimator can autonomously learn and update Q and R based on observed data, environmental changes, or system failures. This adaptability enhances the performance of CKF, enabling it to better handle uncertainties and dynamic conditions.

[0117] In addition, in step S100:

[0118] The attitude quaternion of the underwater robot and the three-axis random drift error of the gyroscope are selected as the state variables of the state equation, which are expressed as follows:

[0119] Equation (1)

[0120] Let the unit quaternion be . The three-axis random drift error of the gyroscope is = , , ;

[0121] Then, according to the Runge-Kutta method, the quaternion equation is constructed as follows:

[0122] Equation (2)

[0123] Where I is the identity matrix, This indicates the time step, also known as the sampling time, which is the time interval between two consecutive time points. It is the angular velocity matrix of the body coordinate system b relative to the carrier coordinate system n, used for the conversion between angular velocity and quaternions;

[0124] Based on the state variables and quaternion equations, the state equations of the system can be constructed as follows:

[0125] Equation (3)

[0126] in The driving matrix for system noise, Let the process noise of the system be... The covariance matrix is ​​Q;

[0127] Using data from the Accelerometer and Magnetometer as observations, the system's measurement equations are established, and the angular velocity measured by the Gyro gyroscope is corrected. The measurement equations are as follows:

[0128] Equation (4)

[0129] Where h is a measurement function that describes the state. The relationship between and the ideal measured value (noise-free) Z. For the measurement noise of the system, The covariance matrix is ​​R;

[0130] Then, the CKF algorithm is used for updates, including state updates, measurement updates, error covariance matrix updates, and noise covariance matrix updates.

[0131] Beneficial effects: Constructing state equations using quaternions can avoid gimbal lock problems. By integrating quaternion state equations with magnetometer / accelerometer measurement equations, we can not only utilize the complementarity of multiple sensors to improve the accuracy of attitude calculation, but also reduce dependence on a single sensor, thereby improving the reliability and availability of the system.

[0132] Furthermore, in step S200, the steps for predicting the state and measurement are as follows:

[0133] CKF uses the third-order spherical phase diameter volume rule to obtain 2n volume points. The CKF algorithm flow is as follows:

[0134] S210, Initialization parameters:

[0135] Equation (5)

[0136] In the formula, These are the initial values ​​of the state variables that were manually set at the beginning. E is the expected value, but initially it was assumed that the AUV was placed horizontally and that the gyroscope did not have zero bias. Set as , It is the initial error covariance, representing the initial guess. The degree of uncertainty; it is a diagonal matrix, where the values ​​on the diagonal represent the variance of each state variable; let... As shown below:

[0137]

[0138] in, = , =0.1;

[0139] S220 calculates the volume point of the state equation:

[0140] Given the state at time k The statistical properties of N( ; , Find the volume point:

[0141] = Equation (6)

[0142] In the formula, m = 2n. It is the average value from the start time to time k. covariance matrix It is the transformed volume point. It is a square root matrix used to generate volume points. It is a unit spherical point, selected according to the third-order spherical-phase volume rule in the CKF algorithm, which can ensure that it conforms to a normal distribution after nonlinearization;

[0143] The volume point is transferred through the state nonlinear function as follows:

[0144] Equation (7)

[0145] In the formula, f is the state transfer function, used to transfer the volume point Pass it to the next time step. These are newly generated volume points;

[0146] Step S200 also includes S230, which calculates the estimated value and estimated covariance of the one-step state prediction:

[0147] Equation (8)

[0148] In the formula, It is an estimate of the state prediction in one step;

[0149] Equation (9)

[0150] In the formula, It is a one-step estimation of covariance, where Q is the process noise;

[0151] S240, Calculate the volume point of the measurement equation:

[0152] Estimating the variance of state prediction By performing Cholesky matrix decomposition, we can obtain: = The corresponding volume points can be calculated as follows:

[0153] + , Equation (10)

[0154] In the formula, This is the newly generated volume point for measuring the equation;

[0155] The newly generated volume point is obtained through a nonlinear measurement function:

[0156] Equation (11)

[0157] In the formula, h is the measurement transfer function. These are newly generated volume points;

[0158] S250 calculates the one-step measurement prediction, prediction variance, and cross-covariance:

[0159] Equation (12)

[0160] +R formula (13)

[0161] Equation (14)

[0162] In the formula, It is a measurement-based prediction. It is the new information covariance. It is the cross-variance between state and prediction.

[0163] Beneficial Effects: This study further supplements the overall scheme by proposing a third-order spherical phase diameter volume rule and the CKF algorithm. It addresses four key issues in underwater robot attitude estimation: nonlinear modeling error, balance between real-time performance and accuracy, noise suppression, and filter stability. The third-order volume rule accurately captures nonlinear characteristics, reducing attitude estimation bias. The 2n volume points control computational load, adapting to embedded hardware. Complete covariance updates and reasonable initialization prevent filter divergence. Ultimately, this provides high-precision, highly stable, and low-latency attitude data support for the autonomous navigation and operational control of underwater robots, enhancing their adaptability in complex underwater environments. = , =0.1 is set based on engineering practice experience.

[0164] Furthermore, step S300 includes step S310, which calculates the Kalman gain:

[0165] Equation (15)

[0166] in, It is the Kalman filter gain;

[0167] It also includes step S320, which updates the status and cooperative defense difference;

[0168] Equation (16)

[0169] Equation (17)

[0170] In the formula, It is the optimal estimate at the current moment. It is the current covariance matrix. It's new information.

[0171] Beneficial effects: Through a closed-loop process of dynamic gain allocation, innovation correction, and covariance convergence, the accuracy, adaptability, and stability of underwater robot attitude estimation are achieved.

[0172] like Figure 2 As shown, step S400 includes step S410, obtaining the action:

[0173] Step 410, the new information provided in step S300 New information covariance and the cross-covariance between state and observation Inputs are fed into the current actor network and the target actor network to obtain the action u( and ( Here, the actions are Q and R in CKF, and the current actor network is the current actor network. The actor network uses a recurrent neural network architecture, such as... Figure 3 As shown, this is an RNN; the model takes sequence data as input, and then passes it through a hidden layer. The sequence data is preprocessed through a fully connected layer; the fully connected layer performs linear transformations and non-linear mappings on the features of each time step in the input sequence, thereby projecting them into a higher-dimensional feature space to extract more abstract feature representations.

[0174] Subsequently, the feature sequence is input into the Long Short-Term Memory (LSTM) unit. The LSTM implements a gating mechanism through the tanh and sigmoid activation functions. To further enhance the network's expressive power, the ReLU activation function is introduced after its output.

[0175] The processed features are then fed into a second fully connected layer, which integrates and abstracts the temporal features extracted by the L unit, further fusing high-level semantic information to prepare for the final output. The last fully connected network uses the sigmoid activation function to map the high-dimensional features to the target output dimension, and finally generates continuous action value outputs through a linear output layer.

[0176] The hidden layer dimension of this scheme is set to 128 neurons. This design ensures that the model can capture complex feature representations while effectively controlling the parameter scale, which helps to suppress overfitting. Long short-term memory units are the core components for processing time dependencies and can effectively model long and short-term dependencies in sequences.

[0177] Step 500 includes step 510;

[0178] The output action in step 400 The current state is obtained by acting on the underwater simulation environment. Reward value And the state in the next moment. These values ​​are then stored in the experience pool.

[0179] By storing and reusing the agent's interaction experience with the environment (states, actions, rewards, new states), the temporal correlation between data is broken, greatly improving data utilization efficiency and algorithm stability. This provides a stable and efficient source of learning samples for the DDPG agent, ensuring the convergence of the actor network and evaluation network training, and ultimately improving the overall learning effect and reliability of the underwater robot posture estimation model.

[0180] In addition, the Q-value prediction Q( , ) and Q-value estimation ( , ):

[0181] A portion of samples is taken from the experience pool and input into the critic network for training. The steps in 400 are then... and u( The input is fed into the current critic network to obtain the Q-value prediction Q( , ), in step 400 The input is fed into the target critic network to obtain the Q-value estimate. ( , ), in calculating the target Q value:

[0182] ( , Equation (18)

[0183] In the formula It is the reward value in step 500. It is a discount factor. ( , ) is the Q-value estimate; α ranges from 0 to 1, with an optimal value of 0.98;

[0184] like Figure 4 As shown, the critic network uses a two-branch input structure, receiving state sequences and action sequences as inputs respectively;

[0185] First, each input undergoes feature transformation and information fusion through a fully connected layer. This process uses the ReLU activation function to introduce nonlinearity, thereby enhancing the network's ability to express complex feature patterns.

[0186] Subsequently, the fused feature sequence is input into the Long Short-Term Memory (LSTM) network layer, which can effectively capture long-term dependencies in sequence data; LSTM maintains the memory of historical information through its gating mechanism and uses contextual information to enhance the representation of the current state;

[0187] Finally, the network generates an estimate of the state-action value function, i.e., the Q-function, through the output layer. The output is a scalar value used to evaluate the expected cumulative reward that can be obtained after performing the corresponding action in a given state.

[0188] Beneficial Effects: Efficient and stable value function learning was achieved through an experience replay mechanism and a specially designed critic network structure. First, training by randomly sampling from the experience pool breaks the temporal correlation between data, improving data utilization and reducing training variance. Second, the critic network structure employing dual-branch input and LSTM layers can process state and action information separately, effectively capturing long-term temporal dependencies in the system. This allows for a more accurate evaluation of the value of state-action pairs, providing a reliable target Q-value signal for policy optimization in the actor network.

[0189] Furthermore, update the current actor network and critic network:

[0190] First, we use the method of minimizing the mean square error to minimize the mean square error between the current Q value and the target Q value. The formula is:

[0191] Equation (19)

[0192] In the formula, N is the number of samples taken from the empirical replay pool. It is the target Q value. This is the current state. This is the current action. These are the parameters of the current CRT network.

[0193] Then, the gradient descent algorithm is used to minimize this loss function L to update the parameters of the current commentator network. Finally, the gradient ascent method is used to update the current actor network parameters. ,

[0194] Equation (20)

[0195] In the formula, N is the number of samples taken from the empirical replay pool. It is the gradient of the Q-value with respect to the action. This is the current state. This is the current action. These are the parameters of the current CRT network. It is the gradient of the action with respect to the network parameters of the actor. These are the parameters of the current actor network.

[0196] Beneficial effects: The loss function design of the critic network enables it to accurately learn and approximate the true value function, while the update of the actor network utilizes the gradient of this value function to optimize the policy in the direction of improving the Q-value. This closed-loop process of "evaluation-improvement" transforms the policy search problem in a high-dimensional continuous action space into an efficient gradient optimization problem, significantly improving the efficiency and stability of policy learning.

[0197] Furthermore, soft update target actor networks and target commentator networks:

[0198] The target actor network and the target commentator network are updated regularly using soft updates.

[0199] Equation (21)

[0200] Equation (22)

[0201] In the formula It is a soft update coefficient. These are the current critic network parameters. These are the parameters of the target critic network. These are the current actor network parameters. These are the network parameters of the target actor.

[0202] Beneficial effects: By introducing a small update coefficient τ, the parameters of the target actor network and the target critic network change slowly and smoothly, thus greatly stabilizing the training process. This mechanism avoids training oscillations or divergences caused by rapid changes in the target value (Q-value estimate) with the current network, providing a relatively fixed reference point for calculating the target Q-value and ensuring reliable convergence of the entire learning system.

[0203] Furthermore, step S900 specifically includes:

[0204] Finally, the adaptive process noise covariance matrix Q and measurement noise covariance matrix R obtained from training are fed back to CKF to update the quaternions, thus obtaining the accurate attitude angles. The specific formula is as follows:

[0205] Equation (23)

[0206] In the formula, It is the pitch angle. It's the roll angle. It is the yaw angle.

[0207] The technical principles of the present invention have been described above with reference to specific embodiments. These descriptions are merely for explaining the principles of the invention and should not be construed as limiting the scope of protection of the invention in any way. Based on this explanation, those skilled in the art can readily conceive of other specific embodiments of the invention without inventive effort, and these embodiments will all fall within the scope of protection of the present invention.

Claims

1. A method for underwater robot attitude estimation that integrates DDPG and CKF, characterized in that, Includes the following steps: S100. Establish state equations using gyroscope data and quaternions, and establish measurement equations using acceleration and magnetometer data. S200: The third-order spherical phase diameter volume rule is used to obtain the volume point and predict the state and measurement. S300, Calculate the Kalman gain and update the state and covariance; S400 provides information through a capacitive Kalman filter algorithm. New information covariance and the cross-covariance between state and observation Given the current actor network and the target actor network in the deep deterministic policy gradient algorithm, output actions respectively. and Here, the actions are Q and R of CKF, the volumetric Kalman filter algorithm is CKF, and the depth deterministic policy gradient algorithm is DDPG; S500, output action The current state is obtained by acting on the underwater simulation environment. Reward value And the state in the next moment. These values ​​will be stored in the experience pool; S600, sample a small portion of the data from the experience pool, and The input is fed into the current critic network for training to obtain the Q-value. and ( Input into the target critic network to obtain ; S700: Update the current commentator network using gradient descent and update the current actor network using gradient ascent. S800: Use a soft update method to slowly update the target actor network and the target commentator network; S900: The adaptive process noise covariance matrix Q and measurement noise covariance matrix R obtained from training are fed back to CKF to update the state. The updated quaternion is extracted from the state vector and then the attitude is calculated to obtain the attitude angle.

2. The underwater robot attitude estimation method integrating DDPG and CKF according to claim 1, characterized in that, In step S100: The attitude quaternion of the underwater robot and the three-axis random drift error of the gyroscope are selected as the state variables of the state equation, which are expressed as follows: Equation (1) Let the unit quaternion be . The three-axis random drift error of the gyroscope is = , , ; Then, according to the Runge-Kutta method, the quaternion equation is constructed as follows: Equation (2) Where I is the identity matrix, This indicates the time step, also known as the sampling time, which is the time interval between two consecutive time points. It is the angular velocity matrix of the body coordinate system b relative to the carrier coordinate system n, used for the conversion between angular velocity and quaternions; Based on the state variables and quaternion equations, the state equations of the system can be constructed as follows: Equation (3) in The driving matrix for system noise, Let the process noise of the system be... The covariance matrix is ​​Q; Using data from the Accelerometer and Magnetometer as observations, the system's measurement equations are established, and the angular velocity measured by the Gyro gyroscope is corrected. The measurement equations are as follows: Equation (4) Where h is a measurement function that describes the state. The relationship between and the ideal measured value (noise-free) Z. For the measurement noise of the system, The covariance matrix is ​​R; Then, the CKF algorithm is used for updates, including state updates, measurement updates, error covariance matrix updates, and noise covariance matrix updates.

3. The underwater robot attitude estimation method integrating DDPG and CKF according to claim 2, characterized in that, In step S200, the steps for predicting the state and measurement are as follows: CKF uses the third-order spherical phase diameter volume rule to obtain 2n volume points. The CKF algorithm flow is as follows: S210, Initialization parameters: Equation (5) In the formula, These are the initial values ​​of the state variables that were manually set at the beginning. E is the expected value, but initially it was assumed that the AUV was placed horizontally and that the gyroscope did not have zero bias. Set as , It is the initial error covariance, representing the initial guess. The degree of uncertainty; it is a diagonal matrix, where the values ​​on the diagonal represent the variance of each state variable; let... As shown below: ； in, = , =0.1; S220 calculates the volume point of the state equation: Given the state at time k The statistical properties of N( ; , Find the volume point: = Equation (6) In the formula, m = 2n. It is the average value from the start time to time k. covariance matrix It is the transformed volume point. It is a square root matrix used to generate volume points. It is a unit spherical point, selected according to the third-order spherical-phase volume rule in the CKF algorithm, which can ensure that it conforms to a normal distribution after nonlinearization; The volume point is transferred through the state nonlinear function as follows: Equation (7) In the formula, f is the state transfer function, used to transfer the volume point Pass it to the next time step. These are newly generated volume points; Step S200 also includes S230, which calculates the estimated value and estimated covariance of the one-step state prediction: Equation (8) In the formula, It is an estimate of the state prediction in one step; Equation (9) In the formula, It is a one-step estimation of covariance, where Q is the process noise; S240, Calculate the volume point of the measurement equation: Estimating the variance of state prediction By performing Cholesky matrix decomposition, we can obtain: = The corresponding volume points can be calculated as follows: + , Equation (10) In the formula, This is the newly generated volume point for measuring the equation; The newly generated volume point is obtained through a nonlinear measurement function: Equation (11) In the formula, h is the measurement transfer function. These are newly generated volume points; S250 calculates the one-step measurement prediction, prediction variance, and cross-covariance: Equation (12) +R formula (13) Equation (14) In the formula, It is a measurement-based prediction. It is the new information covariance. It is the cross-variance between state and prediction.

4. The underwater robot attitude estimation method integrating DDPG and CKF according to claim 3, characterized in that, Step S300 includes step S310, which calculates the Kalman gain: Equation (15) in, It is the Kalman filter gain; It also includes step S320, which updates the status and cooperative defense difference; Equation (16) Equation (17) In the formula, It is the optimal estimate at the current moment. It is the current covariance matrix. It's new information.

5. The underwater robot attitude estimation method integrating DDPG and CKF according to claim 4, characterized in that, Step S400 includes step S410, obtaining the action: Step 410, the new information provided in step S300 New information covariance and the cross-covariance between state and observation Inputs are fed into the current actor network and the target actor network to obtain the action u( and ( Here, the actions are Q and R in CKF. The current actor network is the current actor network, which adopts a recurrent neural network architecture, i.e., RNN. The model takes sequence data as input, and then passes it through a hidden layer. The sequence data is preprocessed through a fully connected layer. The fully connected layer performs linear transformations and non-linear mappings on the features of each time step in the input sequence, thereby projecting them into a higher-dimensional feature space to extract more abstract feature representations. Subsequently, the feature sequence is input into the Long Short-Term Memory (LSTM) unit. The LSTM implements a gating mechanism through the tanh and sigmoid activation functions. To further enhance the network's expressive power, the ReLU activation function is introduced after its output. The processed features are then fed into a second fully connected layer, which integrates and abstracts the temporal features extracted by the L unit, further fusing high-level semantic information to prepare for the final output. The last fully connected network uses the sigmoid activation function to map the high-dimensional features to the target output dimension, and finally generates continuous action value outputs through a linear output layer.

6. The underwater robot attitude estimation method integrating DDPG and CKF according to claim 5, characterized in that, Step 500 includes step 510; The output action u in step 400 ( The current state is obtained by acting on the underwater simulation environment. Reward value And the state in the next moment. These values ​​are then stored in the experience pool.

7. The underwater robot attitude estimation method integrating DDPG and CKF according to claim 6, characterized in that, Calculate the Q-value prediction Q( , ) and Q-value estimation ( , ): A portion of samples is taken from the experience pool and input into the critic network for training. The steps in 400 are then... and u( The input is fed into the current critic network to obtain the Q-value prediction Q( , ), in step 400 The input is fed into the target critic network to obtain the Q-value estimate. ( , ), in calculating the target Q value: ( , Equation (18) In the formula It is the reward value in step 500. It is a discount factor. ( , ) is the Q-value estimate; α ranges from 0 to 1, with an optimal value of 0.98; The critic network employs a two-branch input structure, receiving state sequences and action sequences as inputs respectively; First, each input undergoes feature transformation and information fusion through a fully connected layer. This process uses the ReLU activation function to introduce nonlinearity, thereby enhancing the network's ability to express complex feature patterns. Subsequently, the fused feature sequence is input into the Long Short-Term Memory (LSTM) network layer, which can effectively capture long-term dependencies in sequence data; LSTM maintains the memory of historical information through its gating mechanism and uses contextual information to enhance the representation of the current state; Finally, the network generates an estimate of the state-action value function, i.e., the Q-function, through the output layer. The output is a scalar value used to evaluate the expected cumulative reward that can be obtained after performing the corresponding action in a given state.

8. The underwater robot attitude estimation method integrating DDPG and CKF according to claim 7, characterized in that, Update the current actor network and commentator network: First, we use the method of minimizing the mean square error to minimize the mean square error between the current Q value and the target Q value. The formula is: Equation (19) In the formula, N is the number of samples taken from the empirical replay pool. It is the target Q value. This is the current state. This is the current action. These are the parameters of the current CRT network; Then, the gradient descent algorithm is used to minimize this loss function L to update the parameters of the current commentator network. Finally, the gradient ascent method is used to update the current actor network parameters. , Equation (20) In the formula, N is the number of samples taken from the empirical replay pool. It is the gradient of the Q-value with respect to the action. This is the current state. This is the current action. These are the parameters of the current CRT network. It is the gradient of the action with respect to the network parameters of the actor. These are the parameters of the current actor network.

9. The underwater robot attitude estimation method integrating DDPG and CKF according to claim 8, characterized in that, Soft update target actor network and target commentator network: The target actor network and the target commentator network are updated regularly using soft updates. Equation (21) Equation (22) In the formula It is a soft update coefficient. These are the current critic network parameters. These are the parameters of the target critic network. These are the current actor network parameters. These are the network parameters of the target actor.

10. The underwater robot attitude estimation method integrating DDPG and CKF according to claim 9, characterized in that, Step S900 is as follows: Finally, the adaptive process noise covariance matrix Q and measurement noise covariance matrix R obtained from training are fed back to CKF to update the quaternions, thus obtaining the accurate attitude angles. The specific formula is as follows: Equation (18) In the formula, It is the pitch angle. It's the roll angle. It is the yaw angle.

Images