Self-adaptive control method and device for underwater mechanical arm

Abstract

The invention provides a self-adaptive control method and device for an underwater mechanical arm, and relates to the technical field of reinforcement learning. Inputting the current state and the target motion trail into the trained adaptive control model to obtain a first control parameter output by the trained adaptive control model; the first control parameter comprises a first control torque and a first angular velocity direction; wherein the trained adaptive control model is constructed on the basis of a Q-learning algorithm; and finally, based on the first control parameter, the underwater mechanical arm is controlled to perform tracking motion on the target motion trail, and a target tracking result is obtained. According to the self-adaptive control method for the underwater mechanical arm, when the mechanical arm moves in the underwater environment, the target movement track can be accurately tracked, and the control precision and robustness of the underwater mechanical arm are improved.

Description

Technical Field

[0001] This invention relates to the field of reinforcement learning technology, and in particular to an adaptive control method and device for an underwater robotic arm. Background Technology

[0002] With the increasing strategic demand for marine resource development, underwater robotic arms, as core components of marine engineering equipment, play a crucial role in complex operations such as deep-sea exploration, resource extraction, and equipment maintenance. However, traditional control methods face numerous challenges in complex marine environments, making it difficult to meet the requirements for high precision and robustness.

[0003] Traditional control methods for underwater robotic arms, such as PID control, rely on fixed parameter adjustments, making it difficult to compensate for dynamic disturbances in real time, leading to decreased trajectory tracking accuracy. Furthermore, underwater robotic arms are typical multibody dynamic systems with strong nonlinear coupling in the motion of their joints. Environmental factors such as seawater resistance and additional mass torque further increase model complexity, making it difficult for traditional modeling methods to accurately identify model parameters. Simultaneously, traditional control methods lack adaptive and learning capabilities, making it difficult to cope with the complex and ever-changing marine environment, resulting in low control accuracy and robustness for underwater robotic arms. Summary of the Invention

[0004] This invention provides an adaptive control method and apparatus for an underwater robotic arm, which solves the technical problem of low control accuracy and robustness of underwater robotic arms in the prior art, and improves the control accuracy and robustness of underwater robotic arms.

[0005] This invention provides an adaptive control method for an underwater robotic arm, comprising the following steps:

[0006] Obtain the current state of the underwater robotic arm and the target's motion trajectory;

[0007] The current state and the target trajectory are input into the trained adaptive control model to obtain the first control parameters output by the trained adaptive control model; the first control parameters include the first control torque and the first angular velocity direction; the trained adaptive control model is constructed based on the Q-learning algorithm;

[0008] Based on the first control parameter, the underwater robotic arm is controlled to track the target's trajectory and obtain the target tracking result.

[0009] According to an adaptive control method for an underwater robotic arm provided by the present invention, after obtaining the target tracking result, the method further includes:

[0010] Based on the target tracking results and the target motion trajectory, the target tracking error is determined;

[0011] The first reward value corresponding to the first control torque is calculated using a preset energy matching reward function;

[0012] The second reward value corresponding to the first angular velocity direction is calculated using a preset angular velocity direction reward function;

[0013] Based on the target tracking error, the first reward value, and the second reward value, determine the first expected return estimate corresponding to the first control parameter;

[0014] Based on the first expected return estimate, the Q-table in the trained adaptive control model is updated to obtain the target Q-table;

[0015] The trained adaptive control model is updated based on the target Q-table.

[0016] According to the adaptive control method for an underwater robotic arm provided by the present invention, the step of obtaining the trained adaptive control model includes:

[0017] Obtain the current state and training trajectory of the underwater robotic arm;

[0018] The current state and the training trajectory are input into the initial adaptive control model to obtain the second control parameters output by the initial adaptive control model; the second control parameters include the second control torque and the second angular velocity direction.

[0019] Based on the second control parameter, the underwater robotic arm is controlled to track the training trajectory to obtain a first tracking result;

[0020] Based on the training trajectory, the first tracking result, and the second control parameters, the Q-table obtained in the first iteration is obtained;

[0021] Based on the Q-table obtained in the first round of iteration, the iteration steps are repeated a preset number of times to obtain the final Q-table;

[0022] The adaptive control model corresponding to the final Q-table is used as the trained adaptive control model.

[0023] According to the adaptive control method for an underwater robotic arm provided by the present invention, the step of obtaining the Q-table obtained in the first round of iteration based on the training trajectory, the first tracking result, and the second control parameters includes:

[0024] Based on the training trajectory and the first tracking result, a first tracking error is determined;

[0025] The third reward value corresponding to the second control torque is calculated using a preset energy matching reward function;

[0026] The fourth reward value corresponding to the second angular velocity direction is calculated using a preset angular velocity direction reward function;

[0027] Based on the first tracking error, the third reward value, and the fourth reward value, determine the second expected return estimate corresponding to the second control parameter;

[0028] Based on the second expected return estimate, the Q table obtained in the first round of iterations is obtained.

[0029] According to the adaptive control method for an underwater robotic arm provided by the present invention, the iterative steps include:

[0030] Based on the Q-table obtained from the previous iteration, the adaptive control model is updated;

[0031] The current state and the training trajectory are input into the updated adaptive control model to obtain the third control parameters obtained in this iteration; the third control parameters include the third control torque and the third angular velocity direction.

[0032] Based on the third control parameter, the underwater robotic arm is controlled to track the training trajectory to obtain a second tracking result.

[0033] Based on the training trajectory, the second tracking result, and the third control parameter, the Q-table obtained in this iteration is obtained.

[0034] According to the adaptive control method for an underwater robotic arm provided by the present invention, the step of obtaining the Q-table obtained in the current iteration based on the training trajectory, the second tracking result, and the third control parameters includes:

[0035] Based on the training trajectory and the second tracking result, a second tracking error is determined;

[0036] The fifth reward value corresponding to the third control torque is calculated using a preset energy matching reward function;

[0037] The sixth reward value corresponding to the third angular velocity direction is calculated using a preset angular velocity direction reward function;

[0038] Based on the second tracking error, the fifth reward value, and the sixth reward value, determine the third expected return estimate corresponding to the third control parameter;

[0039] Based on the third expected return estimate, the Q table obtained in this iteration is obtained.

[0040] The present invention also provides an adaptive control device for an underwater robotic arm, comprising the following modules:

[0041] The acquisition module is used to acquire the current state of the underwater robotic arm and the target's motion trajectory;

[0042] The control module is used to input the current state and the target motion trajectory into a trained adaptive control model to obtain the first control parameters output by the trained adaptive control model; the first control parameters include a first control torque and a first angular velocity direction; the trained adaptive control model is constructed based on the Q-learning algorithm;

[0043] The tracking module is used to control the underwater robotic arm to track the target's trajectory based on the first control parameters, thereby obtaining the target tracking result.

[0044] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and running on the processor, wherein the processor executes the computer program to implement the underwater robotic arm adaptive control method as described above.

[0045] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the adaptive control method for an underwater robotic arm as described above.

[0046] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the adaptive control method for an underwater robotic arm as described above.

[0047] This invention provides an adaptive control method for an underwater robotic arm. By acquiring the current state of the underwater robotic arm and the target trajectory, the initial state of the robotic arm and the target trajectory to be tracked are determined. Then, the current state and the target trajectory are input into a trained adaptive control model to obtain the first control parameters output by the trained adaptive control model. The first control parameters include a first control torque and a first angular velocity direction. The trained adaptive control model is constructed based on the Q-learning algorithm, thereby generating an optimal control strategy for the target trajectory during the interaction between the robotic arm and the underwater environment through reinforcement learning, and determining the corresponding first control parameters. Finally, based on the first control parameters, the underwater robotic arm is controlled to track the target trajectory, obtaining the target tracking result. This enables the robotic arm to accurately track the target trajectory when moving in the underwater environment, improving the control accuracy and robustness of the underwater robotic arm. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0049] Figure 1 This is a flowchart illustrating an adaptive control method for an underwater robotic arm provided by the present invention.

[0050] Figure 2 This is a schematic diagram of the structure of an underwater robotic arm model provided by the present invention.

[0051] Figure 3 This is a schematic diagram of the target tracking error under the test conditions provided by the present invention.

[0052] Figure 4 This is a schematic diagram of the target tracking error under test condition two provided by the present invention.

[0053] Figure 5 This is a schematic diagram of the target tracking error under test condition three provided by the present invention.

[0054] Figure 6 This is a schematic diagram of the target tracking error under test condition four provided by the present invention.

[0055] Figure 7 This is a schematic diagram of the target tracking error under test condition five provided by the present invention.

[0056] Figure 8 This is a schematic diagram of the target tracking error under test condition six provided by the present invention.

[0057] Figure 9 This is a schematic diagram of the target tracking error under test condition seven provided by the present invention.

[0058] Figure 10 This is a schematic diagram comparing the target tracking error curves of the present invention with those of the traditional PID control method under test conditions.

[0059] Figure 11 This is a schematic diagram of the structure of an adaptive control device for an underwater robotic arm provided by the present invention.

[0060] Figure 12 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0062] The following is combined with Figures 1 to 12 This invention describes an adaptive control method and apparatus for an underwater robotic arm.

[0063] Figure 1 This is a flowchart illustrating an adaptive control method for an underwater robotic arm provided by the present invention, as shown below. Figure 1 As shown, the method includes the following:

[0064] Step 101: Obtain the current state of the underwater robotic arm and the target motion trajectory;

[0065] Step 102: Input the current state and the target motion trajectory into the trained adaptive control model to obtain the first control parameters output by the trained adaptive control model; the first control parameters include the first control torque and the first angular velocity direction; the trained adaptive control model is constructed based on the Q-learning algorithm;

[0066] Step 103: Based on the first control parameters, control the underwater robotic arm to track the target's trajectory and obtain the target tracking result.

[0067] Specifically, this embodiment of the invention employs the Q-learning algorithm from reinforcement learning to construct an adaptive control model, which is then applied to the control of an underwater robotic arm. Reinforcement learning learns through interaction data between the agent and the environment, eliminating the need for a precise dynamic model and directly approximating complex nonlinear mapping relationships. The reinforcement learning agent can perceive the environmental state in real time and automatically adjust control parameters, making it suitable for complex tasks such as single-arm trajectory tracking, multi-arm collaborative operations, and obstacle avoidance planning.

[0068] Q-learning algorithm is based on Markov decision processes. Based on Markov decision processes, the underwater robotic arm control problem is modeled as a four-tuple (S, A, P, R), containing the state space S, action space A, and transition probabilities P(s, A, P, R). t+1 |s t+1 ,a t ) and reward function R(s t ,a t ,s t+1 The transition probability is determined by the underwater robotic arm's dynamics model, describing the execution action a.t Then from the original state s t Transition to the next state s t+1 The probability of the action. The reward function is used to evaluate the quality of the action and guide the robotic arm to learn a tracking strategy for the target's motion trajectory. The construction of the underwater robotic arm's dynamic model includes:

[0069] Based on the Newton-Euler method and Morrison's formula, a dynamic model of the underwater robotic arm is constructed, including rigid body dynamics modeling, underwater environment interaction modeling, and environmental disturbance modeling. The mathematical expression is as follows:

[0070]

[0071] In the formula, the inertia matrix M(q) describes the inertial resistance of the mass distribution of each link of the robotic arm to acceleration, and is related to the joint angle q. In the case of a single joint, it degenerates into a constant M; the Coriolis force and centrifugal force matrices... Characterized by the joint motion coupling effect, which is zero in the case of a single joint. Represents joint angular acceleration. The angular velocity of the joint is represented by q; the gravity vector G(q) is determined by the position of the center of mass of the connecting rod.

[0072] In controlling an underwater robotic arm, the first step is to acquire the current state of the robotic arm and the target trajectory. Then, these two data are input into a trained adaptive control model. Using the current state as the initial state and tracking the target trajectory as the control objective, the optimal tracking strategy for the robotic arm is determined. The optimal tracking strategy is approximated by the current state-action value function Q(s,a) in the adaptive control model, thereby determining the corresponding first control parameters in the action value function, including the first control torque and the first angular velocity direction.

[0073] Specifically, the state space and discretized model are first constructed, resulting in the following expression for the discretized model:

[0074]

[0075] In the formula, x1 is the angle, x2 is the angular velocity, and Δt is the discretized time step. m represents the mass of the robotic arm's link; g represents the acceleration due to gravity; L represents the length of the robotic arm's link; M represents the moment of inertia; C D Indicates the drag coefficient; C M denoted by ; k represents the time k; u(k) represents the control torque; d(k) represents the underwater environmental disturbance parameter.

[0076] By setting the conditions and parameters of the discretized model, a simulation environment for the underwater robotic arm can be constructed.

[0077] For example, the initial conditions are set as x1(0) = 0 and x2(0) = 0. The parameters used are: moment of inertia M = 2 kg·m. 2 The mass of the robotic arm's link is m = 5 kg, the length of the robotic arm's link is L = 0.8 m, and the drag coefficient is C. D = 1.6 kg·m 2 / s, inertial force coefficient C M = 2.1 kg·m 2 / s, gravitational acceleration g=9.8m / s 2 Reference output signal y d =sin(0.1t). Substituting the numerical values, the expression for the discretized model simplifies to:

[0078]

[0079] In the formula, the discretization time step Δt = 0.001s.

[0080] After constructing the simulation environment for the underwater robotic arm, the first control torque and the first angular velocity direction for controlling the robotic arm in the simulation environment are determined.

[0081] The first control torque is obtained through the energy matching reward function R of the robotic arm dynamics. dyn Calculations are performed on the energy matching reward function R. dyn The expression is:

[0082]

[0083] In the formula, u is the first control torque, u req The actual torque required to counteract gravity and underwater resistance, without considering interference and other factors, is a theoretical approximation. By using an energy-matching reward function to constrain energy consumption, the robotic arm can be guided to learn a control strategy that conforms to physical laws. This allows the robotic arm to efficiently learn a tracking strategy for the target's trajectory while calculating the reward value corresponding to the first control torque. Based on the reward value feedback, the first control torque is adjusted, achieving high-precision control of the underwater robotic arm.

[0084] The angular velocity direction reward function is used to evaluate whether the robot arm's angular velocity direction matches the expected direction, encouraging the robot arm to move in the expected direction, suppressing ineffective reverse movements, and enhancing the directional bias of the control strategy. The angular velocity direction reward function R... dir The expression is as follows:

[0085]

[0086] In the formula, q represents the absolute value of the joint angular velocity. dLet q represent the desired joint angle (expected direction) and q represent the actual joint angle. Based on the angular velocity direction reward function, the reward value corresponding to the first angular velocity direction for controlling the movement of the robotic arm can be determined. Then, the first angular velocity direction can be adjusted based on the reward value feedback, so that the dynamic trend of the robotic arm matches the expected direction.

[0087] Based on the first control torque and the first angular velocity direction, the underwater robotic arm is controlled to track the target's trajectory and obtain the target tracking result.

[0088] This invention constructs an adaptive control model based on the Q-learning algorithm. The trained adaptive control model generates an optimal tracking strategy for the target trajectory based on the current state of the underwater manipulator and the target trajectory. It calculates the first control torque and first angular velocity direction of the underwater manipulator at any time step under the optimal tracking strategy, thereby controlling the underwater manipulator to continuously perform high-precision tracking motion on the target trajectory. This overcomes the dependence on fixed models of traditional control methods (such as PID control), realizes autonomous decision-making and high-precision control in complex environments, and improves the control accuracy and robustness of the underwater manipulator.

[0089] This invention provides an adaptive control method for an underwater robotic arm. By acquiring the current state of the underwater robotic arm and the target trajectory, the initial state of the robotic arm and the target trajectory to be tracked are determined. Then, the current state and the target trajectory are input into a trained adaptive control model to obtain the first control parameters output by the trained adaptive control model. The first control parameters include a first control torque and a first angular velocity direction. The trained adaptive control model is constructed based on the Q-learning algorithm, thereby generating an optimal control strategy for the target trajectory during the interaction between the robotic arm and the underwater environment through reinforcement learning, and determining the corresponding first control parameters. Finally, based on the first control parameters, the underwater robotic arm is controlled to track the target trajectory, obtaining the target tracking result. This enables the robotic arm to accurately track the target trajectory when moving in the underwater environment, improving the control accuracy and robustness of the underwater robotic arm.

[0090] Furthermore, after obtaining the target tracking result, the method further includes:

[0091] Based on the target tracking results and the target motion trajectory, the target tracking error is determined;

[0092] The first reward value corresponding to the first control torque is calculated using a preset energy matching reward function;

[0093] The second reward value corresponding to the first angular velocity direction is calculated using a preset angular velocity direction reward function;

[0094] Based on the target tracking error, the first reward value, and the second reward value, determine the first expected return estimate corresponding to the first control parameter;

[0095] Based on the first expected return estimate, the Q-table in the trained adaptive control model is updated to obtain the target Q-table;

[0096] The trained adaptive control model is updated based on the target Q-table.

[0097] Specifically, the adaptive control model designed in this invention, in addition to pre-trained supervised training, incorporates an online learning mechanism during actual use. Specifically, by designing multiple reward functions, continuous feedback adjustment is performed during the control process to update the Q-table, enabling the model to adapt to unforeseen dynamic disturbances, thereby adjusting the control parameters of the underwater robotic arm and improving its control robustness.

[0098] Based on the target tracking results, a two-dimensional state space is defined, including the robotic arm joint angle error *e* and the error change rate *de*. The robotic arm joint angle error *e* reflects the deviation between the current position of the robotic arm after tracking the target's trajectory (target tracking error), while the error change rate *de* reflects the trend of the target tracking error.

[0099] The joint angle error e of the robotic arm is:

[0100] e = qq d

[0101] In the formula, q represents the actual joint angle, q d This indicates the desired joint angle.

[0102] The error change rate de is:

[0103]

[0104] In the formula, This represents the actual joint angular velocity. This represents the desired joint angular velocity.

[0105] Figure 2 This is a structural schematic diagram of an underwater robotic arm model provided by the present invention, as shown below. Figure 2 As shown, taking the single-joint upper arm of a deep-sea manned vehicle as an example, the robotic arm is simplified as a single-link rigid body system with the center of mass at the midpoint of the link. In the tracking task, the amplitude of the target motion trajectory does not exceed 1 rad. Therefore, the actual fluctuation range of the robotic arm joint angle error e is concentrated in [-1 rad, 1 rad]. The reserved ±2 rad redundancy range can cover extreme working conditions.

[0106] If the actual motion trajectory of the robotic arm fails to be accurately tracked, causing the target tracking error to exceed the amplitude of the target motion trajectory, this still satisfies the consideration of limiting the movement of the robotic arm. The maximum rotation angle of a single joint of an underwater robotic arm is usually ±170°, approximately ±3 rad. Therefore, the theoretical maximum value of the joint angle error e of the robotic arm does not exceed this range. Thus, the range of the joint angle error of the robotic arm is set to [-3 rad, 3 rad], and based on the error change obtained from the preliminary experimental test, the range of the error change rate is set to [-30, 30].

[0107] Since the maximum control torque actually required for the upper arm of a single-joint deep-sea manned vehicle is determined by gravity and underwater resistance, the range of the first control torque can be estimated by calculation. Based on the calculation, the maximum value of the first control torque, |u|, can be set. max =400 N·m, with a safety margin of 3 to 4 times to prevent instantaneous impact from damaging the actuator.

[0108] To achieve more precise feedback adjustment of the initial control torque, this invention designs a three-level hierarchical reward function system in the adaptive control model to guide the robotic arm in learning a tracking strategy for the target motion trajectory. The three-level hierarchical reward function system includes multi-objective optimizations such as error penalty, energy consumption constraint, and dynamic trend matching, enabling the robotic arm to learn efficiently in a two-dimensional state space.

[0109] The error penalty is constructed as follows:

[0110] R = -w1|e|-w2|de|-w3u 2

[0111] In the formula, w1, w2, and w3 are the weights of the joint angle error e, the error change rate de, and the first control torque u of the robotic arm, respectively.

[0112] Energy matching reward function R based on robotic arm dynamics dyn for:

[0113]

[0114] In the formula, u req The first control torque required to counteract gravity and drag, without considering interference and other factors, is a theoretical approximation. By using an energy-matching reward function to constrain energy consumption, the robotic arm can be guided to learn a control strategy that conforms to physical laws. This allows the robotic arm to efficiently learn a tracking strategy for the target's motion trajectory while calculating the first reward value corresponding to the first control torque, thus achieving high-precision control of the underwater robotic arm.

[0115] The precision-based reward in the three-level reward function system is constructed as follows:

[0116]

[0117] In the formula, This represents the amplitude of the target motion trajectory. The first three levels represent the high-precision, medium-precision, and low-precision zones, respectively. The bonus for the high-precision zone is awarded when the absolute value of the robotic arm joint angle error, |e|, is less than the target precision. The positive reward for the first time is +2000, and the reward for the medium precision zone is when the absolute value of the robot arm joint angle error |e| is greater than or equal to the target precision. Less than target precision A positive reward of +1000 is given when the accuracy is high; the reward in the low-precision zone is given when the absolute value of the robot arm joint angle error |e| is greater than or equal to the target accuracy. Less than target precision A positive reward of +500 is given when the absolute value of the robotic arm joint angle error |e| is greater than the target accuracy. At this time, the positive reward is 0. This hierarchical setting effectively highlights the goal-oriented characteristics of reinforcement learning and strengthens the robotic arm's behavioral memory during trajectory tracking. It should be noted that in this embodiment of the invention, the target accuracy threshold for each level can be adaptively set, and there are no limitations on this.

[0118] To provide more precise feedback adjustment of the first angular velocity direction, guide the robotic arm to move in the expected direction, suppress ineffective reverse movements, and enhance the directional bias of the control strategy, this invention utilizes an angular velocity direction reward function R... dir Calculate the second bonus value, R, corresponding to the direction of the first angular velocity. dir The expression is as follows:

[0119]

[0120] Based on the angular velocity direction reward function, the second reward value corresponding to the first angular velocity direction controlling the movement of the robotic arm can be determined, and then the first angular velocity direction can be adjusted to tend towards the direction with a larger reward value.

[0121] Based on the target tracking error, the first reward value, and the second reward value, the first expected reward estimate can be determined after the robotic arm tracks the target trajectory using the first control parameters. Then, the Q-table in the trained adaptive control model is updated according to the first expected reward estimate to obtain the target Q-table.

[0122] For example, based on the adaptive learning mechanism in the Q-learning algorithm, the Q-table update strategy in the adaptive control model is determined, and the initial values ​​of the learning rate and discount factor, as well as the decay strategy, are selected as follows:

[0123] Q(s t ,a t)←Q(s t ,a t )+α(r t +γmax a′ Q(s t+1 ,a′)-Q(s t ,a t ))

[0124] In the formula, s t Indicates the state of the robotic arm at the current time step, a t This indicates the action performed by the robotic arm at the current time step under the first control parameters, r. t s represents the immediate reward obtained at the current time step. t+1 Let represent the robotic arm state at the next time step, a′ represent one of the possible actions in the next state, α represent the learning rate (used to control the magnitude of model updates), and γ represent the discount factor (used to weigh the importance of future rewards).

[0125] The Q-table is updated in the direction of reducing the target tracking error and increasing the first and second reward values ​​to obtain the Q-table corresponding to the next time step, which is then used as the target Q-table. The original Q-table is then replaced with the target Q-table to update the trained adaptive control model.

[0126] For example, during the testing phase, the learning rate is set to 0.05, far lower than the initial value during the training phase, and the discount factor is 0.9. This allows the robotic arm to fine-tune based on real-time feedback while retaining most of the model training results. Simultaneously, to avoid mechanical shocks to the actuators caused by abrupt changes in the control signal, a torque change rate limit is introduced, ensuring that the change in control input between adjacent time steps does not exceed 20 N*m.

[0127] Table 1 is a comparison table of simulation experimental conditions provided by the present invention. As shown in Table 1, in order to facilitate observation and comparison, the embodiments of the present invention adopt the controlled variable method and design the following seven typical simulation experimental conditions:

[0128] Table 1 Comparison of Simulation Experiment Conditions

[0129]

[0130] The following three sets of simulations were performed:

[0131] (1) Test the adaptive capability to changes in the desired output signal under interference-free conditions.

[0132] Figure 3 This is a schematic diagram of the target tracking error under the test conditions provided by the present invention, as shown below. Figure 3 As shown. Figure 4 This is a schematic diagram of the target tracking error under test condition two provided by the present invention, as shown below. Figure 4 As shown. Figure 5 This is a schematic diagram of the target tracking error under test condition three provided by the present invention, as shown below. Figure 5 As shown in the table. The simulation conditions are conditions one through three, which represent the reference sine wave and sine waves with varying amplitudes and frequencies, respectively. The simulation error curves of the underwater robotic arm adaptive control method proposed in this invention are shown in the table. Figures 3 to 5 As shown, the underwater robotic arm adaptive control method proposed in this invention can achieve robustness to changes in the desired output signal, and the target tracking error does not change significantly, thus exhibiting high accuracy.

[0133] (2) Test the adaptive capability under different types of disturbances.

[0134] Figure 6 This is a schematic diagram of the target tracking error under test condition four provided by the present invention, as shown below. Figure 6 As shown. Figure 7 This is a schematic diagram of the target tracking error under test condition five provided by the present invention, as shown below. Figure 7 As shown. Figure 8 This is a schematic diagram of the target tracking error under test condition six provided by the present invention, as shown below. Figure 8 As shown in the table. The simulation conditions are conditions four to six in the table, that is, sinusoidal disturbance, step disturbance, and random disturbance are applied to condition one respectively. The underwater robotic arm adaptive control method proposed in this invention was used for simulation, and the error curves are shown in the table. Figures 6 to 8 As shown, under the adaptive control method for underwater robotic arms proposed in this invention, the adaptive controller can adapt well to small-amplitude, high-frequency periodic disturbances, large-amplitude abrupt disturbances, or unknown random disturbances. The response speed is very fast, and the maximum position error remains stable at around 0.024.

[0135] (3) Test the fast and accurate response capability under step signal.

[0136] Figure 9 This is a schematic diagram of the target tracking error under test condition seven provided by the present invention, as shown below. Figure 9 As shown in the table. The simulation conditions are condition seven in the table. The desired output signal type is changed to a step signal. Under interference-free conditions, the underwater robotic arm adaptive control method proposed in this invention was simulated. The target tracking error result curve is shown in the figure. Figure 9 As shown, the settling time is 8.41s and the overshoot is 0.009, or 0.9%. This demonstrates that the underwater robotic arm adaptive control method proposed in this invention can achieve a fast and accurate response to step signals.

[0137] Figure 10This is a schematic diagram comparing the target tracking error curves of the present invention with those of the traditional PID control method under test conditions, as provided by the present invention. Figure 10 As shown in the figure, a comparative experiment was conducted between the proposed adaptive control method for underwater robotic arms and the traditional PID control method. Only the comparison results under the following conditions are presented here. Figure 10 As shown in the figure. From the perspective of tracking performance, the adaptive control method for the underwater manipulator proposed in this invention exhibits excellent tracking performance under various conditions, with a maximum tracking error not exceeding 0.025, approximately 2.5% of the amplitude. Under step signals, the settling time is short and the overshoot is also small. In contrast, the tracking error of traditional PID controllers is relatively large, reaching a maximum of 0.074, approximately 7.4% of the amplitude. Therefore, compared with traditional controllers, the adaptive control method for the underwater manipulator proposed in this invention achieves a 40% reduction in error, a 9.38% reduction in settling time, and a more than 60% improvement in anti-interference compliance rate in practical cases. It stabilizes the maximum trajectory tracking error at the end of the manipulator at 0.02 rad, approximately 2% of the maximum position. This indicates that the adaptive control method for the underwater manipulator proposed in this invention improves the control accuracy and response speed of the underwater manipulator and shortens the settling time.

[0138] This invention, through the design of multiple reward functions and online learning mechanisms, enables the trained adaptive control model to adjust based on the first expected reward estimate during actual use. It possesses online update capabilities, continuously updating the Q-table to cope with unforeseen dynamic disturbances in the real environment, achieving self-adaptation. The updated model then determines the control parameters for the next time step, allowing the robotic arm to continuously approach the optimal tracking strategy during target trajectory tracking. This effectively resists interference factors and achieves continuous tracking of the target trajectory, overcoming the dependence on fixed models inherent in traditional control methods (such as PID control). It realizes autonomous decision-making and high-precision control in complex environments, improving the control accuracy and robustness of the underwater robotic arm.

[0139] Furthermore, the step of obtaining the trained adaptive control model includes:

[0140] Obtain the current state and training trajectory of the underwater robotic arm;

[0141] The current state and the training trajectory are input into the initial adaptive control model to obtain the second control parameters output by the initial adaptive control model; the second control parameters include the second control torque and the second angular velocity direction.

[0142] Based on the second control parameter, the underwater robotic arm is controlled to track the training trajectory to obtain a first tracking result;

[0143] Based on the training trajectory, the first tracking result, and the second control parameters, the Q-table obtained in the first iteration is obtained;

[0144] Based on the Q-table obtained in the first round of iteration, the iteration steps are repeated a preset number of times to obtain the final Q-table;

[0145] The adaptive control model corresponding to the final Q-table is used as the trained adaptive control model.

[0146] Further, obtaining the Q-table obtained from the first iteration based on the training trajectory, the first tracking result, and the second control parameters includes:

[0147] Based on the training trajectory and the first tracking result, a first tracking error is determined;

[0148] The third reward value corresponding to the second control torque is calculated using a preset energy matching reward function;

[0149] The fourth reward value corresponding to the second angular velocity direction is calculated using a preset angular velocity direction reward function;

[0150] Based on the first tracking error, the third reward value, and the fourth reward value, determine the second expected return estimate corresponding to the second control parameter;

[0151] Based on the second expected return estimate, the Q table obtained in the first round of iterations is obtained.

[0152] Further, the iterative steps include:

[0153] Based on the Q-table obtained from the previous iteration, the adaptive control model is updated;

[0154] The current state and the training trajectory are input into the updated adaptive control model to obtain the third control parameters obtained in this iteration; the third control parameters include the third control torque and the third angular velocity direction.

[0155] Based on the third control parameter, the underwater robotic arm is controlled to track the training trajectory to obtain a second tracking result.

[0156] Based on the training trajectory, the second tracking result, and the third control parameter, the Q-table obtained in this iteration is obtained.

[0157] Further, obtaining the Q-table obtained in this iteration based on the training trajectory, the second tracking result, and the third control parameters includes:

[0158] Based on the training trajectory and the second tracking result, a second tracking error is determined;

[0159] The fifth reward value corresponding to the third control torque is calculated using a preset energy matching reward function;

[0160] The sixth reward value corresponding to the third angular velocity direction is calculated using a preset angular velocity direction reward function;

[0161] Based on the second tracking error, the fifth reward value, and the sixth reward value, determine the third expected return estimate corresponding to the third control parameter;

[0162] Based on the third expected return estimate, the Q table obtained in this iteration is obtained.

[0163] Specifically, the adaptive control model is trained in advance using relatively complex signals as training trajectories. This allows the model to learn tracking strategies for complex signals, enabling it to easily cope with unknown environments and accurately track the target's motion trajectory during actual use. The training trajectory can be a sine wave signal or a step signal of various frequencies and amplitudes.

[0164] In the training process of the adaptive control model, the parameters and Q-table are essentially updated continuously through feedback adjustment. After completing a preset number of iterations, the adaptive control model corresponding to the finally updated parameters and Q-table is used as the trained adaptive control model. Since the reinforcement learning agent needs to map continuous states to discrete indices to update the Q-table, the choice of discretization method directly affects the control accuracy and computational efficiency. For the discretization of the state space and action space, the embodiments of this invention select the uniform binning method. The uniform binning strategy divides the continuous state into equally spaced discrete intervals. Therefore, both state variables are divided into 300 discrete intervals, while the action variables are uniformly divided into 800 action levels.

[0165] Based on the above embodiments, the training process is similar to the online learning process of the trained adaptive control model in actual use. The main difference is that the learning rate of the adaptive control model is much larger during training than during actual use. This is because the Q-table update mechanism affects the convergence speed and learning effect. For example, the initial learning rate during training can be set to 0.3, which is much larger than 0.05 in actual use.

[0166] Specifically, in the first iteration, the current state and training trajectory are first input into the initial adaptive control model to obtain the second control parameters output by the initial adaptive control model, including the second control torque and the second angular velocity direction. Then, based on the second control parameters, the underwater robotic arm is controlled to track the training trajectory to obtain the first tracking result. Then, based on the training trajectory and the first tracking result of the underwater robotic arm on the training trajectory, the first tracking error is determined. Then, based on the energy matching reward function mentioned above, the third reward value corresponding to the second control torque is calculated. Through the preset angular velocity direction reward function, the fourth reward value corresponding to the second angular velocity direction is calculated. Then, based on the first tracking error, the third reward value, and the fourth reward value, the second expected reward estimate corresponding to the second control parameter is determined. Finally, based on the second expected reward estimate, the Q table obtained in the first iteration is obtained, and the original Q table is updated to obtain the Q table obtained in the first iteration.

[0167] Repeat the iteration steps a preset number of times, updating the Q-table once in each iteration, and updating the adaptive control model once with the updated Q-table, until all iterations are completed, obtaining the final Q-table and the corresponding adaptive control model, thus obtaining the trained adaptive control model.

[0168] After multiple tests, the initial learning rate of this embodiment is designed to be 0.3 (other values ​​can also be set according to actual needs, but 0.3 is preferred in this embodiment) to facilitate rapid absorption of new experience and suitability for exploration in the early stages of training. During training, the learning rate decreases linearly over time until it reaches 0.1, reducing the update amplitude in later stages and avoiding oscillations in the optimal Q-value. The discount factor balancing current and future rewards is set to 0.85 to ensure that the agent mainly focuses on rewards within the current to the next 0.01 seconds, avoiding policy hesitation due to excessive focus on long-term rewards. The greedy strategy used in action selection requires an exploration rate designed according to the actual situation. To ensure sufficient exploration in the first 100 rounds, covering state-action pairs as much as possible, and avoiding local optima due to insufficient exploration, this invention designs the exploration rate to decay exponentially with the number of rounds, achieving a smooth transition from exploration to utilization. For example, the initial exploration rate is set to ε = 0.95, and the decay rate is 0.98.

[0169] Furthermore, during reinforcement learning training, the initialization strategy of the Q-table directly affects the convergence speed and final performance of the algorithm. In this embodiment of the invention, a normal distribution is used to randomly initialize the Q-table in the initial adaptive control model, setting the mean to 0 and the standard deviation to 0.1, thereby avoiding learning bias caused by initial values ​​that are too large or too small. Through the randomness of the normal distribution, a neutral distribution of the initial Q-values ​​is ensured, while the standard deviation controls the dispersion of the initial values, thus balancing the initial stages of exploration and utilization.

[0170] Optionally, in the model training process of this embodiment of the invention, the duration of each iteration is 60 seconds, the time step is 0.001 seconds, and the preset number of iterations is 200. The training target signal adopts a dynamic random switching strategy, including sine waves, step signals, and sigmoid transition curves with different amplitudes and frequencies, to enhance the agent's adaptability to different task conditions and improve the model's generalization ability.

[0171] This invention trains an initial adaptive control model by using sine waves, step signals, and sigmoid transition curves of different amplitudes and frequencies as training trajectories. This allows the model to learn control strategies for tracking complex signals in advance, improving the generalization ability of the adaptive control model. As a result, it can effectively cope with complex and ever-changing marine environments and improve the control accuracy and robustness of the underwater robotic arm.

[0172] The following describes an adaptive control device for an underwater robotic arm provided by the present invention. The adaptive control device for an underwater robotic arm described below can be referred to in correspondence with the adaptive control method for an underwater robotic arm described above.

[0173] Based on any of the above embodiments Figure 11 This is a schematic diagram of the structure of an adaptive control device for an underwater robotic arm provided by the present invention, as shown below. Figure 11 As shown. This embodiment of the invention provides an adaptive control device for an underwater robotic arm, including an acquisition module 1101, a control module 1102, and a tracking module 1103, wherein:

[0174] The acquisition module 1101 is used to acquire the current state of the underwater robotic arm and the target motion trajectory; the control module 1102 is used to input the current state and the target motion trajectory into a trained adaptive control model to obtain the first control parameters output by the trained adaptive control model; the first control parameters include a first control torque and a first angular velocity direction; the trained adaptive control model is constructed based on the Q-learning algorithm; the tracking module 1103 is used to control the underwater robotic arm to track the target motion trajectory based on the first control parameters to obtain the target tracking result.

[0175] This invention provides an adaptive control device for an underwater robotic arm. By acquiring the current state of the underwater robotic arm and the target trajectory, the initial state of the robotic arm and the target trajectory to be tracked are determined. Then, the current state and the target trajectory are input into a trained adaptive control model to obtain the first control parameters output by the trained adaptive control model. The first control parameters include a first control torque and a first angular velocity direction. The trained adaptive control model is constructed based on the Q-learning algorithm, thereby generating the optimal control strategy for the target trajectory during the interaction between the robotic arm and the underwater environment through reinforcement learning, and determining the corresponding first control parameters. Finally, based on the first control parameters, the underwater robotic arm is controlled to track the target trajectory, obtaining the target tracking result. This enables the robotic arm to accurately track the target trajectory when moving in the underwater environment, improving the control accuracy and robustness of the underwater robotic arm.

[0176] Figure 12 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 12 As shown, the electronic device may include: a processor 1210, a communication interface 1220, a memory 1230, and a communication bus 1240, wherein the processor 1210, the communication interface 1220, and the memory 1230 communicate with each other via the communication bus 1240. The processor 1210 can call logic instructions in the memory 1230 to execute an adaptive control method for the underwater robotic arm, the method including:

[0177] Obtain the current state of the underwater robotic arm and the target's motion trajectory;

[0178] The current state and the target trajectory are input into the trained adaptive control model to obtain the first control parameters output by the trained adaptive control model; the first control parameters include the first control torque and the first angular velocity direction; the trained adaptive control model is constructed based on the Q-learning algorithm;

[0179] Based on the first control parameter, the underwater robotic arm is controlled to track the target's trajectory and obtain the target tracking result.

[0180] Furthermore, the logical instructions in the aforementioned memory 1230 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0181] On the other hand, the present invention also provides a computer program product, the computer program product comprising a computer program that can be stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer is able to execute the underwater robotic arm adaptive control method provided by the above methods, the method comprising:

[0182] Obtain the current state of the underwater robotic arm and the target's motion trajectory;

[0183] The current state and the target trajectory are input into the trained adaptive control model to obtain the first control parameters output by the trained adaptive control model; the first control parameters include the first control torque and the first angular velocity direction; the trained adaptive control model is constructed based on the Q-learning algorithm;

[0184] Based on the first control parameter, the underwater robotic arm is controlled to track the target's trajectory and obtain the target tracking result.

[0185] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the adaptive control method for an underwater robotic arm provided by the methods described above, the method comprising:

[0186] Obtain the current state of the underwater robotic arm and the target's motion trajectory;

[0187] The current state and the target trajectory are input into the trained adaptive control model to obtain the first control parameters output by the trained adaptive control model; the first control parameters include the first control torque and the first angular velocity direction; the trained adaptive control model is constructed based on the Q-learning algorithm;

[0188] Based on the first control parameter, the underwater robotic arm is controlled to track the target's trajectory and obtain the target tracking result.

[0189] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0190] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0191] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0192] It should also be noted that the terms "target," "first," and "second" in this invention are used to distinguish similar objects, not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, without limiting the number of objects; for example, the first object can be one or more.

[0193] In this application's embodiments, "determine B based on A" means that factor A must be considered when determining B. It is not limited to "B can be determined based solely on A," but should also include: "determine B based on A and C," "determine B based on A, C, and E," "determine C based on A, and further determine B based on C," etc. Additionally, it can include using A as a condition for determining B, for example, "when A meets the first condition, determine B using the first method"; another example, "when A meets the second condition, determine B," etc.; another example, "when A meets the third condition, determine B based on the first parameter," etc. Of course, it can also be a condition where A is a factor in determining B, for example, "when A meets the first condition, determine C using the first method, and further determine B based on C," etc.

[0194] In this invention, the term "multiple" refers to two or more, and other quantifiers are similar.

[0195] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An adaptive control method for an underwater robotic arm, the method comprising: The method comprises: obtaining a current state of an underwater manipulator and a target motion trajectory; inputting the current state and the target motion trajectory into a trained adaptive control model to obtain a first control parameter output by the trained adaptive control model; the first control parameter comprises a first control torque and a first angular velocity direction; the trained adaptive control model is constructed based on a Q-learning algorithm; based on the first control parameter, controlling the underwater manipulator to perform tracking motion on the target motion trajectory to obtain a target tracking result.

2. The adaptive control method for an underwater robotic arm according to claim 1, wherein, After the target tracking result is obtained, the method further comprises: determining a target tracking error based on the target tracking result and the target motion trajectory; calculating a first reward value corresponding to the first control torque through a preset energy matching reward function; calculating a second reward value corresponding to the first angular velocity direction through a preset angular velocity direction reward function; determining a first expected return estimation value corresponding to the first control parameter based on the target tracking error, the first reward value, and the second reward value; updating a Q table in the trained adaptive control model based on the first expected return estimation value to obtain a target Q table; updating the trained adaptive control model based on the target Q table.

3. The adaptive control method for an underwater robotic arm according to claim 1, wherein, The obtaining step of the trained adaptive control model comprises: obtaining a current state of an underwater manipulator and a training trajectory; inputting the current state and the training trajectory into an initial adaptive control model to obtain a second control parameter output by the initial adaptive control model; the second control parameter comprises a second control torque and a second angular velocity direction; controlling the underwater manipulator to perform tracking motion on the training trajectory based on the second control parameter to obtain a first tracking result; obtaining a Q table obtained through a first round of iteration based on the training trajectory, the first tracking result, and the second control parameter; repeating an iteration step for a preset number of times based on the Q table obtained through the first round of iteration to obtain a final Q table; using an adaptive control model corresponding to the final Q table as the trained adaptive control model.

4. The adaptive control method for an underwater robotic arm according to claim 3, wherein, The obtaining of the Q table obtained through the first round of iteration based on the training trajectory, the first tracking result, and the second control parameter comprises: determining a first tracking error based on the training trajectory and the first tracking result; calculating a third reward value corresponding to the second control torque through a preset energy matching reward function; calculating a fourth reward value corresponding to the second angular velocity direction through a preset angular velocity direction reward function; determining a second expected return estimation value corresponding to the second control parameter based on the first tracking error, the third reward value, and the fourth reward value; obtaining the Q table obtained through the first round of iteration based on the second expected return estimation value.

5. The adaptive control method for an underwater robotic arm according to claim 3, wherein, The iteration step comprises: updating the adaptive control model based on the Q table obtained through the last round of iteration; input the current state and the training trajectory into the updated adaptive control model to obtain third control parameters obtained in the current iteration; the third control parameters comprise a third control torque and a third angular velocity direction; control the underwater manipulator to perform tracking motion on the training trajectory based on the third control parameters to obtain a second tracking result; obtain a Q table obtained in the current iteration based on the training trajectory, the second tracking result, and the third control parameters.

6. The adaptive control method for an underwater robotic arm according to claim 5, wherein, The obtaining of the Q table obtained in the current iteration based on the training trajectory, the second tracking result, and the third control parameters comprises: determining a second tracking error based on the training trajectory and the second tracking result; calculating a fifth reward value corresponding to the third control torque by using a preset energy matching reward function; calculating a sixth reward value corresponding to the third angular velocity direction by using a preset angular velocity direction reward function; determining a third expected return estimation value corresponding to the third control parameters based on the second tracking error, the fifth reward value, and the sixth reward value; obtaining the Q table obtained in the current iteration based on the third expected return estimation value.

7. An underwater manipulator adaptive control device, characterized by, comprise: an acquisition module configured to acquire a current state of an underwater manipulator and a target motion trajectory; a control module configured to input the current state and the target motion trajectory into a trained adaptive control model to obtain first control parameters output by the trained adaptive control model; the first control parameters comprise a first control torque and a first angular velocity direction; the trained adaptive control model is constructed based on a Q-learning algorithm; a tracking module configured to control the underwater manipulator to perform tracking motion on the target motion trajectory based on the first control parameters to obtain a target tracking result.

8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, The processor implements the underwater manipulator adaptive control method in any one of claims 1 to 6 when executing the computer program. 9.A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program implements the underwater manipulator adaptive control method in any one of claims 1 to 6 when executed by the processor.

10. A computer program product comprising a computer program, characterized in that, The computer program implements the underwater manipulator adaptive control method in any one of claims 1 to 6 when executed by the processor.

Images