An underwater bionic snake robot and a hybrid adaptive control method thereof

Abstract

The application relates to the technical field of underwater bionic robots, solves the technical problems that a traditional snake-shaped robot cannot autonomously adapt to environmental changes, optimize motion performance and intelligently coordinate multiple propulsion modes, and particularly relates to an underwater bionic snake-shaped robot and a hybrid adaptive control method thereof, which comprises a snake-shaped robot composed of a snake body assembly, a snake head assembly and a jet propulsion module; control instructions for each electric joint and control instructions for the jet propulsion module are output by using a reinforcement learning intelligent agent according to real-time environmental perception and robot state feedback. The control of the electric joint can be adjusting the parameters of the central pattern generator to generate the expected joint angle, or directly outputting the expected driving torque of each electric joint. The snake-shaped robot provided by the application is suitable for pipeline inspection, underwater detection and other tasks, the modular design reduces the manufacturing cost and maintenance difficulty, and the advanced control method improves the intelligent level of the robot.

Description

Technical Field

[0001] This invention relates to the field of underwater biomimetic robot technology, and in particular to an underwater biomimetic snake robot and its hybrid adaptive control method. Background Technology

[0002] With the increasing demand for marine resource development and underwater environment exploration, underwater robots have become one of the important directions for current research and application. Common underwater robots currently include propeller-driven autonomous underwater vehicles (AUVs) and remotely operated underwater vehicles (ROVs). Although these traditional underwater robots have been widely used in many fields, their flexibility and maneuverability are significantly insufficient in narrow, current-rich, and topographically challenging underwater environments.

[0003] In recent years, the development of bionics has driven research into snake-like robots in the underwater field. Underwater snake-like robots mimic the segmental undulation propulsion of creatures such as water snakes and eels, possessing excellent flexibility and controllability, making them suitable for tasks such as pipeline inspection, underwater maintenance, and ecological monitoring.

[0004] However, in the complex and ever-changing real underwater environment, enabling snake-like robots to autonomously adapt to environmental changes, optimize their motion performance, and intelligently coordinate their multiple propulsion modes remains a challenging problem. Traditional control methods based on preset parameters or fixed models are insufficient to efficiently cope with water flow disturbances, unknown obstacles, and efficient maneuvering under multi-tasking requirements. Therefore, developing an advanced control strategy with stronger adaptability and learning capabilities is crucial for improving the practicality and intelligence of underwater snake-like robots. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides an underwater biomimetic snake robot and its hybrid adaptive control method, which solves the technical problems of traditional snake robots being unable to autonomously adapt to environmental changes, optimize motion performance, and intelligently coordinate their multiple propulsion modes.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: an underwater biomimetic snake-like robot, comprising:

[0007] The snake-body component is composed of multiple modular joint segments connected in series, and multi-degree-of-freedom wave motion of the snake-body component is achieved by rotating the joint segments;

[0008] The snake head assembly, which has a built-in camera and sensors, is installed at one end of the snake body assembly and is used for environmental perception and obstacle avoidance.

[0009] The jet propulsion module includes a miniature water pump and a water reservoir integrated into a closed cavity and mounted on the other end of the snake body assembly. It generates reaction thrust by rapidly draining water, providing the snake robot with a short burst of propulsion.

[0010] Furthermore, each joint segment of the snake body assembly includes a motor joint and a rotary joint threadedly fixed to both ends of the motor joint. The motor joint is sequentially linked with the rotary joint of another joint segment to form a snake body assembly capable of multi-directional swinging.

[0011] Furthermore, the motor joint incorporates a motor and a waterproof structure. The waterproof structure includes a waterproof plug disposed within the inner cavity of the motor joint. The outer ring of the waterproof plug is provided with an O-ring rubber ring that provides static waterproofing. The interior of the waterproof plug is provided with a double-lip stainless steel oil seal that provides dynamic sealing during motor rotation. Additionally, a rubber sheet is installed on the rotor portion of the motor to prevent water from seeping in.

[0012] Furthermore, the rotation speed and angle of the motor are independently adjusted by the ARM control board via the CAN bus protocol to enable the snake robot to move forward, turn, and adjust its posture.

[0013] Furthermore, the micro water pump is connected to the water storage bladder and is controlled by the ARM control board to start and stop. It generates instantaneous thrust through rapid drainage, which, together with the multi-degree-of-freedom undulating motion of the snake body component, enables multimodal maneuvering.

[0014] To achieve the aforementioned control objectives, this invention proposes a hybrid adaptive control method for the aforementioned underwater biomimetic snake robot, based on the robot structure described above. This method includes the following steps:

[0015] A reinforcement learning agent is constructed and run, which receives environmental perception information and robot's own state information from the snake head component as state information s;

[0016] The reinforcement learning agent outputs control commands for each motor joint of the snake body component and / or control commands for the jet propulsion module based on the state information s and the policies it has learned.

[0017] The control commands for each motor joint include:

[0018] The desired joint angle is generated by the central pattern generator module, wherein the reinforcement learning agent outputs the adjustment amount of the CPG parameters to update the CPG parameters in real time and generate the final swing command to drive each motor joint. The CPG parameters include amplitude, frequency and phase difference between adjacent joints.

[0019] Alternatively, the reinforcement learning agent can directly output the desired torque vector for each motor joint.

[0020] Furthermore, the desired joint angle generated by the central pattern generator module includes:

[0021] S11. Using the central pattern generator module to generate initial rhythmic swing commands for the multi-joint snake body components of the snake robot, the initial joint angle θ of the i-th motor joint is then determined. i Determined by the following formula:

[0022] θ i =A b sin(2πf b t+φ i )

[0023] φ i =(i-1)Δφ b

[0024] In the formula, A b f is the reference amplitude for the joint oscillation of the motor; b φ is the reference frequency for the joint oscillation of the motor; t is time; φ i The phase lag of the i-th motor joint; Δφ b This represents the reference phase difference between adjacent joint segments;

[0025] S12, The reinforcement learning agent obtains the state information s output from the snake head component, including the current speed and posture of the snake robot, the relative position and distance of the target or obstacle, water flow information, and current CPG parameters;

[0026] S13. Establish a reward function R(t) to guide the learning and decision-making process of the reinforcement learning agent;

[0027] S14. The reinforcement learning agent outputs the adjustment amount of the CPG parameters, as well as whether to activate the jet propulsion module and its propulsion level, based on the input state information s.

[0028] The adjustment amount includes the adjustment amount ΔA for the amplitude. RL (t) Adjustment amount Δf for frequency RL (t), and the adjustment amount Δ(Δφ) for the phase difference between adjacent joints. RL (t);

[0029] S15. After receiving the adjustment amount output by the reinforcement learning agent, the central pattern generator module updates the CPG parameters and recalculates the expected joint angle θ for each joint segment. i (t), the expected joint angle θ of the i-th motor joint at time t generated by the central pattern generator module. i (t) is determined by the following formula:

[0030] θ i(t)=A(t)sin(2πf(t)t+(i-1)Δφ(t))

[0031] Where A(t) is the real-time amplitude after adjustment by the RL agent; f(t) is the real-time frequency after adjustment by the RL agent; Δφ(t) is the real-time phase difference between adjacent joints after adjustment by the RL agent;

[0032] S16. The updated desired joint angle command is sent to each motor joint for execution via the CAN bus.

[0033] Furthermore, the establishment of the reward function R(t) for guiding the learning and decision-making process of the reinforcement learning agent includes:

[0034] Let d(t) be the distance between the robot and the target at time t. p (t) represents the distance to the target at the previous moment, ψ e (t) represents the error between the robot's current orientation and the target orientation. A reward component R is established to encourage the robot to move towards the target and maintain the correct orientation, thus approaching the target. t (t), the expression is:

[0035] R t (t)=k d ·(d p (t)-d(t))-k o ·|ψ e (t)|+C a

[0036] In the formula, k d It is the reward factor for distance reduction; k o It is the penalty factor for the direction of error; C a A reward for reaching the goal;

[0037] Let v b (t) represents the rate of change of the linear velocity of the robot body, ω b (t) represents the rate of change of the robot's angular velocity, where r(t) and p(t) are the current roll and pitch angles, respectively. d and p d For the desired stable roll and pitch angles, establish a motion stability-related reward component R to encourage smooth robot movement and punish violent swaying or attitude instability. s (t), the expression is:

[0038]

[0039] In the formula, k l k a k r k pThis is the corresponding penalty coefficient;

[0040] Let τ j (t) represents the output torque of the j-th joint motor at time t. For its angular velocity, P j (t) represents the instantaneous power consumption of the jet propulsion module at time t. An energy consumption-related reward component R is established to penalize excessive motor energy consumption and jet propulsion energy consumption. e (t), the expression is:

[0041]

[0042] In the formula, k m and k j It is the penalty coefficient for motor energy consumption and injection energy consumption;

[0043] Let C d (t) represents the collision detection flag, d o (t) represents the distance between the robot and the nearest obstacle, d s To establish a safe distance threshold, a collision and obstacle avoidance-related reward component R is created to severely punish collision behavior and encourage maintaining a safe distance from obstacles. c (t), the expression is:

[0044] R c (t)=-C p ·C d (t)+k c ·max(0,d o (t)-d s )

[0045] In the formula, C p k is the collision penalty value. c It is a reward factor that encourages maintaining a safe distance;

[0046] Based on each reward component R t (t), R s (t), R e (t), R c The weighted sum of (t) is used to construct the reward function R(t), which is expressed as:

[0047] R(t)=ω t ·R t (t)+ω s ·R s (t)+ω e ·R e (t)+ω t ·R c (t)

[0048] In the formula, ω t ωs ω e ω t These are the weighting coefficients for each reward component.

[0049] Furthermore, when the reinforcement learning agent directly outputs the desired torque vector of each motor joint, it is implemented using a DDPG module based on a deep deterministic policy gradient algorithm. The DDPG module includes:

[0050] An Actor network μ(s|θ) μ The input is state information s, and the output is a definite action a. t =μ(s|θ) μ ), the action a here t It can be directly the desired torque vector τ(t) of each motor joint, θ μ These are the parameters of the Actor network;

[0051] A Critic network Q(s,a) t |θ Q The input is the state information s and the action a output by the Actor network. t The output is the state-action value Q(s,a). t ), used to evaluate the effectiveness of the current strategy, θ Q These are the parameters of the Critic network;

[0052] The Actor network and Critic network update their parameters by sampling data from the experience replay pool and according to the update rules of the DDPG algorithm.

[0053] Furthermore, the reinforcement learning agent directly outputs the desired torque vector for each motor joint, including:

[0054] S21. Initialize the Actor network μ(s|θ) of the DDPG module. μ ), Critic network Q(s,a t |θ Q ) and its corresponding target network parameters;

[0055] S22. The reinforcement learning agent acquires state information s from the snake head component, including the snake robot’s current speed and posture, the relative position and distance of the target or obstacle, water flow information, and current CPG parameters.

[0056] S23, the Actor network of the DDPG module, based on the current state information s t Output the desired torque vector τ(t) = μ(s) for each joint t |θ μ ), and control commands for the jet propulsion module;

[0057] S24. The desired torque vector τ(t) is sent directly to each motor joint via the CAN bus for execution.

[0058] By employing the above technical solution, the present invention provides an underwater biomimetic snake robot and its hybrid adaptive control method, which has at least the following beneficial effects:

[0059] 1. The snake-like robot proposed in this invention is suitable for tasks such as pipeline inspection and underwater exploration. Each segment is connected by joints to form a modular segment structure that can realize multi-degree-of-freedom wave motion. The modular design reduces manufacturing costs and maintenance difficulty, and the advanced control method improves the robot's intelligence level.

[0060] 2. The proposed hybrid adaptive control method can autonomously learn and optimize its motion strategy to adapt to the complex and ever-changing underwater environment, and coordinate body undulation and jet propulsion, thereby significantly improving its maneuverability, environmental adaptability and mission execution efficiency. Attached Figure Description

[0061] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0062] Figure 1 This is a schematic diagram of the snake-like robot in Embodiment 1 of the present invention;

[0063] Figure 2 This is a schematic diagram of the snake-body component in Embodiment 1 of the present invention;

[0064] Figure 3 This is a schematic diagram of the waterproof structure in Embodiment 1 of the present invention;

[0065] Figure 4 This is a schematic diagram of the internal structure of the jet propulsion module in Embodiment 1 of the present invention;

[0066] Figure 5 This is a block diagram illustrating the principle of the hybrid adaptive CPG-RL control method in Embodiment 2 of the present invention.

[0067] Figure 6 This is a flowchart illustrating the modulation of CPG parameters by an RL agent in Embodiment 2 of the present invention;

[0068] Figure 7 This is a flowchart of motor torque control using the DDPG algorithm in Embodiment 2 of the present invention.

[0069] In the picture:

[0070] 1. Snake body assembly; 11. Motor joint; 12. Upper rotary joint; 13. Lower rotary joint; 14. Waterproof plug;

[0071] 2. Snake head component;

[0072] 3. Jet propulsion module; 31. Miniature water pump; 32. Water storage bladder. Detailed Implementation

[0073] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. This will allow for a full understanding of how the present application uses technical means to solve technical problems and achieve technical effects, and to facilitate its implementation.

[0074] Example 1

[0075] The primary objective of this embodiment is to provide a biomimetic snake-like robot, aiming to solve the following technical challenges:

[0076] Complex structure and difficult manufacturing: Existing snake robots mostly adopt a highly modular structure, with complex segment connections and transmission systems, resulting in high processing and assembly costs, and poor reliability and maintainability.

[0077] Limited propulsion efficiency: In high-velocity or turbulent environments, the existing serpentine propulsion method suffers from significant energy loss and low power transmission efficiency, making it impossible to achieve stable operation over a long period of time.

[0078] Insufficient flexibility and controllability: Some design schemes are not yet perfect in biomimetic wave patterns, have poor motion trajectory control accuracy, and are difficult to adapt to unstructured environments.

[0079] Low integration: Most underwater snake robots have scattered functional modules, low space utilization, and cannot meet the system integration requirements for multi-task execution.

[0080] Please refer to Figures 1-4 This embodiment addresses the aforementioned technical challenges by proposing an underwater biomimetic snake-like robot, comprising a snake body component 1, a snake head component 2, and a tail jet propulsion module 3, as shown below. Figure 1 As shown. The snake-body component 1 is composed of multiple modular joint segments connected in series, and multi-degree-of-freedom undulating motion of the snake-body component 1 is achieved by rotating these joint segments. Figure 2As shown, the joint segment includes a motor joint 11 and upper and lower rotary joints 12 and 13, which are threadedly fixed to both sides of one end of the motor joint 11. The motor joint 11 is sequentially connected to the upper and lower rotary joints 12 of another joint segment to form a multi-directionally swinging snake body assembly 1. Multiple snake body assemblies 1 are sequentially connected to form the body of the snake robot. A waterproof plug 14 and a motor are installed inside the cavity of the motor joint 11. The rotation speed and angle of the motor are independently adjusted by the ARM control board through the CAN bus protocol to realize the forward movement, turning, and posture adjustment of the snake robot.

[0081] Each joint segment is connected by joints to form a modular body structure capable of multi-degree-of-freedom undulating motion. The drive module is located inside each joint segment, using a combination of motors to achieve precise angle control and synchronized segmental oscillation. Therefore, it is suitable for tasks such as pipeline inspection and underwater exploration. The modular design reduces manufacturing costs and maintenance difficulty, while advanced control methods improve the robot's intelligence level.

[0082] Specifically, each motor within motor joint 11 drives the upper rotary joint 12 and the lower rotary joint 13 to reciprocate left and right, thereby driving the movement of the snake body assembly 1 to simulate the undulating motion of a snake. The movement state of the snake robot is controlled by adjusting the motion parameters of each motor, such as rotation speed and rotation angle. The forward movement and direction changes of the snake robot are controlled by adjusting the rotation speed and rotation angle of the front and rear motors. For example, controlling the front motor to rotate at a low speed and small angle, and the rear motor to rotate at a high speed, propels the snake robot forward; adjusting the rotation angle of the motors on both sides completes the turning of the snake robot.

[0083] The motors are connected to the sensing device and another motor in joint 11 via a CAN bus. The lower-level machine of the snake robot is an ARM board, which controls the motors. The control method is through the USB serial port on the ARM board, which is connected to the USB-to-CAN module via a DuPont wire-to-USB female connector. The USB-to-CAN module is directly linearly connected to the motors via a data cable, thus enabling a single ARM board and module to control a group of motors. The ARM board sends 16-bit data frames to the module via motor control code. The module then verifies and converts the data frames, converting them into 8-bit CAN signal frames. The CAN signal frames are further divided into transmit frames and feedback frames. After receiving the CAN transmit frame, the motor makes corresponding control changes and returns a CAN feedback frame to the module in real time. The module then converts the 8-bit CAN signal frame back into a 16-bit data frame and returns it to the ARM board for parsing. This allows the ARM board to understand the position, speed, torque, temperature, and drive temperature of each motor. To achieve precise control of a single motor using data frames, Data4 to Data7 of the 16-bit data frame are involved. Data4 to Data7 contains CAN... Each motor has a pre-set CAN ID for control purposes. When a motor receives a data frame, it checks the CAN ID for similarities and differences to make further responses, thereby achieving precise control.

[0084] Motor joint 11 has a built-in motor and waterproof structure, such as Figure 3 As shown, the waterproof structure includes a waterproof plug 14 disposed in the inner cavity of the motor joint 11. The motor is fixed in the motor joint by a screw at the bottom and the top is fixed to the waterproof plug 14 by screws. A rubber sheet is added to the rotor part of the motor and connected to another rotating joint by screws. The outer ring of the waterproof plug 14 is provided with an O-ring rubber ring to provide static waterproofing. The inside of the waterproof plug 14 is provided with a double-lip stainless steel oil seal to provide dynamic sealing when the motor rotates. A rubber sheet is added to the rotor part of the motor to prevent water from seeping in.

[0085] This embodiment is made of a multi-layer sealing structure and waterproof materials. Waterproof rubber rings and pressure sealing components are provided at the connection between the shell and each section to prevent water from seeping into the internal electronic and drive systems, ensuring the sealing reliability and equipment safety during long-term underwater operation.

[0086] The snake head component 2 has a built-in camera and sensors, which are installed at one end of the snake body component 1 and used for environmental perception and obstacle avoidance.

[0087] like Figure 4As shown, a jet propulsion module 3 is integrated at the tail of the snake-body component 1, containing a micro-jet chamber to simulate the jet propulsion mechanism of an octopus. The jet propulsion module 3 includes a micro-water pump 31 and a water reservoir 32 integrated into a closed cavity and installed at the other end of the snake-body component 1. It generates reaction thrust by rapidly draining water, providing the snake robot with a short burst of propulsion. The micro-water pump 31 is connected to the water reservoir 32 and is controlled by an ARM control board to start and stop. It generates instantaneous thrust by rapidly draining water, which, in conjunction with the multi-degree-of-freedom undulating motion of the snake-body component 1, enables multimodal maneuvering.

[0088] Specifically, the micro water pump 31 generates reaction thrust through rapid drainage. The micro water pump 31 is started and stopped by the ARM board for short-term acceleration or emergency obstacle avoidance, providing explosive propulsion and enabling multimodal maneuvering such as cruising and sprinting in conjunction with wave motion. In some embodiments, such as emergency pipeline inspection, when the snake robot encounters a collapse or obstacle in the pipeline, the jet propulsion module 3 can be activated to quickly escape the danger zone in a narrow space.

[0089] The torso of the snake body component 1 is modularly designed and consists of several joint segments connected in series. Each joint segment is structurally independent, which facilitates modular production and maintenance. The head camera of the snake head component 2 enables precise positioning and environmental perception, which promotes the successful completion of the task. Therefore, the snake robot proposed in this invention has the advantages of easy manufacturing and low cost, and effectively reduces manufacturing costs.

[0090] Example 2

[0091] This embodiment addresses the problems of existing underwater snake robots, such as poor adaptability in complex environments, insufficient optimization of motion patterns, and lack of intelligent coordination among multiple propulsion methods. Another objective of this invention is to provide an advanced control method applicable to an underwater snake robot comprising a modular multi-jointed snake body component 1, a snake head component 2 integrating environmental perception and obstacle avoidance functions, and a tail jet propulsion module 3.

[0092] Please refer to Figures 5-7 This embodiment proposes a hybrid adaptive control method for an underwater biomimetic snake-like robot, which includes the following steps:

[0093] A reinforcement learning (RL) agent is constructed and run, which receives environmental perception information and robot's own state information from the snake head component as state information s;

[0094] The reinforcement learning (RL) agent, based on the state information s and its learned policy (i.e., the established reward function mechanism), outputs control commands for each motor joint of the snake-body component and / or control commands for the jet propulsion module. The control commands for each motor joint include:

[0095] The desired joint angles are generated via the Central Pattern Generator (CPG) module. Alternatively, the reinforcement learning agent directly outputs the desired torque vectors for each motor joint. Please refer to [reference needed]. Figure 5 and Figure 6 The reinforcement learning (RL) agent outputs the adjustment amount of the CPG parameters to update the CPG parameters in real time and generate the final swing command to drive each motor joint. The CPG parameters include amplitude, frequency and phase difference between adjacent joints.

[0096] In this embodiment, the desired joint angle generated by the Central Pattern Generator (CPG) module includes:

[0097] S11. Using the Central Pattern Generator (CPG) module to generate initial rhythmic oscillation commands for the multi-joint snake body components of the snake robot, the initial joint angle θ of the i-th motor joint is then... i Determined by the following formula:

[0098] θ i =A b sin(2πf b t+φ i )

[0099] φ i =(i-1)Δφ b

[0100] In the formula, A b f is the reference amplitude for the joint oscillation of the motor; b φ is the reference frequency for the joint oscillation of the motor; t is time; φ i The phase lag of the i-th motor joint; Δφ b This represents the reference phase difference between adjacent joint segments. The calculation of these CPG parameters and the transmission of commands are achieved through the robot's ARM control board and CAN bus.

[0101] S12. The reinforcement learning (RL) agent acquires state information s from the snake head component, including the snake robot's current speed and posture, the relative position and distance of the target or obstacle, water flow information (if there are corresponding sensors), current CPG parameters, etc.

[0102] S13. Establish a reward function R(t) to guide the learning and decision-making process of a reinforcement learning (RL) agent. Its design aims to balance multiple control objectives. The reward function R(t) can be defined as a weighted sum of the rewards for each component, including:

[0103] Let d(t) be the distance between the robot and the target at time t. p (t) represents the distance to the target at the previous moment, ψ e(t) represents the error between the robot's current orientation and the target orientation. A reward component R is established to encourage the robot to move towards the target and maintain the correct orientation, thus approaching the target. t (t), the expression is:

[0104] R t (t)=k d ·(d p (t)-d(t))-k o ·|ψ e (t)|+C a

[0105] In the formula, k d It is the reward factor for distance reduction; k o It is the penalty factor for the direction of error; C a This is the reward for reaching the goal. When the robot reaches the target area, a large positive reward C can be given. a .

[0106] Let v b (t) represents the rate of change of the linear velocity of the robot body (acceleration / jerk), ω b (t) represents the rate of change of the robot's angular velocity, where r(t) and p(t) are the current roll and pitch angles, respectively. d and p d For the desired stable roll and pitch angles, establish a motion stability-related reward component R to encourage smooth robot movement and punish violent swaying or attitude instability. s (t). To encourage smooth robot movement and avoid violent shaking or instability, the expression is:

[0107]

[0108] In the formula, k l k a k r k p This represents the corresponding penalty coefficient.

[0109] Let τ j (t) represents the output torque of the j-th joint motor at time t. For its angular velocity, P j (t) represents the instantaneous power consumption of the jet propulsion module at time t. A reward component R, related to energy consumption (such as motor torque and jet consumption), is established to penalize excessive motor energy consumption and jet propulsion energy consumption. e (t), the expression is:

[0110]

[0111] In the formula, k m and kj It is the penalty coefficient for motor energy consumption and injection energy consumption.

[0112] Let C d (t) is the collision detection flag, a Boolean value that is 1 when a collision is detected and 0 otherwise; d o (t) represents the distance between the robot and the nearest obstacle, d s To establish a safe distance threshold, a collision and obstacle avoidance-related reward component R is created to severely punish collision behavior and encourage maintaining a safe distance from obstacles. c (t). Severely punish collisions and encourage maintaining a safe distance from obstacles, expressed as:

[0113] R c (t)=-C p ·C d (t)+k c ·max(0,d o (t)-d s )

[0114] In the formula, C p The collision penalty value is a very large positive number used as the penalty; k c It is a reward factor that encourages maintaining a safe distance.

[0115] Based on each reward component R t (t), R s (t), R e (t), R c The weighted sum of (t) is used to construct the reward function R(t), which is expressed as:

[0116] R(t)=ω t ·R t (t)+ω s ·R s (t)+ω e ·R e (t)+ω t ·R c (t)

[0117] In the formula, ω t ω s ω e ω t These are the weighting coefficients for each reward component.

[0118] S14. The reinforcement learning (RL) agent, based on the input state information s, outputs adjustments to the CPG parameters, as well as whether to activate the jet propulsion module and its propulsion level; the adjustments include the adjustment amount ΔA to the amplitude. RL (t) Adjustment amount Δf for frequency RL(t), and the adjustment amount Δ(Δφ) for the phase difference between adjacent joints. RL (t);

[0119] S15. After receiving the adjustment amount output by the reinforcement learning (RL) agent, the Central Pattern Generator (CPG) module updates the CPG parameters and recalculates the expected joint angle θ for each joint segment. i (t), the expected joint angle θ of the i-th motor joint at time t generated by the Central Pattern Generator (CPG) module. i (t) is determined by the following formula:

[0120] θ i (t)=A(t)sin(2πf(t)t+(i-1)Δφ(t))

[0121] Where A(t) is the real-time amplitude adjusted by the RL agent; f(t) is the real-time frequency adjusted by the RL agent; and Δφ(t) is the real-time phase difference between adjacent joints adjusted by the RL agent. The updated CPG parameters A(t), f(t), and Δφ(t) are derived from their base values ​​A. b f b , Δφ b It is obtained by adding the corresponding adjustment amount.

[0122] S16. The updated desired joint angle command is sent to each motor joint for execution via the CAN bus.

[0123] In this embodiment, for the control of continuous motion spaces such as motor torque, the Deep Deterministic Policy Gradient (DDPG) algorithm based on the Actor-Critic framework can be used. Please refer to... Figure 7 The DDPG module contains an Actor network (policy network) and a Critic network (value network).

[0124] Actor network μ(s|θ) μ The input is state information s, and the output is a definite action a. t =μ(s|θ) μ ), the action a here t It can be directly the desired torque vector τ(t) of each motor joint, θ μ These are the parameters of the Actor network.

[0125] Critic network Q(s,a) t |θ Q The input is the state information s and the action a output by the Actor network. t The output is the state-action value Q(s,a).t ), used to evaluate the effectiveness of the current strategy, θ Q These are the parameters of the Critic network.

[0126] The Actor network and Critic network update their parameters by sampling data from the experience replay pool and according to the specific update rules of the DDPG algorithm.

[0127] 1. Critic Network Update: The Critic Network updates by minimizing the loss function L(θ). Q The loss function is typically based on the Bellman equation, where the target y is updated. t Defined as:

[0128] y t =r t +γQ′(s t+1 ,μ'(s t+1 |θ μ′ )|θ Q′ )

[0129] Where, r t It is an instant reward; s t+1 It represents the state information at the next moment; γ is the discount factor; θ μ′ θ Q′ ...

[0130]

[0131] Where D is the experience replay pool; Represents the expected value, (s) t ,a t ,r t ,s t+1 ) represents an empirical tuple, and ~D represents an empirical tuple (s) t ,a t ,r t ,s t+1 () was obtained by sampling from the experience replay pool D. The overall meaning is the expected value calculated on a batch of experience samples, where each experience sample is randomly drawn from the experience replay pool D.

[0132] In this embodiment, the Critic network parameter update direction is... The negative direction. This indicates the relationship between the Critic network parameters θ. Q The gradient of the loss function L(θ).Q When ), it represents the gradient of the loss function with respect to the parameters of the Critic network, indicating the direction and rate of the fastest increase in the loss function value. The Critic network minimizes the loss function by adjusting its parameters along the negative direction of this gradient.

[0133] 2. Actor Network Update: The Actor network updates by maximizing the expected reward, and its policy gradient... It can be approximated as:

[0134]

[0135] Where s = s t The expectation is for state s. t It is calculated as a = μ(s) t |θ μ ) represents the state s given by the state s t Below, action a is determined by the current Actor network (policy network) μ(s). t |θ μ It is determined by ) The expected value represents the state s sampled from the experience replay pool. t The calculation is performed, and in these states, action a is based on the current Actor network's policy μ(s). t |θ μ It is certain.

[0136] This indicates that the gradient is calculated relative to action 'a'. This represents the Q-value (i.e., the state-action value Q(s,a)) output by the Critic network. t The gradient relative to action a. It measures the rate and direction of change of the Q-value with respect to action a in state s.

[0137] This indicates that the gradient is relative to the parameters θ of the Actor network. μ Calculated This represents the action output by the Actor network (policy) relative to its own parameters θ. μ The gradient of the Actor network. It measures the change in the parameters θ of the Actor network if the parameters θ are slightly altered. μ How will the output action change?

[0138] In this embodiment, the Actor network parameter update direction is the policy gradient. The positive direction.

[0139] 3. Target network update: Target network parameters θ Q′ and θ μ′The parameters are slowly replicated from the main network using a "soft" update method, expressed as:

[0140] θ Q′ ←τ s θ Q +(1-τ s )θ Q′

[0141] θ μ′ ←τ s θ μ +(1-τ s )θ μ′

[0142] Where, τ s <<1 is a small update coefficient.

[0143] Through the DDPG module, the reinforcement learning (RL) agent can directly learn and output the desired torque for each motor joint, thereby achieving precise control of the robot's motion. These desired torques are then sent to the motors for execution via the CAN bus.

[0144] In this embodiment, the reinforcement learning agent directly outputs the desired torque vector for each motor joint, including:

[0145] S21. Initialize the Actor network μ(s|θ) of the DDPG module. μ ), Critic network Q(s,a t |θ Q ) and its corresponding target network parameters.

[0146] S22. The reinforcement learning (RL) agent acquires state information s from the snake head component, including the snake robot's current speed and posture, the relative position and distance of the target or obstacle, water flow information (if there are corresponding sensors), current CPG parameters, etc.

[0147] S23, the Actor network of the DDPG module, based on the current state information s t Output the desired torque vector τ(t) = μ(s) for each joint t |θ μ ), and control commands for the jet propulsion module.

[0148] S24. The desired torque vector τ(t) is sent directly to each motor joint via the CAN bus for execution.

[0149] Furthermore, the hybrid control operation flow in this embodiment is as follows:

[0150] 1. Reinforcement learning (RL) agents obtain current state information s from robot sensors.

[0151] 2. If the DDPG module is used to directly control the torque: The Actor network of the reinforcement learning (RL) agent outputs the expected torque vector τ(t) of each joint and the injection module command according to the current state s.

[0152] If the RL agent adjusts the CPG parameters: The reinforcement learning (RL) agent outputs an action based on its learned policy: the adjustment amount ΔA of the CPG parameters. RL (t),Δf RL (t),Δ(Δφ) RL (t) and control commands to the jet propulsion module.

[0153] 3. If the RL agent adjusts the CPG parameters: The CPG module updates its parameters based on the output of the reinforcement learning (RL) agent.

[0154] A(t)=A b +ΔA RL (t)

[0155] f(t) = f b +Δf RL (t)

[0156] Δφ(t)=Δφ b +Δ(Δφ) RL (t)

[0157] Among them, A b ,f b ,Δφ b Based on or preset parameter values.

[0158] 4. If the RL agent adjusts the CPG parameters: the updated CPG model generates the target angle command θ for each joint. i (t), at which point the joint angle is:

[0159] θ i (t)=(A b +ΔA RL (t))sin(2π(f b +Δf RL (t))t+(i-1)(Δφ b +Δ(Δφ) RL (t)))

[0160] These target angle commands can then be converted into motor torque.

[0161] 5. If DDPG is used to directly control torque: the desired torque τ(t) is sent to the motors of each snake-body component via the CAN bus.

[0162] If the reinforcement learning (RL) agent adjusts the CPG parameters: the target angle command is sent to the motors of each snake body component via the CAN bus for execution.

[0163] 6. The jet propulsion module starts or stops according to the instructions of the RL agent, and works in coordination with body undulations to achieve more efficient or more flexible maneuvering.

[0164] Based on the robot's behavior in the environment (e.g., whether it gets closer to the target point, whether it successfully avoids obstacles, the amount of energy consumed, the stability of its swimming posture, etc.), the immediate reward r is calculated according to the preset reward function R(t). t Reinforcement learning (RL) agents will apply experience (s) t ,a t ,r t ,s t+1 The data is stored in the experience replay pool D. Mini-batch data is sampled from D, and the network parameters are updated according to the RL algorithm.

[0165] By employing the hybrid adaptive control method proposed in this invention, the underwater biomimetic snake robot also possesses the following outstanding advantages:

[0166] Through reinforcement learning, robots can dynamically adjust their motion parameters based on real-time environmental feedback and task requirements, autonomously adapting to unknown or changing underwater environments, such as different water flow speeds and obstacle distributions.

[0167] By learning to optimize CPG parameters or directly optimizing motor torque, and coordinating jet propulsion strategies, RL agents can achieve more efficient swimming, higher speeds, more agile steering, and more stable attitude control.

[0168] It achieves intelligent coordination between body undulation propulsion and jet propulsion. The RL agent can learn when and how to combine these two propulsion modes to meet different mission requirements, such as activating jet propulsion when a rapid sprint or emergency obstacle avoidance is needed, while relying mainly on body undulation to conserve energy during cruising.

[0169] Through a carefully designed reward function, the robot can learn to achieve complex and advanced task objectives (such as path tracking, target search, and underwater structure inspection), rather than simply executing a predetermined sequence of actions, thus improving the robot's level of intelligence.

[0170] It should be noted that the underwater biomimetic snake robot has core components that enable it to function normally. These core components are essential for its normal operation and are not innovative in this application. They are common knowledge and those skilled in the art can conceive of the specific structure and layout.

[0171] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0172] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. Since the above embodiments are substantially similar to the method embodiments, their descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0173] The above embodiments provide a detailed description of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A hybrid adaptive control method for an underwater biomimetic snake-like robot, characterized in that, include: The snake body component (1) is composed of multiple modular joint segments connected in series, and the multi-degree-of-freedom wave motion of the snake body component (1) is realized by rotating the joint segments; The snake head assembly (2), which has a built-in camera and sensors, is installed at one end of the snake body assembly (1) and is used for environmental perception and obstacle avoidance; The jet propulsion module (3) includes a micro water pump (31) and a water storage bladder (32) integrated into a closed cavity and installed at the other end of the snake body assembly (1). It generates reaction thrust by rapidly draining water, providing short bursts of propulsion for the snake robot. The hybrid adaptive control method applied to the underwater biomimetic snake robot includes the following steps: A reinforcement learning agent is constructed and run, which receives environmental perception information and the robot's own state information from the snake head component as state information. ; Reinforcement learning agents based on state information Based on the learned strategies, output control commands for each motor joint of the snake body component and / or control commands for the jet propulsion module; The control commands for each motor joint include: The desired joint angle is generated by the central pattern generator module, wherein the reinforcement learning agent outputs the adjustment amount of the CPG parameters to update the CPG parameters in real time and generate the final swing command to drive each motor joint. The CPG parameters include amplitude, frequency and phase difference between adjacent joints. Alternatively, the reinforcement learning agent can directly output the desired torque vector for each motor joint; The desired joint angle generated by the central pattern generator module includes: S11. Using the central pattern generator module to generate initial rhythmic swing commands for the multi-joint snake body components of the snake robot, then the... Initial joint angles of each motor joint Determined by the following formula: ； In the formula, This is the reference amplitude for the joint oscillation of the motor; This is the reference frequency for the joint oscillation of the motor; For time; For the first Phase lag of each motor joint; This represents the reference phase difference between adjacent joint segments; S12. The reinforcement learning agent obtains state information from the snakehead component. This includes the snake robot’s current speed and attitude, the relative position and distance of the target or obstacle, water flow information, and current CPG parameters; S13. Establish a reward function to guide the learning and decision-making process of the reinforcement learning agent. ; S14. The reinforcement learning agent, based on the input state information... It outputs the adjustment amount of the CPG parameters, as well as whether to activate the jet propulsion module and its propulsion level; The adjustment includes the adjustment to the amplitude. Adjustment amount of frequency and the adjustment amount of the phase difference between adjacent joints. ; S15. After receiving the adjustment amount output by the reinforcement learning agent, the central pattern generator module updates the CPG parameters and recalculates the expected joint angles of each joint segment. The first generated by the central pattern generator module Each motor joint in time Expected joint angle Determined by the following formula: ； in, This is the real-time amplitude after adjustment by the reinforcement learning agent; The real-time frequency adjusted by the reinforcement learning agent; The real-time phase difference between adjacent joints is adjusted by the reinforcement learning agent; S16. The updated desired joint angle command is sent to each motor joint for execution via the CAN bus.

2. The hybrid adaptive control method according to claim 1, characterized in that, Each joint segment of the snake body assembly (1) includes a motor joint (11) and a rotary joint (12, 13) threadedly fixed to both ends of the motor joint (11). The motor joint (11) is sequentially linked with the rotary joint (12, 13) of another joint segment to form a multi-directionally swinging snake body assembly (1).

3. The hybrid adaptive control method according to claim 2, characterized in that, The motor joint (11) has a built-in motor and a waterproof structure. The waterproof structure includes a waterproof plug (14) set in the inner cavity of the motor joint (11). The outer ring of the waterproof plug (14) is provided with an O-ring rubber ring that provides static waterproofing. The inside of the waterproof plug (14) is provided with a double-lip stainless steel oil seal that provides dynamic sealing when the motor rotates. The rotor part of the motor is equipped with a rubber sheet to prevent water from seeping in.

4. The hybrid adaptive control method according to claim 3, characterized in that, The rotation speed and angle of the motor are independently adjusted by the ARM control board via the CAN bus protocol to enable the snake robot to move forward, turn, and adjust its posture.

5. The hybrid adaptive control method according to claim 1, characterized in that, The micro water pump (31) is connected to the water storage bladder (32) and is controlled by the ARM control board to start and stop. It generates instantaneous thrust by rapidly draining water, and achieves multimodal maneuvering in conjunction with the multi-degree-of-freedom wave motion of the snake body component (1).

6. The hybrid adaptive control method according to claim 1, characterized in that, The establishment of a reward function to guide the learning and decision-making process of a reinforcement learning agent. ,include: set up For robots in time Distance to target This represents the distance from the target at the previous moment. For the error between the robot's current orientation and the target orientation, establish a reward component that encourages the robot to move towards the target and maintain the correct orientation, as well as a reward component that approaches the target. The expression is: ； In the formula, It is a reward factor for distance reduction; It is a penalty factor for errors. A reward for reaching the goal; set up The rate of change of the linear velocity of the robot body. The rate of change of the robot's angular velocity. and These are the current roll angle and pitch angle, respectively. and To achieve the desired stable roll and pitch angles, a motion stability-related reward component is established to encourage smooth robot movement and penalize violent swaying or attitude instability. The expression is: ； In the formula, This is the corresponding penalty coefficient; set up For the first Each joint motor in time 'output torque' Its angular velocity, For the jet propulsion module in time The instantaneous power consumption is used to establish an energy consumption-related reward component to penalize excessive motor energy consumption and jet propulsion energy consumption. The expression is: ； In the formula, and It is the penalty coefficient for motor energy consumption and injection energy consumption; set up For collision detection, The distance between the robot and the nearest obstacle. To establish a safe distance threshold, a reward component related to collision and obstacle avoidance is created to severely punish collision behavior and encourage maintaining a safe distance from obstacles. The expression is: ； In the formula, This is the collision penalty value; It is a reward factor that encourages maintaining a safe distance; Based on each reward component Weighted sum to establish reward function The expression is: ； In the formula, These are the weighting coefficients for each reward component.

7. The hybrid adaptive control method according to claim 6, characterized in that, When the reinforcement learning agent directly outputs the desired torque vectors of each motor joint, the DDPG module based on the deep deterministic policy gradient algorithm is used. The DDPG module includes: An Actor Network The input is status information. The output is a definite action. The actions here It is directly the desired torque vector of each motor joint. , These are the parameters of the Actor network; A Critic network The input is status information. Actions output by the Actor network The output is a state-action value. Used to evaluate the effectiveness of the current strategy. These are the parameters of the Critic network; The Actor network and Critic network update their parameters by sampling data from the experience replay pool and according to the update rules of the DDPG algorithm.

8. The hybrid adaptive control method according to claim 7, characterized in that, The reinforcement learning agent directly outputs the desired torque vector for each motor joint, including: S21. Initialize the Actor network of the DDPG module. Critic Network and its corresponding target network parameters; S22. The reinforcement learning agent obtains state information output from the snakehead component. This includes the snake robot’s current speed and attitude, the relative position and distance of the target or obstacle, water flow information, and current CPG parameters; The Actor network of the S23 and DDPG modules is based on the current state information. Output the desired torque vector for each joint. , and control commands to the jet propulsion module; S24. The desired torque vector... The data is sent directly to each motor joint via the CAN bus for execution.

Images