Construction method and control method of underwater robot cooperative control system

By constructing a collaborative control system for underwater robots and employing deep reinforcement learning and a lightweight Transformer model, the limitations of traditional underwater robot control systems in collaborative control in complex environments have been overcome, enabling efficient and safe underwater operations.

CN118915797APending Publication Date: 2024-11-08HOHAI UNIV +2
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202410970653.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-19
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Traditional underwater robot control systems struggle to achieve precise control in complex and dynamic underwater environments. In particular, the collaborative work between buoys, winches, and underwater robots is limited, failing to fully consider real-time changes and nonlinear factors in complex underwater scenarios.

Method used

A collaborative control system for underwater robots is constructed. An umbilical cable simulation model is built by simulating the environment. Combining buoy following strategy and winch control strategy, the SAC algorithm of deep reinforcement learning and a lightweight Transformer model are used to train the buoy following and winch control strategy models to achieve intelligent and adaptive collaborative control.

Benefits of technology

It improves the operational efficiency and safety of underwater robots in complex environments, enhances the stability and adaptability of the system, achieves wide applicability to various underwater scenarios, reduces cable vibration, and ensures the safety and efficiency of collaborative control.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118915797A_ABST
    Figure CN118915797A_ABST
Patent Text Reader

Abstract

The invention provides a construction method of an underwater robot cooperative control system. The construction method comprises the following steps: constructing a simulation environment; constructing an umbilical cable simulation model; constructing a buoy following strategy model and performing training; constructing a winch control strategy model and performing training; combining the buoy control strategy model and the winch control strategy model into an underwater robot cooperative control system; wherein the buoy following strategy model is based on a deep reinforcement learning SAC algorithm, and efficient exploration and intelligent control of a buoy are achieved; the expected length of the umbilical cable is predicted in real time by using a winch control strategy model of a lightweight Transform through a behavior cloning method. On the other hand, the invention provides the cooperative control method for the underwater robot, the safety of the cooperative control system is ensured, and the operation efficiency is remarkably improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The invention relates to a construction method of an underwater robot cooperative control system and a control method thereof, belonging to the technical field of underwater robots. Background Art

[0002] An underwater robot is an extreme operation robot that works underwater and is an important tool for marine development and underwater operations. When the underwater robot is working, the shore unit connects the buoy and the underwater robot body in series through the umbilical cable on the retractable device. The umbilical cable provides power to the underwater robot and is responsible for signal transmission. When the underwater robot is working, the buoy floats on the water surface in a real-time following state, which facilitates the positioning of the underwater robot.

[0003] Underwater robots are increasingly used in complex and dynamic underwater environments, mainly in many fields such as dam inspection, underwater resource exploration and scientific research. However, with the continuous expansion of application scenarios and the increase in underwater tasks, traditional underwater robot control systems face a series of challenges. These challenges mainly include factors such as environmental noise, complex obstacles, and water flow speed. The high randomness of these factors makes it difficult to accurately model and achieve precise control through traditional control algorithms.

[0004] In the actual application scenarios of underwater operations, the coordinated work of underwater robots, buoys and umbilical cable winches is particularly critical. The buoy not only needs to effectively follow the movement of the underwater robot, but also maintain a reasonable tension in the umbilical cable to avoid the risk of cable entanglement or breakage. Traditional control systems have great limitations in the coordinated control between buoys, winches and underwater robots, and cannot fully consider the real-time changes and nonlinear factors in complex underwater scenes. Therefore, there is an urgent need for more intelligent and adaptive control systems to improve the performance of underwater robots in complex environments, enable them to perform tasks more flexibly, and overcome the limitations of traditional control algorithms in complex underwater environments. Summary of the invention

[0005] The purpose of the present invention is to overcome the deficiencies in the prior art and to provide a method for constructing an underwater robot collaborative control system and a control method thereof.

[0006] To achieve the above object, the present invention is implemented by adopting the following technical solutions: In the first aspect, the present invention provides a method for constructing a collaborative control system of an underwater robot, comprising: building a simulation environment according to the physical and dynamic characteristics of the underwater robot body, buoy, umbilical cable and winch; constructing an umbilical cable simulation model according to the characteristics of the underwater environment and taking into account the influence of gravity, buoy and underwater robot tension on the umbilical cable; constructing a buoy following strategy model, and according to the buoy following control requirements and the simulation results of the umbilical cable simulation model, using buoy following related historical experience data to train the buoy following strategy model in the simulation environment; constructing a winch control strategy model, and using winch control related historical experience data to train the winch control strategy model in the simulation environment; combining the trained buoy control strategy model with the trained winch control strategy model to complete the construction of the collaborative control system of the underwater robot.

[0007] Furthermore, the aforementioned construction of the umbilical cable simulation model includes: dividing the umbilical cable into a surface cable R AB and underwater cables R BC ; The underwater cable R BC Decomposed into horizontal cables R BC(A) , vertical cable R BC(L) ; Use the number of nodes to represent the surface cable R AB , horizontal cable R BC(A) , vertical cable R BC(L) Length, the deformation and displacement of the corresponding rope are determined according to the change of the number of nodes; Wherein, the surface cable R AB The first end is connected to a buoy and the end is connected to a winch to simulate the deformation and displacement of the umbilical cable on water; the cable R BC(A) The first end is connected to the underwater robot's water surface projection, and the end is connected to the buoy; used to simulate underwater cables R BC Horizontal deformation and displacement; the cable R BC(L) The first end is connected to the underwater robot's water surface projection, and the end is connected to the underwater robot to simulate the underwater cable R BC Deformation and displacement in the vertical direction.

[0008] Furthermore, the aforementioned construction of the buoy following strategy model includes: constructing a reward and punishment function model, performing rewards and punishments on the movement of the buoy and the underwater robot and the retraction and release of the umbilical cable, and calculating the reward and punishment values; A buoy following strategy is constructed based on the SAC algorithm. The simulated environmental state is used as the input of the buoy following strategy. Combined with the reward and punishment values, the distribution probability of the buoy action is calculated, and the buoy action with the largest distribution probability value is selected as the output of the buoy following strategy.

[0009] Furthermore, the aforementioned construction of the reward and punishment function model includes: based on the principle that the greater the difference between the horizontal coordinates of the buoy and the horizontal coordinates of the underwater robot, the smaller the basic reward value, a basic reward function is constructed to reflect the following effect of the buoy and the underwater robot, and the basic reward is calculated; based on the principle that if the distance between the buoy and the underwater robot is smaller, an incremental additional reward point is given, and if the buoy deviates from the underwater robot, the reward point is cleared, a reward point calculation function is constructed to calculate the reward point; based on the principle that the surface cable is moved away from the underwater robot, the reward point is calculated. R AB and underwater cables R BC Based on the principle that length has negatively correlated rewards, a rope jitter penalty function is introduced to calculate the rope jitter penalty reward; the sum of the basic reward, bonus points, and rope jitter penalty reward is calculated as the reward and punishment value.

[0010] Furthermore, the expression of the aforementioned basic reward function is: ; in, For basic return, is the Manhattan distance, is the horizontal coordinate of the buoy, is the horizontal coordinate of the underwater robot; The expression of the reward point calculation function is as follows: ; in, for Bonus points for the moment; for Bonus points for the moment; is the reward constant, ; for Simulate the environment status at all times; for Simulate the environment status at all times; for Moment buoy action; for Moment buoy action; is the buoy speed; The expression of the rope jitter penalty function is as follows: ; in, is the Euclidean distance.

[0011] Furthermore, the aforementioned buoy following strategy model includes: value function network , target value function network , Q-value function network and actor networks The method of using the buoy following related historical experience data to train the buoy following strategy model in the simulation environment includes: Collect historical experience data of simulated environmental conditions and corresponding buoy actions; Set the target value update frequency , target reward and penalty value ; Iteratively update the value function network , target value function network , Q-value function network and actor networks , until the reward or penalty value is greater than , stop iteration; The value function network updating method comprises: calculating the update target of the value function network : ; Value Function Network The updated gradient is: ; In the formula, To simulate the environment The policy vector of the buoy action output by the actor network; To simulate the environment Q-value function network Output Q value vector; is the temperature coefficient, T is the transposition operation; is the mathematical expectation; D is the historical empirical data of the simulated environmental state and the corresponding buoy action; To simulate the environment Update the target with the state value below; To simulate the environment The state value output by the down-value function network; The target value function network updating method comprises: By frequency Will The parameters of each layer of the network are directly assigned to The network parameters of each layer in are used to update the network parameters of the target value function network; The Q-value function network and actor networks The update methods include: The update gradient of the Q-value function network is: ; Actor Network Update the gradient to: ; In the formula, To simulate the environmental state-buoy action Q-value function network The Q value of the output; To simulate the environmental state-buoy action The reward and penalty values ​​returned by the next environment, To simulate the environment The state value output by the target value function; To simulate the environmental state-buoy action The state transition probability of the next environment; is the discount factor, a constant with a value of (0, 1).

[0012] Furthermore, in the simulation environment, using the buoy following related historical experience data to train the buoy following strategy model also includes updating the temperature coefficient , temperature coefficient The updated gradient is: ; In the formula, is a constant vector.

[0013] Furthermore, a winch control strategy model is constructed based on the behavioral cloning algorithm. The simulated environmental state and the depth of the underwater robot are used as the input of the winch control strategy model, and the umbilical cable length is used as the output of the winch control strategy model. A lightweight Transformer model is used to learn the mapping from input to output.

[0014] Furthermore, the aforementioned lightweight Transformer model includes an embedding layer, a multi-head attention layer, a feedforward network, and a fully connected layer; the winch control strategy model is trained using winch control related historical experience data in the simulation environment, including: Collect historical experience data on the simulated environment status, the depth of the underwater robot, and the corresponding umbilical cable length; Input sequence , After being encoded by the embedding layer, a time-step-based position code is added and input into the multi-head attention layer; the output of the multi-head attention layer is superimposed with the position code and output as the attention feature; the attention feature is layer-normalized and input into the feedforward network; the input and output of the feedforward network are superimposed and input into the fully connected layer, and the expected umbilical cable length is finally output; the input sequence for Time has come All simulation environment states at the moment , the depth of underwater robots ,as well as Time has come Length of all umbilical cables at the moment Combination according to The umbilical cable length at the moment and the expected umbilical cable length are used to train the network parameters of the winch control strategy model; The update gradient of the winch control strategy model is : ; in, is the network parameter of the winch control strategy model; B is the historical experience data of the simulated environment state, the depth of the underwater robot, and the corresponding umbilical cable length; For the input sequence The expected umbilical cable length output by the winch control strategy model when for Umbilical cable length at the moment; The historical experience data is input repeatedly in batches to train the network parameters of the winch control strategy model, and the training is stopped after exceeding the preset number of training times.

[0015] In a second aspect, the present invention further provides an underwater robot cooperative control method, comprising: collecting above-water and underwater environmental data; inputting the above-water and underwater environmental data into an underwater robot cooperative control system to obtain a control signal; the control signal comprises: a buoy action signal and an umbilical cable retracting and releasing length; The buoy action signal in the control signal is output to the buoy control unit, and the umbilical cable retraction and extension length is output to the winch control unit, so that the winch control unit retracts and extends the umbilical cable according to the control signal, and the buoy follows the underwater robot to run, thereby realizing coordinated control; wherein the above-water and underwater environmental data include the underwater robot coordinates, the buoy coordinates, the buoy velocity vector, the underwater robot velocity vector, the buoy angular velocity and the water flow velocity; The underwater robot collaborative control system is obtained by using any of the underwater robot collaborative control system construction methods described in the first aspect.

[0016] Compared with the prior art, the present invention has the following beneficial effects: (1) The present invention uses multi-level environmental simulation and state definition. By simulating the horizontal and vertical directions of the cable and considering the stretching and deformation of the underwater cable, the system's restoration of the real underwater environment and the system's adaptability are improved. It is widely applicable to a variety of underwater scenarios, including but not limited to dam inspection and underwater detection. It also achieves a significant improvement in operating efficiency and enhances the safety of the system in complex underwater environments. (2) The present invention uses a SAC algorithm based on deep reinforcement learning to achieve efficient exploration and intelligent control of the buoy. The winch control algorithm based on a lightweight Transformer is used to predict the expected length of the umbilical cable in real time through a behavioral cloning method, thus ensuring the safety and efficiency of the collaborative control system. (3) The present invention uses deep reinforcement learning guided by a reward and punishment function, designs basic rewards and bonus points based on the relative distance between the buoy and the underwater robot, and guides the intelligent agent to optimize the effect of the buoy following the underwater robot; introduces cable jitter penalty to reduce the umbilical cable jitter phenomenon caused by the reinforcement learning strategy and improve the stability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a flow chart of constructing an underwater robot cooperative control system in an embodiment of the present invention; Figure 2 is a schematic diagram of a simulation environment constructed in an embodiment of the present invention; Figure 3 is a simulation schematic diagram of an umbilical cable in an embodiment of the present invention; Figure 4 It is a winch control strategy diagram based on lightweight Transformer in an embodiment of the present invention. DETAILED DESCRIPTION

[0018] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention. Example

[0019] like Figure 1 As shown, this embodiment provides a method for constructing an underwater robot cooperative control system, comprising the following steps: Build a simulation environment: Figure 2As shown in the figure, a simulation environment including three locations of underwater robot, buoy and winch is built based on the physical and dynamic characteristics of the underwater robot body, buoy, umbilical cable and winch. Among them, the shaded part is the movable water area, and the underwater robot and buoy will only operate within this range; the underwater robot can move horizontally and vertically when operating underwater, and is responsible for underwater operations; the buoy only moves horizontally on the water surface, used to follow the movement of the underwater robot, facilitate the upper computer to locate the position of the underwater robot, and keep the underwater umbilical cable vertical to avoid the cable from self-entanglement or entanglement with the underwater robot; the winch is fixed to the shore and remains stationary during the operation of the underwater robot, and does not move.

[0020] Construct an umbilical cable simulation model; Figure 3 As shown in the figure, according to the characteristics of the underwater environment, the umbilical cable is affected by gravity, buoys and underwater robot pulling force to build a simulation model of the umbilical cable. The positions of the winch, power buoy and underwater robot are A, B and C respectively. The umbilical cable is divided into surface cable and R AB and underwater cables R BC ; and the underwater cable R BC Decomposed into horizontal cables R BC(A) , vertical cable R BC(L) ; Use the number of nodes to represent the surface cable R AB , horizontal cable R BC(A) , vertical cable R BC(L) length, and the deformation and displacement of the corresponding cable are determined according to the change in the number of nodes; among them, the surface cable R AB The head end is connected to the buoy, and the tail end is connected to the winch. The winch position A is fixed, so the end of the cable is fixed and the head end moves with the buoy. The powered buoy does not dive or float, so R AB It can be directly used to simulate the deformation and displacement of the umbilical cable on water; R BC(A) The first end is connected to the underwater robot's water surface projection, and the end is connected to the buoy; the cable R BC(A) The nodes on the surface are only affected by the horizontal tension from the buoy and the underwater robot, which is used to simulate the underwater cable. R BC Horizontal deformation and displacement; cable R BC(L) The first end is connected to the underwater robot's water surface projection, and the end is connected to the underwater robot. RBC(L) The nodes on the robot are only affected by the vertical force from the buoy and the underwater robot, as well as gravity, and are used to simulate underwater cables. R BC Deformation and displacement in the vertical direction.

[0021] Build a buoy following strategy model and train it: Firstly, a reward and punishment function model is constructed to reward and punish the movement of buoys and underwater robots and the retraction and release of umbilical cables, and the reward and punishment values ​​are calculated. Then, a buoy following strategy model is constructed based on the SAC algorithm. In this embodiment, the buoy following strategy model includes a value function network , target value function network , a Q-value function network and actor networks ; Take the simulated environment state as the input of the buoy following strategy model, combine the reward and punishment values, calculate the distribution probability of the buoy action, and select the buoy action with the largest distribution probability value as the output of the buoy following strategy model; Finally, collect historical experience data of simulated environment status and corresponding buoy action to train the buoy following strategy model. The process includes: setting the target value update frequency , target reward and penalty value ; Iterate and update the 4 deep networks until the reward and punishment value is greater than , stop iteration; The value function network updating method includes: calculating the update target of the value function network : ; Value Function Network The updated gradient is: ; In the formula, To simulate the environment The policy vector of the buoy action output by the actor network; To simulate the environment Q-value function network Output Q value vector; is the temperature coefficient, T is the transposition operation; is the mathematical expectation; D is the historical empirical data of the simulated environmental state and the corresponding buoy action; To simulate the environment Update the target with the state value below; To simulate the environment The state value output by the down-value function network; The target value function network updating method comprises: By frequency Will The parameters of each layer of the network are directly assigned to The network parameters of each layer in are used to update the network parameters of the target value function network; Among them, the Q value function network and actor networks The update methods include: The update gradient of the Q-value function network is: ; Actor Network Update the gradient to: ; In the formula, To simulate the environmental state-buoy action Q-value function network The Q value of the output; To simulate the environmental state-buoy action The reward and penalty values ​​returned by the next environment, To simulate the environment The state value output by the target value function; To simulate the environmental state-buoy action The state transition probability of the next environment; is the discount factor, a constant with a value of (0, 1).

[0022] Construct a winch control strategy model and train it: First, a winch control strategy model is constructed based on the behavioral cloning algorithm. The simulated environment state and the depth of the underwater robot are used as the input of the winch control strategy model, and the umbilical cable length is used as the output of the winch control strategy model. A lightweight Transformer model is used to learn the mapping from input to output; Figure 4 As shown in the figure, the lightweight Transformer model includes an embedding layer, a multi-head attention layer, a feedforward network, and a fully connected layer. Then, the historical experience data of the simulated environment state, the depth of the underwater robot, and the corresponding umbilical cable length are collected to train the winch control strategy model, including: the input sequence , After being encoded by the embedding layer, a time-step-based position code is added and input into the multi-head attention layer; the output of the multi-head attention layer is superimposed with the position code and output as the attention feature; the attention feature is layer-normalized and input into the feedforward network; the input and output of the feedforward network are superimposed and input into the fully connected layer, and the expected umbilical cable length is finally output; the input sequence for Time has come All simulation environment states at the moment , the depth of underwater robots ,as well as Time has come Length of all umbilical cables at the moment Combination according to The umbilical cable length at the moment and the expected umbilical cable length are used to train the network parameters of the winch control strategy model; The update gradient of the winch control strategy model is : ; in, is the network parameter of the winch control strategy model; B is the historical experience data of the simulated environment state, the depth of the underwater robot, and the corresponding umbilical cable length; For the input sequence The expected umbilical cable length output by the winch control strategy model when for The length of the umbilical cable at each moment; repeatedly input historical experience data in batches to train the network parameters of the winch control strategy model, and stop training after 1000 times.

[0023] The buoy control strategy model and the winch control strategy model are combined into an underwater robot collaborative control system; the buoy control strategy model and the winch control strategy model are embedded in the underwater robot collaborative control system, so that the underwater robot body, buoy, umbilical cable and winch can be collaboratively controlled.

[0024] The method for constructing an underwater robot cooperative control system provided in this embodiment includes all the steps in Embodiment 1. On this basis, this embodiment provides a method for constructing a reward and punishment function model: according to the principle that the greater the difference between the horizontal coordinates of the buoy and the coordinates of the underwater robot, the smaller the basic reward value, a basic reward function for reflecting the following effect of the buoy and the underwater robot is constructed, and the expression of the basic reward function is as follows: ; in, For basic return, is the Manhattan distance, is the horizontal coordinate of the buoy, is the horizontal coordinate of the underwater robot; According to the principle that if the distance between the buoy and the underwater robot is smaller, the additional bonus points are given incrementally, and if the buoy deviates from the underwater robot, the bonus points are cleared, a bonus point calculation function is constructed, and the expression of the bonus point calculation function is as follows: ; in, for Bonus points for the moment; for Bonus points for the moment; is the reward constant, ; for Simulate the environment status at all times; for Simulate the environment status at all times; for Moment buoy action; for Moment buoy action; is the buoy speed; According to the surface cable R AB and underwater cables R BC Based on the principle that the length has a negative correlation return, a cable jitter penalty function is introduced. The expression of the cable jitter penalty function is as follows: ; in, is the Euclidean distance; The sum of basic rewards, bonus points, and cable jitter penalties is calculated as the reward and penalty value.

[0025] Embodiment 3 of the present invention provides a method for collaborative control of an underwater robot, comprising: collecting above-water and underwater environmental data; in this embodiment, the above-water and underwater environmental data include the coordinates of the underwater robot, the coordinates of the buoy, the buoy velocity vector, the underwater robot velocity vector, the buoy angular velocity and the water flow velocity.

[0026] The above-water and underwater environmental data are input into the underwater robot collaborative control system to obtain a control signal; in this embodiment, the control signal includes: buoy movement and umbilical cable retraction and release length; the underwater robot collaborative control system is obtained by the underwater robot collaborative control system construction method in Example 1 or Example 2.

[0027] The control signal is output to the buoy and the winch control unit, so that the winch control unit retracts and releases the umbilical cable according to the control signal, and the buoy follows the operation of the underwater robot, thereby realizing coordinated control.

[0028] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for constructing an underwater robot cooperative control system, characterized in that: include: Build a simulation environment based on the physical and dynamic characteristics of the underwater robot, buoy, umbilical cable and winch; According to the characteristics of the underwater environment, considering the influence of gravity, buoys and underwater robot pulling force on the umbilical cable, a simulation model of the umbilical cable is constructed; Constructing a buoy following strategy model, and training the buoy following strategy model in the simulation environment using buoy following related historical experience data according to the buoy following control requirements and the simulation results of the umbilical cable simulation model; Constructing a winch control strategy model, and training the winch control strategy model using winch control related historical experience data in the simulation environment; The trained buoy control strategy model and the trained winch control strategy model are combined to complete the construction of the underwater robot collaborative control system.

2. The method according to claim 1, characterized in that The construction of the umbilical cable simulation model comprises: The umbilical cable is divided into a surface cable R AB and underwater cables R BC ; The underwater cable R BC Decomposed into horizontal cables R BC(A) , vertical cable R BC(L) ; Using the number of nodes to represent the surface cable R AB , horizontal cable R BC(A) , vertical cable R BC(L) Length, the deformation and displacement of the corresponding rope are determined according to the change of the number of nodes; Wherein, the surface cable R AB The first end is connected to a buoy and the end is connected to a winch to simulate the deformation and displacement of the umbilical cable on water; the cable R BC(A) The first end is connected to the underwater robot's water surface projection, and the end is connected to the buoy; used to simulate underwater cables R BC Horizontal deformation and displacement; the cable R BC(L) The first end is connected to the underwater robot's water surface projection, and the end is connected to the underwater robot to simulate the underwater cable R BC Deformation and displacement in the vertical direction.

3. The method according to claim 2, characterized in that The construction of the buoy following strategy model includes: constructing a reward and punishment function model, performing rewards and punishments on the movement of the buoy and the underwater robot and the retraction and release of the umbilical cable, and calculating the reward and punishment values; A buoy following strategy is constructed based on the SAC algorithm. The simulated environmental state is used as the input of the buoy following strategy. Combined with the reward and punishment values, the distribution probability of the buoy action is calculated, and the buoy action with the largest distribution probability value is selected as the output of the buoy following strategy.

4. The method according to claim 3, characterized in that: The constructing of the reward and punishment function model includes: According to the principle that the greater the difference between the horizontal coordinates of the buoy and the underwater robot, the smaller the basic reward value, a basic reward function for reflecting the following effect of the buoy and the underwater robot is constructed to calculate the basic reward; According to the principle that if the distance between the buoy and the underwater robot is kept smaller, the additional bonus points are given incrementally, and if the buoy deviates from the underwater robot, the bonus points are cleared, a bonus point calculation function is constructed to calculate the bonus points; According to the surface cable R AB and underwater cables R BC Based on the principle that the length has a negative correlation with the reward, a rope jitter penalty function is introduced to calculate the cable jitter penalty reward; Calculate the sum of basic rewards, bonus points, and cable jitter penalty rewards as the reward and penalty value.

5. The method according to claim 4, characterized in that The expression of the basic reward function is as follows: ; in, For basic return, is the Manhattan distance, is the horizontal coordinate of the buoy, is the horizontal coordinate of the underwater robot; The expression of the reward point calculation function is as follows: ; in, for Bonus points for the moment; for Bonus points for the moment; is the reward constant, ; for Simulate the environment status at all times; for Simulate the environment status at all times; for Moment buoy action; for Moment buoy action; is the buoy speed; The expression of the rope jitter penalty function is as follows: ; in, is the Euclidean distance.

6. The method according to claim 3, characterized in that: The buoy following strategy model includes: value function network , target value function network , Q-value function network and actor networks ; The method of training the buoy following strategy model using the buoy following related historical experience data in the simulation environment includes: Collect historical experience data of simulated environmental conditions and corresponding buoy actions; Set the target value update frequency , target reward and penalty value ; Iteratively update the value function network , target value function network , Q-value function network and actor networks , until the reward or penalty value is greater than , stop iteration; The value function network updating method comprises: calculating the update target of the value function network : ; Value Function Network The updated gradient is: ; In the formula, To simulate the environment The policy vector of the buoy action output by the actor network; To simulate the environment Q-value function network Output Q value vector; is the temperature coefficient, T is the transposition operation; is the mathematical expectation; D is the historical empirical data of the simulated environmental state and the corresponding buoy action; To simulate the environment Update the target with the state value below; To simulate the environment The state value output by the down-value function network; The target value function network updating method comprises: By frequency Will The parameters of each layer of the network are directly assigned to The network parameters of each layer in are used to update the network parameters of the target value function network; The Q-value function network and actor networks The update methods include: The update gradient of the Q-value function network is: ; Actor Network Update the gradient to: ; In the formula, To simulate the environmental state-buoy action Q-value function network The Q value of the output; To simulate the environmental state-buoy action The reward and penalty values ​​returned by the next environment, To simulate the environment The state value output by the target value function; To simulate the environmental state-buoy action The state transition probability of the next environment; is the discount factor, a constant with a value of (0, 1).

7. The method according to claim 6, characterized in that Also includes updated temperature coefficients , Temperature coefficient The updated gradient is: ; In the formula, is a constant vector.

8. The method according to claim 1, characterized in that A winch control strategy model is constructed based on the behavioral cloning algorithm. The simulated environmental state and the depth of the underwater robot are used as the input of the winch control strategy model, and the length of the umbilical cable is used as the output of the winch control strategy model. A lightweight Transformer model is used to learn the mapping from input to output.

9. The method according to claim 8, characterized in that The lightweight Transformer model includes an embedding layer, a multi-head attention layer, a feedforward network, and a fully connected layer; The winch control strategy model is trained using winch control related historical experience data in the simulation environment, including: Collect historical experience data on the simulated environment status, the depth of the underwater robot, and the corresponding umbilical cable length; Input sequence , After being encoded by the embedding layer, a time-step-based position code is added and input into the multi-head attention layer; the output of the multi-head attention layer is superimposed with the position code and output as the attention feature; the attention feature is layer-normalized and input into the feedforward network; the input and output of the feedforward network are superimposed and input into the fully connected layer, and the expected umbilical cable length is finally output; the input sequence for Time has come All simulation environment states at the moment , the depth of underwater robots ,as well as Time has come Length of all umbilical cables at the moment Combination according to The umbilical cable length at the moment and the expected umbilical cable length are used to train the network parameters of the winch control strategy model; The update gradient of the winch control strategy model is : ; in, is the network parameter of the winch control strategy model; B is the historical experience data of the simulated environment state, the depth of the underwater robot, and the corresponding umbilical cable length; For the input sequence The expected umbilical cable length output by the winch control strategy model when for Umbilical cable length at the moment; The historical experience data is input repeatedly in batches to train the network parameters of the winch control strategy model, and the training is stopped after exceeding the preset number of training times.

10. A method for cooperative control of an underwater robot, characterized in that: The method comprises: Collecting above-water and underwater environmental data; Inputting the above-water and underwater environmental data into the underwater robot cooperative control system to obtain a control signal; The control signals include: buoy action signal, umbilical cable retraction and release length; Outputting the buoy action signal in the control signal to the buoy control unit, and outputting the umbilical cable retracting and releasing length to the winch control unit, so that the winch control unit retracts and releases the umbilical cable according to the control signal, and the buoy follows the underwater robot to operate, thereby realizing coordinated control; The above-water and underwater environmental data include the underwater robot coordinates, the buoy coordinates, the buoy velocity vector, the underwater robot velocity vector, the buoy angular velocity and the water flow velocity; The underwater robot cooperative control system is constructed and obtained by using the construction method described in any one of claims 1 to 8.

Citation Information

Cited By

  • Ship, umbilical cable and underwater robot system prediction control method

    CN122284269A