Rope-driven manipulator path planning method for underwater robot operation

Abstract

The invention provides a rope-driven manipulator path planning method for underwater robot operation, and relates to the technical field of underwater robot path planning, and the method specifically comprises the following steps: collecting multi-source time sequence data by using a sensor carried on a manipulator; preprocessing the multi-source time sequence data; constructing a self-adaptive multi-channel deep reinforcement learning network; a first experience playback module in the source network is initialized, random noise is initialized, environment information is obtained through a sensor, the environment information is transmitted to a self-adaptive multi-channel deep reinforcement learning network, and the self-adaptive multi-channel deep reinforcement learning network carries out path planning according to the environment information, so that a manipulator reaches a target position; and after the manipulator arrives at the target position, the adaptive multi-channel deep reinforcement learning network is continuously utilized to carry out accurate target planning, and the maintenance action of the manipulator is realized. According to the technical scheme, the problem that in the prior art, the underwater rope-driven manipulator path planning operation precision is low is solved.

Description

Technical Field

[0001] The invention relates to the technical field of underwater robot path planning, and in particular to a path planning method for a rope-driven manipulator for underwater robot operation. Background Art

[0002] The underwater rope-driven manipulator is an underwater intelligent device designed to imitate the functions of human arms and wrists, and is designed to complete complex underwater operation tasks such as detection, sampling, installation and maintenance. In complex underwater environments, the position and attitude control of the manipulator end is prone to errors, especially in the case of long rope transmission, tension and slippage problems will lead to control deviations; in the face of dynamic water flow or complex terrain, the manipulator lacks sufficient environmental perception and adjustment capabilities. When maintenance operations are required underwater, the manipulator needs to accurately reach the location where maintenance is required.

[0003] Therefore, there is a need for a method that can enable the manipulator to perform tasks such as accurate path planning in complex underwater environments and improve the operating accuracy and autonomous operation capability of the underwater rope-driven manipulator. Summary of the invention

[0004] The main purpose of the present invention is to provide a path planning method for a rope-driven manipulator for underwater robot operation, so as to solve the problem of low operation accuracy of the path planning of the underwater rope-driven manipulator in the prior art.

[0005] To achieve the above object, the present invention provides a path planning method for a rope-driven manipulator for underwater robot operation, which specifically comprises the following steps: S1, uses sensors mounted on the manipulator to collect multi-source time series data.

[0006] S2, preprocessing of multi-source time series data, including: cleaning, formatting, normalization and normalization.

[0007] S3, construct an adaptive multi-channel deep reinforcement learning network, which includes: an original network and a twin network connected to the original network.

[0008] S4, initializes the first experience replay module in the source network, initializes the random noise, and uses sensors to obtain environmental information, and passes the environmental information to the adaptive multi-channel deep reinforcement learning network. The adaptive multi-channel deep reinforcement learning network performs path planning based on the environmental information to enable the robot to reach the target position.

[0009] S5, when the manipulator reaches the target position, it continues to use the adaptive multi-channel deep reinforcement learning network for precise target planning to realize the maintenance action of the manipulator.

[0010] Furthermore, the source network includes: a first experience playback module, an Actor1 network, a Critic network and an Actor2 network connected in sequence; the Actor1 network includes: a first intelligent optimization module, a first online strategy network and a first target strategy network connected in sequence; the Critic network includes: a first target Q network, a first online Q network, a first advantage evaluation and regularization module and a first Adma optimizer connected in sequence; the Actor2 network includes: a second intelligent optimization module, a second online strategy network and a second target strategy network; the first online Q network is connected to the first intelligent optimization module through a first AC interaction module, and the first online Q network is also connected to the second intelligent optimization module through a second AC interaction module.

[0011] The twin network includes: a second experience replay module, an Actor1' network, a Critic' network and an Actor2' network connected in sequence; the Actor1' network includes: a third intelligent optimization module, a third online strategy network and a third target strategy network connected in sequence; the Critic' network includes: a second target Q network, a second online Q network, a second advantage evaluation and regularization module and a second Adma optimizer connected in sequence; the Actor2' network includes: a fourth intelligent optimization module, a fourth online strategy network and a fourth target strategy network connected in sequence; the second online Q network is connected to the third intelligent optimization module through a third AC interaction module, and the second online Q network is also connected to the fourth intelligent optimization module through a fourth AC interaction module.

[0012] Furthermore, step S4 specifically includes the following steps: S4.1, passing the environmental information to the first online policy network of the Actor 1 network, the first online policy network processes the current state and generates an action : ; in, Represents the first online strategy network, is the current state, Represents the first online strategy network parameters.

[0013] S4.2, Actor1 network will act and current status The first online Q network passed to the Critic network evaluates the current action and generates a decision value ; ; in, Representing the First Online Q Network, are the parameters of the first online Q network.

[0014] S4.3, Actor1 network simultaneously takes action Pass it to the environment module to get the Environmental information at a time step ,The environment module includes: physical scene, task requirements and external disturbances.

[0015] S4.4, the environment module will The first target strategy network of Actor1 network is passed to the first target strategy network of Actor1 network, which calculates and generates the corresponding Actions in time steps ; ; in, represents the first target strategy network, are the parameters of the first target policy network.

[0016] S4.5, the environment module simultaneously passes the multidimensional array to the first experience replay module, and then after the first experience replay module makes a judgment, passes the multidimensional array to the path experience replay pool in the first experience replay module; the multidimensional array includes: the current state , No. Environmental information at a time step , instant rewards and actions .

[0017] S4.6, randomly select a group of multidimensional arrays from the path experience pool through the sampling strategy, and pass them to the Actor1 network and the Critic network respectively. The first target strategy network of Actor1 will The first target Q network passed to Critic, the first target Q network Evaluate and generate learning values ; ; in, represents the first target Q network, It is The state of the time step, is the action taken, are the parameters of the first target Q network.

[0018] Further, step S4 also includes: S4.7, updating the critic network parameters; step S4.7 specifically includes the following steps: S4.7.1, the first online Q network according to Calculate target value : ; in, is the target value, It’s an instant reward. is the discount factor, Is the first online strategy network in The following output action.

[0019] S4.7.2, Calculate the advantage function of the Critic network : ; Is the first online Q network in the state Take action The estimated value of is the state value function: ; in, is the parameter set, Indicates in status Take action The probability of For strategy; For online Q network in state Take action The estimated value of is a parameter set.

[0020] S4.7.3, calculate the loss function of the first target Q network : ; Indicates the batch size, represents the advantage function, Indicates in status Take action Prediction value, Represents the sum of the absolute values ​​of the advantage functions of all samples in the batch.

[0021] S4.7.4, Yes Choose L2 regularization: ; in, is L2 regularization, is the regularization strength.

[0022] ; in, represents the overall loss function of the Critic network, is the loss function of the first online Q-network.

[0023] S4.7.5, Calculate the regularization gradient : .

[0024] S4.7.6, add the regularized gradient to the original gradient: + ; in, is the total gradient, is the gradient of the loss function, is the regularized gradient.

[0025] S4.7.7, Optimize Q network parameters using the first Adam optimizer .

[0026] Furthermore, step S4 also includes: S4.8, optimizing the network parameters of Actor1, and step S4.8 specifically includes the following steps: S4.8.1, and The data is transmitted to the first AC interaction module for data processing to generate an enhanced signal. .

[0027] S4.8.2, The data are transmitted to the first intelligent optimization module for parameter optimization; Calculate the loss function: .

[0028] Compute the gradient: ; is the loss function About learning rate gradient. is the online policy network about the learning rate gradient. Is an online strategy network in the state The output of the following, yes The variance of .

[0029] S4.8.3, update Actor1 network parameters: ; in, is the learning rate.

[0030] Furthermore, step S4 further includes the following steps: S4.9, each time S4.1 to S4.8 are cycled once, the distance from the robot in the environment module to the target point is calculated. If the distance is less than D and D is a positive number, it means that the data in Actor1 and Critic at this time are excellent data.

[0031] S4.10, copy the excellent data layer by layer, and copy them into the corresponding layers of the Actor1' network and the Critic' network in the twin network respectively; after the excellent data is completely copied to the twin network, the Actor1' network and the Critic' network in the twin network continue to calculate, repeating S4.1 to S4.8; until the maximum number of updates is reached, determine whether the path planning is slow.

[0032] S4.11, if it is judged that the calculation is slow, discard the current data, switch to the original network, use the excellent data in S4.9 to calculate, and repeat S4.1 to S4.10.

[0033] S4.14, if it is judged that the calculation is not slow, continue to use the data in the twin network for calculation, and repeat S4.1 to S4.10 until the target location is reached.

[0034] Furthermore, step S5 is specifically as follows: Input the environmental information of the current time step into the Actor2 network and the Critic network in the source network and repeat steps S4.1 to S4.14; wherein the second intelligent optimization module in the Actor2 network corresponds to the first intelligent optimization module in the Actor1 network, the second online strategy network corresponds to the first online strategy network, the second target strategy network corresponds to the first target strategy, and the second AC interaction module corresponds to the first AC interaction module.

[0035] The present invention has the following beneficial effects: The method provided by the present invention can accurately predict the specific posture, accurate position and complete motion trajectory of the underwater manipulator during the execution of tasks in a complex marine environment, ensuring the accuracy and reliability of the operation.

[0036] The present invention designs an adaptive multi-channel deep reinforcement learning network, which can focus more on the optimization of specific tasks and improve learning efficiency. At the same time, the weights and parameters of Actor1 network, Actor2 network, Actor1' network and Actor2' network can be dynamically adjusted according to different task requirements to adapt to different environmental changes and task requirements. Improve its generalization ability, by training Actor1 network, Actor2 network, Actor1' network and Actor2' network separately, the network can have stronger generalization ability on specific tasks, and improve adaptability and robustness in different scenarios.

[0037] Classification and management of the data stored in the experience replay pool can improve sample efficiency. By extracting specific types of experience in a targeted manner, such as those that lead to high rewards or failures, the agent can train more frequently in critical situations, thereby learning important strategic knowledge faster and avoiding wasting computing resources on less critical data. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the specific implementation of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for the specific implementation or the prior art description. Obviously, the drawings described below are some implementations of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings: Figure 1 A flow chart of a method for path planning of a rope-driven manipulator for underwater robot operation according to the present invention is shown.

[0039] Figure 2 A structural diagram of the source network of the present invention is shown.

[0040] Figure 3 A structural diagram of the twin network of the present invention is shown.

[0041] Figure 4 The figure shows the training effects of the method provided by the present invention at different stages.

[0042] Figure 5 A simulation diagram of the path planning of a rope-driven manipulator using the method provided by the present invention is shown. DETAILED DESCRIPTION

[0043] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0044] like Figure 1 The path planning method of a rope-driven manipulator for underwater robot operation shown in the figure specifically comprises the following steps: S1, using sensors mounted on the manipulator to collect multi-source time series data; the sensors include speedometers, accelerometers, sonar systems and pressure sensors, and the collected multi-source time series data include position, speed, acceleration and pressure.

[0045] S2, preprocessing of multi-source time series data, including: cleaning, formatting, normalization and normalization.

[0046] S3, construct an adaptive multi-channel deep reinforcement learning network, which includes: an original network and a twin network connected to the original network.

[0047] S4, initializes the first experience replay module in the source network, initializes the random noise, and uses sensors to obtain environmental information, and passes the environmental information to the adaptive multi-channel deep reinforcement learning network. The adaptive multi-channel deep reinforcement learning network performs path planning based on the environmental information to enable the robot to reach the target position.

[0048] S5, when the manipulator reaches the target position, it continues to use the adaptive multi-channel deep reinforcement learning network for precise target planning to realize the maintenance action of the manipulator.

[0049] Specifically, Figure 2 As shown, the original network includes: a first experience playback module, an Actor1 network, a Critic network and an Actor2 network connected in sequence; the Actor1 network includes: a first intelligent optimization module, a first online strategy network and a first target strategy network connected in sequence; the Critic network includes: a first target Q network, a first online Q network, a first advantage evaluation and regularization module and a first Adma optimizer connected in sequence; the Actor2 network includes: a second intelligent optimization module, a second online strategy network and a second target strategy network; the first online Q network is connected to the first intelligent optimization module through the first AC interaction module, and the first online Q network is also connected to the second intelligent optimization module through the second AC interaction module.

[0050] like Figure 3 As shown, the twin network includes: a second experience playback module, an Actor1' network, a Critic' network and an Actor2' network connected in sequence; the Actor1' network includes: a third intelligent optimization module, a third online strategy network and a third target strategy network connected in sequence; the Critic' network includes: a second target Q network, a second online Q network, a second advantage evaluation and regularization module and a second Adma optimizer connected in sequence; the Actor2' network includes: a fourth intelligent optimization module, a fourth online strategy network and a fourth target strategy network connected in sequence; the second online Q network is connected to the third intelligent optimization module through the third AC interaction module, and the second online Q network is also connected to the fourth intelligent optimization module through the fourth AC interaction module.

[0051] First, the path planning of the robot is carried out on a large scale. In this process, the Actor1 network, Critic network, first AC interaction module, second AC interaction module and first experience playback module in the original network, as well as the Actor1' network, Critic' network, third AC interaction module, fourth AC interaction module and second experience playback module in the twin network are mainly involved; the structures of the twin network and the original network are the same.

[0052] The first AC interaction module is mainly used to receive the parameters of the Actor1 network and the Critic network, and pass them to the Actor1 network after processing and calculation, so as to update the parameters of the Actor1 network; the same is true for the second AC interaction module.

[0053] Specifically, step S4 includes the following steps: S4.1, passing the environmental information to the first online policy network of the Actor 1 network, the first online policy network processes the current state and generates an action : ; in, Represents the first online strategy network, is the current state, Represents the first online strategy network parameters.

[0054] S4.2, Actor1 network will act and current status The first online Q network passed to the Critic network evaluates the current action and generates a decision value ; ; in, Representing the First Online Q Network, are the parameters of the first online Q network.

[0055] S4.3, Actor1 network simultaneously takes action Pass it to the environment module to get the Environmental information at a time step ,The environment module includes : physical scene, task requirements and external disturbances; the physical scene is the underwater environment where the manipulator is located; the task requirement is to plan the path of the manipulator to reach the maintenance location; the external disturbance is underwater noise, etc.

[0056] S4.4, the environment module will The first target strategy network of Actor1 network is passed to the first target strategy network of Actor1 network, which calculates and generates the corresponding Actions in time steps ; ; in, represents the first target strategy network, are the parameters of the first target policy network.

[0057] S4.5, the environment module simultaneously passes the multidimensional array to the first experience replay module, and then after the first experience replay module makes a judgment, passes the multidimensional array to the path experience replay pool in the first experience replay module; the multidimensional array includes: the current state , No. Environmental information at a time step , instant rewards and actions .

[0058] The first and second experience replay modules are composed of two experience replay pools, namely the path experience replay pool and the maintenance experience replay pool. After receiving external information data, the first and second experience replay modules will first judge the current state of the robot. If the target point has not been reached, the multidimensional array will be stored in the path experience replay pool. If the target point has been reached, the multidimensional array will be stored in the maintenance experience replay pool.

[0059] S4.6, randomly select a group of multidimensional arrays from the path experience pool through the sampling strategy, and pass them to the Actor1 network and the Critic network respectively. The first target strategy network of Actor1 will The first target Q network passed to Critic, the first target Q network Evaluate and generate learning values ; ; in, represents the first target Q network, It is The state of the time step, is the action taken, are the parameters of the first target Q network.

[0060] Specifically, step S4 also includes: S4.7, updating the Critic network parameters; step S4.7 specifically includes the following steps: S4.7.1, the first online Q network according to Calculate target value : ; in, is the target value, It’s an instant reward. is a discount factor used to balance immediate rewards and future rewards, is the target Q network, used to evaluate the next state The value of the action given by the execution policy network is, Is the first online strategy network in The following output action.

[0061] S4.7.2, Calculate the advantage function of the Critic network : ; in, Is the first online Q network in the state Take action The estimated value of is the state value function: ; in, is the parameter set, By parameter set Sure. Indicates in status Take action The probability of For strategy; By Strategy Sure. For online Q network in state Take action The estimated value of is the parameter set; By parameter set Sure.

[0062] S4.7.3, calculate the loss function of the first target Q network : ; is a weighted mean square error, which is used to measure the prediction Values ​​and Goals The difference between the values. Represents the batch size, that is, the number of samples used to update the network. represents the advantage function, which means that in the state Take action Advantages over random actions. Indicates in status Take action Prediction Value, determined by the online Q network parameters calculate. Represents the sum of the absolute values ​​of the advantage function for all samples in the batch; used as a normalization factor to ensure that the sum of the weights is 1.

[0063] S4.7.4, Yes Choose L2 regularization: ; in, is L2 regularization, which is based on the model parameters The sum of the squares of is the regularization strength, which is a hyperparameter that controls the relative importance of the regularization term in the overall loss function. The larger the value of , the heavier the penalty for model complexity. is the online Q network parameter, which can be weight w or bias b, This is for all The sum of the squares of . , its square is added to the total.

[0064] ; represents the overall loss function of the Critic network, is the loss function of the first online Q-network.

[0065] S4.7.5, Calculate the regularization gradient : .

[0066] S4.7.6, add the regularized gradient to the original gradient: + ; in, is the total gradient, is the gradient of the loss function, is the regularized gradient.

[0067] S4.7.7, Optimize Q network parameters using the first Adam optimizer .

[0068] Specifically, step S4 also includes: S4.8, optimizing the network parameters of Actor1, and step S4.8 specifically includes the following steps: S4.8.1, and The data is transmitted to the first AC interaction module for data processing to generate an enhanced signal. .

[0069] The first AC interaction module has two inputs, namely the actions of Actor1 and ; The data is processed in the first AC interaction module as follows: Passing the two inputs to the first hidden layer of the first AC interaction module, concatenating the two inputs to generate an output; Passing the output of the first hidden layer to the second hidden layer of the first AC interaction module for further processing to generate an output; The output of the second hidden layer is passed to the fusion layer of the first AC interaction module, and the outputs of the two hidden layers are concatenated to generate the output; The output of the fusion layer is passed to the output layer of the first AC interaction module to finally generate a reinforcement signal. .

[0070] The first AC interaction module is composed of a neural network. The adaptive multi-channel deep reinforcement learning network with the first AC interaction module added can be optimized in terms of information fusion, learning stability, strategy adaptability, exploration efficiency, sample efficiency and generalization ability, thereby showing better performance in reinforcement learning tasks.

[0071] S4.8.2, Pass it to the first intelligent optimization module for parameter optimization.

[0072] The first intelligent optimization module is mainly used to update the parameters of the Actor1 network. The first intelligent optimization module receives the output from the first AC interaction module. , calculate the loss function of the Actor1 network, and after calculating the gradient, use it to update the parameters, which increases the efficiency and accuracy of network update and exploration.

[0073] Calculate the loss function: ; Compute the gradient: ; in, is the loss function About learning rate The gradient is a vector that contains the contribution of each parameter to the loss function and is used to guide the update of the parameters. is the output of the above AC interaction module; is the online policy network about the learning rate The gradient of . This gradient measures the impact of parameter changes on the policy output. Is an online strategy network in the state The output of the following, yes The variance measures the dispersion of the probability distribution and is often used to compute the Gaussian noise of the gradient. is the variance of the difference between the output of the online policy network and the actual action. This difference measures the deviation between the prediction of the online policy network and the actual execution.

[0074] S4.8.3, update Actor1 network parameters: ; in, is the learning rate, which controls the step size of parameter updates and is the gradient of the loss function of the Actor1 network.

[0075] Specifically, step S4 also includes the following steps: S4.9, each time S4.1 to S4.8 are cycled once, the distance from the manipulator to the target point in the environment module is calculated. If the distance is less than D and D is a positive number, it means that the data in Actor1 and Critic at this time is excellent data; the distance from the manipulator to the target point is stored in the task requirements in the environment module.

[0076] S4.10, copy the excellent data layer by layer, and copy them into the corresponding layers of the Actor1' network and the Critic' network in the twin network respectively; after the excellent data is completely copied to the twin network, the Actor1' network and the Critic' network in the twin network continue to calculate, repeating S4.1 to S4.8; until the maximum number of updates is reached, determine whether the path planning is slow.

[0077] S4.11, if it is judged that the calculation is slow, discard the current data, switch to the original network, use the excellent data in S4.9 to calculate, and repeat S4.1 to S4.10.

[0078] S4.14, if it is judged that the calculation is not slow, continue to use the data in the twin network for calculation, and repeat S4.1 to S4.10 until the target location is reached.

[0079] Specifically, step S5 is as follows: Input the environmental information of the current time step into the Actor2 network and the Critic network in the source network and repeat steps S4.1 to S4.14; wherein the second intelligent optimization module in the Actor2 network corresponds to the first intelligent optimization module in the Actor1 network, the second online strategy network corresponds to the first online strategy network, the second target strategy network corresponds to the first target strategy network, and the second AC interaction module corresponds to the first AC interaction module.

[0080] In step S5, the manipulator performs path planning in a small range, mainly responsible for planning the path of the end of the manipulator near the maintenance point, and assisting the end effector to perform maintenance actions. In this process, the Actor2 network, Critic network, first AC interaction module, second AC interaction module and first experience playback module in the original network, as well as the Actor2' network, Critic' network, third AC interaction module, fourth AC interaction module and experience playback module in the twin network are mainly involved; the twin network and the original network have the same structure.

[0081] Figure 4 It shows the training effect of deep reinforcement learning algorithm at different stages, including initial stage, mid-stage and late stage. Figure 4 The original return in is coherent noise. In the middle stage, it shows steady growth; in the later stage, a better strategy is found and the performance tends to be stable. This proves the effectiveness and reliability of the method provided by the present invention in path planning tasks.

[0082] like Figure 5 As shown in the figure, the robot's motion trajectory clearly demonstrates its ability to avoid circular obstacles. Starting from the starting point, the robot cleverly bypasses the obstacle under the guidance of the deep reinforcement learning algorithm. Throughout the movement, the robot always maintains a stable speed and precise control, ensuring the safety and efficiency of the movement. Until the robot successfully reaches the end point and completes the predetermined task.

[0083] Of course, the above description is not a limitation of the present invention, and the present invention is not limited to the above examples. Changes, modifications, additions or substitutions made by technicians in this technical field within the essential scope of the present invention should also fall within the protection scope of the present invention.

Claims

1. A method for path planning of a rope-driven manipulator for underwater robot operation, characterized in that: The specific steps include: S1, collects multi-source time series data using sensors mounted on the manipulator; S2, preprocessing of multi-source time series data, including cleaning, formatting, normalization and normalization; S3, constructing an adaptive multi-channel deep reinforcement learning network, which includes: an original network and a twin network connected to the original network; S4, initialize the first experience replay module in the source network, initialize the random noise, and use the sensor to obtain environmental information, and pass the environmental information to the adaptive multi-channel deep reinforcement learning network. The adaptive multi-channel deep reinforcement learning network performs path planning based on the environmental information to enable the manipulator to reach the target position; S5, when the manipulator reaches the target position, it continues to use the adaptive multi-channel deep reinforcement learning network for precise target planning to realize the maintenance action of the manipulator.

2. A method for path planning of a rope-driven manipulator for underwater robot operation according to claim 1, characterized in that: The original network includes: a first experience playback module, an Actor1 network, a Critic network, and an Actor2 network connected in sequence; the Actor1 network includes: a first intelligent optimization module, a first online strategy network, and a first target strategy network connected in sequence; the Critic network includes: a first target Q network, a first online Q network, a first advantage evaluation and regularization module, and a first Adma optimizer connected in sequence; the Actor2 network includes: a second intelligent optimization module, a second online strategy network, and a second target strategy network; the first online Q network is connected to the first intelligent optimization module through a first AC interaction module, and the first online Q network is also connected to the second intelligent optimization module through a second AC interaction module; The twin network includes: a second experience replay module, an Actor1' network, a Critic' network and an Actor2' network connected in sequence; the Actor1' network includes: a third intelligent optimization module, a third online strategy network and a third target strategy network connected in sequence; the Critic' network includes: a second target Q network, a second online Q network, a second advantage evaluation and regularization module and a second Adma optimizer connected in sequence; the Actor2' network includes: a fourth intelligent optimization module, a fourth online strategy network and a fourth target strategy network connected in sequence; the second online Q network is connected to the third intelligent optimization module through a third AC interaction module, and the second online Q network is also connected to the fourth intelligent optimization module through a fourth AC interaction module.

3. A method for path planning of a rope-driven manipulator for underwater robot operation according to claim 1, characterized in that: Step S4 specifically includes the following steps: S4.1, passing the environmental information to the first online policy network of the Actor 1 network, the first online policy network processes the current state and generates an action : ； in, Represents the first online strategy network, is the current state, represents the first online strategy network parameters; S4.2, Actor1 network will act and current status The first online Q network passed to the Critic network evaluates the current action and generates a decision value ; ； in, Representing the First Online Q Network, are the parameters of the first online Q network; S4.3, Actor1 network simultaneously takes action Pass it to the environment module to get the Environmental information at a time step ,The environment module includes: physical scenes, task requirements and external disturbances; S4.4, the environment module will The first target strategy network of Actor1 network is passed to the first target strategy network of Actor1 network, which calculates and generates the corresponding Actions in time steps ; ； in, represents the first target strategy network, are the parameters of the first target policy network; S4.5, the environment module simultaneously passes the multidimensional array to the first experience replay module, and then after the first experience replay module makes a judgment, passes the multidimensional array to the path experience replay pool in the first experience replay module; the multidimensional array includes: the current state , No. Environmental information at a time step , instant rewards and actions ; S4.6, randomly select a group of multidimensional arrays from the path experience pool through the sampling strategy, and pass them to the Actor1 network and the Critic network respectively. The first target strategy network of Actor1 will The first target Q network passed to Critic, the first target Q network Evaluate and generate learning values ; ； in, represents the first target Q network, It is The state of the time step, is the action taken, are the parameters of the first target Q network.

4. A method for path planning of a rope-driven manipulator for underwater robot operation according to claim 3, characterized in that: Step S4 also includes: S4.7, updating the critic network parameters; step S4.7 specifically includes the following steps: S4.7.1, the first online Q network according to Calculate target value : ； in, is the target value, It’s an instant reward. is the discount factor, Is the first online strategy network in The output action is: S4.7.2, Calculating the Advantage Function of the Critic Network : ； Is the first online Q network in the state Take action The estimated value of is the state value function: ； in, is the parameter set, Indicates in status Take action The probability of For strategy; For online Q network in state Take action The estimated value of is the parameter set; S4.7.3, calculate the loss function of the first target Q network : ； Indicates the batch size, represents the advantage function, Indicates in status Take action Prediction value, Represents the sum of the absolute values ​​of the advantage functions of all samples in the batch; S4.7.4, Yes Choose L2 regularization: ； in, is L2 regularization, is the regularization strength; ； in, represents the overall loss function of the Critic network, is the loss function of the first online Q network; S4.7.5, Calculate the regularization gradient : ； S4.7.6, add the regularized gradient to the original gradient: + ； in, is the total gradient, is the gradient of the loss function, is the regularization gradient; S4.7.7, Optimize Q network parameters using the first Adam optimizer .

5. A method for path planning of a rope-driven manipulator for underwater robot operation according to claim 4, characterized in that: Step S4 also includes: S4.8, optimizing the network parameters of Actor 1, and step S4.8 specifically includes the following steps: S4.8.1, and The data is transmitted to the first AC interaction module for data processing to generate an enhanced signal. ; S4.8.2, The data are transmitted to the first intelligent optimization module for parameter optimization; Calculate the loss function: ； Compute the gradient: ； is the loss function About learning rate The gradient of is the online policy network about the learning rate The gradient of Is an online strategy network in the state The output of the following, yes The variance of S4.8.3, update Actor1 network parameters: ； in, is the learning rate.

6. A method for path planning of a rope-driven manipulator for underwater robot operation according to claim 5, characterized in that: Step S4 also includes the following steps: S4.9, each time S4.1 to S4.8 are cycled once, the distance from the robot in the environment module to the target point is calculated. If the distance is less than D, and D is a positive number, it means that the data in Actor1 and Critic at this time are excellent data; S4.10, copy the excellent data layer by layer, and copy them into the corresponding layers of the Actor1' network and the Critic' network in the twin network respectively; after the excellent data is completely copied to the twin network, the Actor1' network and the Critic' network in the twin network continue to calculate, and repeat S4.1 to S4.8; After reaching the maximum number of updates, determine whether the path planning is slow; S4.11, if the calculation is judged to be slow, discard the current data, switch to the original network, use the excellent data in S4.9 to calculate, and repeat S4.1 to S4.10; S4.14, if it is judged that the calculation is not slow, continue to use the data in the twin network for calculation, and repeat S4.1 to S4.10 until the target location is reached.

7. A method for path planning of a rope-driven manipulator for underwater robot operation according to claim 1, characterized in that: Step S5 is specifically: inputting the environmental information of the current time step into the Actor2 network and the Critic network in the source network to repeat steps S4.1 to S4.14; wherein, the second intelligent optimization module in the Actor2 network corresponds to the first intelligent optimization module in the Actor1 network, the second online strategy network corresponds to the first online strategy network, the second target strategy network corresponds to the first target strategy network, and the second AC interaction module corresponds to the first AC interaction module.

Images