Course control method and device for trenching and cable laying robot and trenching and cable laying robot

Abstract

The present invention relates to the field of trenching and cable-laying robot control technology, and specifically discloses a trenching and cable-laying robot heading control method, device, and trenching and cable-laying robot, including: obtaining the real-time heading angle of the trenching and cable-laying robot and obtaining heading error information; calling an optimized PID controller and inputting the heading error information into the optimized PID controller, wherein the optimized PID controller can perform a comprehensive calculation of the heading error information based on input control parameters to obtain a heading control strategy, wherein the input control parameters are obtained by optimization training based on a reinforcement learning algorithm based on a three-pool structure, and the input control parameters include differential parameters, integral parameters, and proportional parameters, and the integral includes a dual-factor parameter consisting of heading deviation and current speed; and the heading control strategy is sent to the heading actuator of the trenching and cable-laying robot. The heading control method of the trenching and cable-laying robot provided by the present invention achieves better control effects and stronger environmental adaptability.

Description

Technical Field

[0001] The present invention relates to the technical field of trenching and cable-laying robot control, and in particular to a trenching and cable-laying robot heading control method, a trenching and cable-laying robot heading control device, and a trenching and cable-laying robot. Background Art

[0002] During the process of trenching and cable-laying robots from launching into the water to landing on the bottom, the performance of their control systems is directly related to the stability and efficiency of the operation. Control algorithm optimization and control parameter optimization are key technologies for improving the operational capabilities of such robots. With the widespread application of trenching and cable-laying robots in fields such as submarine cable laying, marine engineering construction, and deep-sea resource development, they face multiple challenges in complex underwater environments, such as ocean current disturbances, terrain changes, depth fluctuations, and equipment mechanical wear. Traditional control methods often find it difficult to balance control accuracy and response speed. To ensure that trenching and cable-laying robots operate stably and efficiently under variable operating conditions, the control algorithm and parameters must be jointly optimized. By designing an efficient and robust control algorithm and finding appropriate control parameters, the overall control performance of the system can be significantly improved, ensuring that the task is completed safely and with high quality. Therefore, how to accurately optimize control performance in complex environments has become an important research direction for improving the performance of trenching and cable-laying robots and expanding their scope of application.

[0003] While some control methods exist in the prior art, such as traditional PID control, traditional PID control methods typically employ simple strategies such as fixed limits and integral anti-windup (such as integral separation and integral hold) to prevent integral windup, which can lead to system oscillation, overshoot, or prolonged recovery time. These methods suffer from poor dynamic adaptability, an inability to balance high-speed oscillation with low-speed accuracy, and strong parameter dependence. Furthermore, commonly used control parameter optimization methods within traditional PID control methods primarily include manual parameter adjustment, heuristic algorithms, and intelligent optimization algorithms. Manual parameter adjustment relies on experience, is inefficient, and struggles with dynamic underwater environments. While heuristic algorithms offer some adaptability, they are prone to local optima and are difficult to scale to high-dimensional, complex systems. Intelligent optimization algorithms, such as genetic algorithms and particle swarm optimization, possess global search capabilities and are commonly used for control parameter self-tuning, but they also suffer from high computational overhead, slow convergence, and dependence on initial values. Although various optimization methods have been applied to the design of control systems for trenching and cable-laying robots, their application in complex underwater environments remains challenging, including low efficiency and limited adaptability.

[0004] Therefore, how to provide a control method for a trenching and cable-laying robot that is both efficient and intelligent to achieve better control effects and stronger environmental adaptability has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the Invention

[0005] The present invention provides a heading control method for a trenching and cable-laying robot, a heading control device for the trenching and cable-laying robot, and a trenching and cable-laying robot, which solve the problems of low efficiency and poor adaptability in complex underwater environments existing in related technologies.

[0006] As a first aspect of the present invention, a heading control method for a trenching and cable-laying robot is provided, comprising:

[0007] Acquire a real-time heading angle of the trenching and cable-laying robot, and obtain heading error information based on the real-time heading angle and the expected heading angle;

[0008] Calling an optimized PID controller and inputting the heading error information into the optimized PID controller to obtain a heading control strategy, wherein the optimized PID controller is capable of performing a comprehensive calculation on the heading error information according to input control parameters to obtain the heading control strategy, wherein the input control parameters are obtained by performing optimization training according to a reinforcement learning algorithm based on a three-pool structure, wherein the three-pool structure includes an excellent experience pool, a time decay pool, and a recent high reward pool, and the input control parameters include a differential parameter, an integral parameter, and a proportional parameter, and the integral parameter includes a dual-factor parameter consisting of a heading deviation and a current speed;

[0009] The heading control strategy is sent to the heading actuator of the trenching and cable-laying robot, and the heading actuator can adjust the heading angle according to the heading control strategy.

[0010] Furthermore, the input control parameters are obtained by performing optimization training based on a reinforcement learning algorithm based on a three-pool structure, including:

[0011] Obtain multiple actual heading angles of the trenching and cable-laying robot within a preset time period;

[0012] Determine heading control state data based on multiple actual heading angles within a preset time period;

[0013] The heading control state data is input into a deep convolutional neural network model based on a three-pool structure for training, and input control parameters are obtained based on the learning results of both the evaluation network and the target network in the deep convolutional neural network model.

[0014] Furthermore, the heading control state data is input into a deep convolutional neural network model based on a three-pool structure for training, and input control parameters are obtained based on the learning results of both the evaluation network and the target network in the deep convolutional neural network model, including:

[0015] Initialize the three-pool structure in the deep convolutional neural network model and initialize the weight parameters of the evaluation network and the target network;

[0016] Inputting the heading control state data into a deep convolutional neural network model, obtaining attitude step response data corresponding to the heading control state data, and repeating the cycle;

[0017] The heading control state data and attitude step response data in each cycle are stored as training experience data in a time-decay experience pool, high-quality samples with long-term value are stored in an excellent experience pool, and recent high-value samples are stored in a recent high-reward experience pool;

[0018] For each cycle, empirical data from the three-pool structure are randomly selected according to the sampling ratio to perform gradient calculation to update the weight parameters of the evaluation network;

[0019] Determine whether to update the weight parameters between the evaluation network and the target network based on the average reward value of the evaluation network;

[0020] Whether to terminate the training process is evaluated based on the results of the most recent N cycles, and when it is determined that the training is terminated, the input control parameters are determined based on the weight parameter update results between the evaluation network and the target network.

[0021] Furthermore, empirical data in the three-pool structure are randomly selected according to the sampling ratio to perform gradient calculation to update the weight parameters of the evaluation network, including:

[0022] Determine the total amount of experience data and randomly determine the sampling ratio of any two experience pools;

[0023] The empirical data in the three-pool structure are selected based on the total number of empirical data and the sampling ratio of any two empirical pools determined randomly;

[0024] The gradient is calculated based on the N selected empirical data, where the gradient calculation formula is:

[0025] ;

[0026] in, represents the network parameters, Represents the old network parameters, Represents the gradient of the current network parameters, represents the reward for the current time step, represents the discount factor, represents the maximum Q value of all possible actions evaluated using the old network parameters in the next state, Represents the Q value estimation of the current network for state and action;

[0027] The weight parameters of the evaluation network are updated according to the results of the gradient calculation.

[0028] Furthermore, determining whether to update the weight parameters between the evaluation network and the target network according to the average reward value of the evaluation network includes:

[0029] Get the average reward value of the last N steps of the evaluation network, and fit it based on the average reward value to obtain the reward change trend of the evaluation network;

[0030] Determining a status of an evaluation network based on a reward change trend of the evaluation network;

[0031] If the evaluation network is determined to be stable, then determine whether the average reward of the target network meets the requirements of the dynamic threshold factor;

[0032] If the average reward of the target network meets the requirements of the dynamic threshold factor, the weight parameters of the evaluation network are updated to the target network;

[0033] If it is determined that the state of the evaluation network is unstable, or the average reward of the target network does not meet the requirements of the dynamic threshold factor, the weight parameters of the target network are updated to the evaluation network.

[0034] Furthermore, determining the status of the evaluation network according to the reward change trend of the evaluation network includes:

[0035] Comparing the reward change trend of the evaluation network with a preset stable positive threshold;

[0036] If the reward change trend of the evaluation network is greater than the preset stable positive threshold, determining that the state of the evaluation network is stable;

[0037] If the reward change trend of the evaluation network is not greater than the preset stable positive threshold, it is determined that the state of the evaluation network is unstable.

[0038] Furthermore, judging whether the average reward of the target network meets the requirements of the dynamic threshold factor includes:

[0039] The dynamic threshold factor is determined according to the reward change trend of the evaluation network, wherein the calculation formula of the dynamic threshold factor is:

[0040] ,

[0041] in, represents the dynamic threshold factor, Indicates the reward change trend of the evaluation network, represents the adjustable factor, Indicates the normalization progress of training;

[0042] Determining an average reward change trend of the target network, and comparing the average reward change trend of the target network with the dynamic threshold factor;

[0043] If the average reward change trend of the target network is greater than the dynamic threshold factor, it is determined that the average reward of the target network meets the requirements of the dynamic threshold factor; otherwise, it does not meet the requirements of the dynamic threshold factor.

[0044] Furthermore, the expression of the optimized PID controller is:

[0045] ,

[0046] in, represents the output of the optimized PID controller, Indicates the current heading error information. Indicates the heading error information at the previous moment, represents the scale parameter, represents the integral parameter, represents the differential coefficient parameter, and the integral parameter The expression is:

[0047] ,

[0048] in, , , represents the deviation suppression factor, Represents the speed suppression factor.

[0049] As another aspect of the present invention, a heading control device for a trenching and cable-laying robot is provided, for implementing the aforementioned heading control method for the trenching and cable-laying robot, comprising:

[0050] An acquisition module is used to obtain the real-time heading angle of the trenching and cable laying robot, and obtain heading error information based on the real-time heading angle and the expected heading angle;

[0051] a calling and control module, configured to call an optimized PID controller and input the heading error information into the optimized PID controller to obtain a heading control strategy, wherein the optimized PID controller is capable of performing a comprehensive calculation on the heading error information according to input control parameters to obtain the heading control strategy, wherein the input control parameters are obtained by performing optimization training based on a reinforcement learning algorithm based on a three-pool structure, wherein the three-pool structure includes an excellent experience pool, a time decay pool, and a recent high reward pool, and the input control parameters include a differential parameter, an integral parameter, and a proportional parameter, and the integral parameter includes a dual-factor parameter consisting of a heading deviation and a current speed;

[0052] The sending module is used to send the heading control strategy to the heading actuator of the trenching and cable-laying robot, and the heading actuator can adjust the heading angle according to the heading control strategy.

[0053] As another aspect of the present invention, a trenching and cable-laying robot is provided, which includes the heading control device of the trenching and cable-laying robot mentioned above.

[0054] The heading control method of the trenching and cable-laying robot provided by the present invention optimizes the PID controller and introduces a dual-factor adaptive integral limitation mechanism, which adaptively adjusts the integral limitation according to dynamic quantities such as the current speed and error of the equipment, thereby optimizing the oscillation caused by integral saturation and the steady-state error accuracy problem at different speeds. By introducing speed variables, the degree of integral participation is dynamically adjusted, and integral suppression is strengthened during high-speed navigation to prevent oscillation and overshoot caused by excessive integral. The integral effect is appropriately released during low-speed operation to improve steady-state accuracy and tracking performance, so that the response speed and steady-state error can be automatically balanced under different speeds and different deviation conditions, which significantly improves the stability and adaptability of the trenching and cable-laying machine heading control, and is superior to the traditional fixed rule or single-factor control integral processing method; in addition, the optimization training process for the input control parameters is based on expanding the traditional single experience pool to three experience pools, namely, an excellent experience pool, a time decay pool, and a recent high reward pool. The excellent experience pool focuses on retaining high-quality samples that have been repeatedly verified and have high long-term value for fine-tuning strategies; the time decay experience pool stores all the interaction data between the trenching and cable-laying robots and the environment, and introduces a time decay mechanism to reduce the sampling probability of outdated experience, ensuring sample diversity while avoiding interference from historical low-quality data; the recent high reward pool is used to temporarily store newly generated high-reward samples, and controls its promotion mechanism by setting a verification period and reward threshold, capturing potential strategy change signals, and strengthening the use of short-term excellent experience. Therefore, the heading control method of the trenching and cable-laying robot provided by the present invention is both efficient and intelligent, and can achieve better control effects and stronger environmental adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. Together with the following specific embodiments, they are used to explain the present invention, but do not constitute a limitation of the present invention.

[0056] Figure 1 This is a flow chart of the heading control method of the trenching and cable-laying robot provided by the present invention.

[0057] Figure 2 This is a schematic diagram of the control process of the heading control method of the trenching and cable-laying robot provided by the present invention.

[0058] Figure 3This is a flow chart of the DSR-DQN algorithm provided by the present invention.

[0059] Figure 4 This is a structural block diagram of the heading control device of the trenching and cable-laying robot provided by the present invention. DETAILED DESCRIPTION

[0060] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments of the present invention may be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0061] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0062] It should be noted that the terms "first," "second," and the like in the specification and claims of the present invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate for the embodiments of the present invention described herein. In addition, the terms "including," "having," and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to these processes, methods, products, or apparatuses.

[0063] In this embodiment, a heading control method for a trenching and cable laying robot is provided. Figure 1 FIG. 1 is a flow chart of a method for controlling the heading of a trenching and cable-laying robot according to an embodiment of the present invention. Figure 1 Shown, including:

[0064] S100, obtaining a real-time heading angle of the trenching and cable-laying robot, and obtaining heading error information based on the real-time heading angle and a desired heading angle;

[0065] In the embodiment of the present invention, the real-time heading angle of the trenching and cable-laying robot is specifically obtained, and the heading error information can be obtained when the expected heading angle is known.

[0066] S200, calling an optimized PID controller, and inputting the heading error information into the optimized PID controller to obtain a heading control strategy, wherein the optimized PID controller is capable of performing a comprehensive calculation on the heading error information according to input control parameters to obtain the heading control strategy, wherein the input control parameters are obtained by performing optimization training based on a reinforcement learning algorithm based on a three-pool structure, wherein the three-pool structure includes an excellent experience pool, a time decay pool, and a recent high reward pool, and the input control parameters include a differential parameter, an integral parameter, and a proportional parameter, and the integral parameter includes a dual-factor parameter consisting of a heading deviation and a current speed;

[0067] In the embodiment of the present invention, Figure 2 The figure shows a control process diagram of the heading control method of the trenching and cable-laying robot. The heading control process of the trenching and cable-laying robot mainly compares the expected heading angle with the actual heading angle, calculates the heading error and inputs it into the optimized PID controller, thereby generating a control quantity output to actuators such as thrusters or servos to achieve heading adjustment and precise control.

[0068] Specifically, to cope with the complex underwater disturbance environment, the optimized PID controller adopts an adaptive integral limitation strategy, that is, the integral parameters include a dual-factor parameter consisting of heading deviation and current speed, so that the integral term limit can be dynamically adjusted according to the error or speed, effectively suppressing the overshoot and oscillation caused by integral saturation, and ensuring the heading stability and operation reliability of the robot in seabed path tracking and cable laying operations.

[0069] In addition, in order to improve the training efficiency of the reinforcement learning algorithm when obtaining input control parameters and the quality of the obtained input control parameters, a three-pool structure consisting of an excellent experience pool, a time-decayed experience pool, and a recent high reward pool is constructed: 1) The excellent experience pool focuses on retaining high-quality samples that have been repeatedly verified and have high long-term value for fine-grained strategy optimization; 2) The time-decayed experience pool stores all interaction data between the trenching and cable-laying robots and the environment, and introduces a time decay mechanism to reduce the sampling probability of outdated experience, ensuring sample diversity while avoiding interference from historical low-quality data; 3) The recent high reward pool is used to temporarily store newly generated high-reward samples, and controls its promotion mechanism by setting a verification period and reward threshold, capturing potential strategy change signals, and strengthening the utilization of short-term high-performance experience.

[0070] S300: Send the heading control strategy to the heading actuator of the trenching and cable-laying robot, and the heading actuator can adjust the heading angle according to the heading control strategy.

[0071] In an embodiment of the present invention, the heading actuator can specifically be a thruster or a servo, and the heading control strategy obtained based on the above-mentioned optimized PID controller is fed back to the heading actuator, so that the heading actuator can adjust and precisely control the heading angle based on the heading control strategy, thereby ensuring the heading stability and reliable operation of the trenching and cable-laying robot.

[0072] Therefore, the heading control method of the trenching and cable-laying robot provided by the present invention optimizes the PID controller, introduces a dual-factor adaptive integral limitation mechanism, and adaptively adjusts the integral limitation according to dynamic quantities such as the current speed and error of the equipment, thereby optimizing the oscillation caused by integral saturation and the steady-state error accuracy problem at different speeds. By introducing speed variables, the degree of integral participation is dynamically adjusted, and integral suppression is strengthened during high-speed navigation to prevent oscillation and overshoot caused by excessive integral. The integral effect is moderately released during low-speed operation to improve steady-state accuracy and tracking performance, so that the response speed and steady-state error can be automatically balanced under different speeds and different deviation conditions, significantly improving the stability and adaptability of the trenching and cable-laying machine heading control, which is better than the traditional fixed rule or single-factor control integral processing method; in addition, the optimization training process for the input control parameters is based on expanding the traditional single experience pool to three experience pools, an excellent experience pool, a time decay pool, and a recent high reward pool. The excellent experience pool focuses on retaining high-quality samples that have been repeatedly verified and have high long-term value for fine-tuning strategies; the time decay experience pool stores all the interaction data between the trenching and cable-laying robots and the environment, and introduces a time decay mechanism to reduce the sampling probability of outdated experience, ensuring sample diversity while avoiding interference from historical low-quality data; the recent high reward pool is used to temporarily store newly generated high-reward samples, and controls its promotion mechanism by setting a verification period and reward threshold, capturing potential strategy change signals, and strengthening the use of short-term excellent experience. Therefore, the heading control method of the trenching and cable-laying robot provided by the present invention is both efficient and intelligent, and can achieve better control effects and stronger environmental adaptability.

[0073] In the embodiment of the present invention, the expression of the optimized PID controller is:

[0074] ,

[0075] in, represents the output of the optimized PID controller, Indicates the current heading error information. Indicates the heading error information at the previous moment, represents the scale parameter, represents the integral parameter, represents the differential coefficient parameter, and the integral parameter The expression is:

[0076] ,

[0077] in, , , represents the deviation suppression factor, Represents the speed suppression factor.

[0078] It should be understood that by introducing a deviation suppression factor, oscillation or overshoot caused by integration can be avoided when the deviation is large, while integration can be smoothly restored when the deviation is small, improving steady-state performance. By introducing a speed variable and dynamically adjusting the degree of integration participation, integration suppression is strengthened at high speeds to prevent oscillation and overshoot caused by excessive integration. At low speeds, the integration effect is moderately released to improve steady-state accuracy and tracking performance. This method can automatically balance response speed and steady-state error under different speed and deviation conditions.

[0079] It should be noted that because traditional DQN algorithms typically randomly extract small batches of training samples from the experience pool, they inevitably introduce a large amount of low-quality, low-reward interaction data, which is not conducive to the model's learning and convergence of high-quality strategies. Furthermore, the single experience pool structure lacks differentiated management of experience value, which can easily lead to problems such as slow strategy learning and low sample utilization. Based on this, the present invention proposes an improved DQN algorithm (Dual-Structure Replay DQN, DSR-DQN) based on hierarchical experience management. By introducing three types of differentiated experience pools, it significantly improves training efficiency and strategy quality.

[0080] In an embodiment of the present invention, the input control parameters are obtained by performing optimization training based on a reinforcement learning algorithm based on a three-pool structure, including:

[0081] 1) Obtain multiple actual heading angles of the trenching and cable-laying robot within a preset time period;

[0082] 2) Determine the heading control status data based on multiple actual heading angles within a preset time period;

[0083] 3) Inputting the heading control state data into a deep convolutional neural network model based on a three-pool structure for training, and obtaining input control parameters based on the learning results of both the evaluation network and the target network in the deep convolutional neural network model.

[0084] It should be noted that, in an embodiment of the present invention, it is possible to obtain multiple actual heading angles of the trenching and cable-laying robot expressed by a heading control curve within a preset time period, and obtain heading control state data including rise time, overshoot, steady-state error and fluctuation according to the heading control curve. These heading control state data are respectively input into a deep convolutional neural network model based on a three-pool structure in the form of control state, control parameters and reward value for training.

[0085] It should be understood that in this embodiment of the present invention, a three-pool structure consisting of an excellent experience pool, a time-decay experience pool, and a recent high reward pool is constructed:

[0086] 1) The excellent experience pool focuses on retaining high-quality samples that have been repeatedly verified and have high long-term value (high-quality samples can be understood as sample data whose estimated cumulative reward value reaches a preset threshold. The composition of sample data can specifically include status, control parameters, and reward value) for fine-tuning strategies;

[0087] 2) The time-decaying experience pool stores all interaction data between the trenching and cable-laying robots and the environment. It also introduces a time-decay mechanism to reduce the probability of sampling outdated experience, ensuring sample diversity while avoiding interference from historical low-quality data.

[0088] 3) The recent high reward pool is used to temporarily store recently generated high-reward samples. By setting the verification period and reward threshold, its promotion mechanism is controlled, potential strategy change signals are captured, and the utilization of short-term high-performance experience is strengthened.

[0089] During the optimization training process of the reinforcement learning algorithm based on the three-pool structure, each sample maintains an estimated cumulative reward :

[0090] ,

[0091] Let the mean of the cumulative rewards in the current environment be and standard deviation When a sample meets the following conditions, it is considered to have long-term value and is deposited into the excellent experience pool:

[0092] ;

[0093] If the timely reward meets the following conditions and is considered to have good short-term effects, it will be deposited into the recent high reward pool:

[0094] ,

[0095] in, Indicates the upper quantile value set to the reward distribution during training.

[0096] All samples are initially stored in the time decay pool by default, and each sample records a timestamp , its sampling probability decays according to the following formula:

[0097] ,

[0098] in, Represents a decay factor that controls the speed at which old samples are screened out. This pool primarily provides diversity and base coverage, maintaining policy robustness in samples that do not have a clear reward advantage.

[0099] During experience sampling, random sampling is performed from three experience pools. The network model is trained by sampling ratio. 、 Determine how much experience is gained from sampling from the three experience pools. If each time you sample from the excellent experience pool , sampled from the time-decaying experience pool , then the number of samples from the recent high reward pool is During the replay maintenance of the three experience pools, the excellent experience pool, time-decayed experience pool, and recent high reward pool will be retained. By adjusting the sampling ratio to change the distribution of sampled experience states, it not only achieves the focus on learning high-value experience, but also ensures a balance between reviewing old experience and exploring new experience, thereby accelerating strategy learning and improving sample utilization.

[0100] In the embodiment of the present invention, the DSR-DQN algorithm is mainly divided into three design ideas: state space design, state transition method design and reward function design.

[0101] State-space design: The optimized PID controller parameter tuning problem is defined as finding a feasible solution that satisfies the following inequality constraints:

[0102] ,

[0103] in, Represents a control parameter triple , Represents the parameter vector to be optimized The value range of Represents the constraint value of the optimization target. Based on the dynamic performance index of the control system, the heading rise time , heading overshoot , heading steady-state error and heading fluctuations is the state of the system, that is, the state is a four-tuple .

[0104] State Transition Method: In the DSR-DQN algorithm, after determining the current state parameters of the trenching and cable-laying robot, the action-value network itself is the policy network. The policy network selects an action based on the maximum value of the action-value network. After determining the current state, action selection, next state, and reward value, the network then solves for the next state parameters.

[0105] The reward function is defined as:

[0106] ,

[0107] If the curve performance index exceeds the maximum value of the artificially set evaluation index, then is 0, that is:

[0108] ,

[0109] The above evaluation indicators heading rise time , heading overshoot , heading steady-state error and heading fluctuations Having different dimensions and dimensional units will affect the results of data analysis. In order to eliminate the unit differences between indicators, data normalization is required to ensure that the indicators are comparable. After data normalization, the indicators in the original data will be on the same order of magnitude and suitable for comprehensive comparative evaluation. Therefore, the calculation result of the above reward function is the result of the evaluation indicators being normalized. The normalization formula is as follows:

[0110] ,

[0111] in, Indicates the maximum value of sample data set artificially, Indicates the minimum value of sample data set manually.

[0112] Specifically, the heading control state data is input into a deep convolutional neural network model based on a three-pool structure for training, and input control parameters are obtained based on the learning results of both the evaluation network and the target network in the deep convolutional neural network model, including:

[0113] 1) Initialize the three-pool structure in the deep convolutional neural network model and initialize the weight parameters of the evaluation network and the target network;

[0114] In the embodiment of the present invention, specifically Figure 3 As shown, the three-pool structure of the embodiment of the present invention is initialized, the hyperparameters of the neural network are set, and the weight parameters of the prediction network are initialized. and the weight parameters of the target network .

[0115] 2) inputting the heading control state data into a deep convolutional neural network model, obtaining attitude step response data corresponding to the heading control state data, and repeating the cycle;

[0116] In the embodiment of the present invention, the initial parameters are given to enable the trenching and cable laying robot to acquire the initial state by interacting with the environment. , and then enter the next loop. In this loop, the trenching and cable laying robot continuously interacts with the environment to obtain attitude step response data. First, 40 random parameter interactions are performed, and then the action is selected using a greedy strategy. If exploited, use Update the parameters and get the next state after executing the action .

[0117] 3) The heading control state data and attitude step response data in each cycle are stored as training experience data in the time decay experience pool, high-quality samples with long-term value are stored in the excellent experience pool, and recent high-value samples are stored in the recent high-reward experience pool;

[0118] In the embodiment of the present invention, this experience Deposit them in the fully retained experience pool, store high-quality samples with long-term value in the excellent experience pool, and store recent high-value samples (specifically, samples whose reward value generated by the current action is higher than that of the upper quantile) in the recent high reward pool.

[0119] 4) For each cycle, empirical data from the three-pool structure is randomly selected according to the sampling ratio to perform gradient calculation to update the weight parameters of the evaluation network;

[0120] In the embodiment of the present invention, N experience data of the three experience pools are randomly taken according to the sampling ratio in the three experience pools for gradient calculation. Specifically, the experience data in the three-pool structure are randomly selected according to the sampling ratio for gradient calculation to achieve the update of the weight parameters of the evaluation network, including:

[0121] 41) Determine the total amount of experience data and the sampling ratio of the three experience pools;

[0122] 42) Select the empirical data in the three-pool structure based on the total number of empirical data and the determined sampling ratios of the three empirical pools;

[0123] 43) Perform gradient calculation based on the selected N empirical data, where the gradient calculation formula is:

[0124] ;

[0125] in, represents the network parameters, Represents the old network parameters, Represents the gradient of the current network parameters, represents the reward for the current time step, represents the discount factor, represents the maximum Q value of all possible actions evaluated using the old network parameters in the next state, Represents the Q value estimation of the current network for state and action;

[0126] 44) Update the weight parameters of the evaluation network based on the results of the gradient calculation.

[0127] It should be understood that the embodiment of the present invention can update the weight parameters of the evaluation network based on gradient calculation.

[0128] 5) Determine whether to update the weight parameters between the evaluation network and the target network based on the average reward value of the evaluation network;

[0129] It should be noted that the traditional DQN algorithm unconditionally updates the target network parameters to the evaluation network parameters after a fixed number of steps, which may cause low-quality strategies to frequently overwrite the original good strategy, affecting convergence stability. The target network parameter replacement method of the embodiment of the present invention, based on the reward trend and dynamic threshold judgment mechanism, judges the current strategy optimization direction by introducing the trend of the multi-step reward mean change, and dynamically adjusts the replacement threshold based on the complexity of the environment. Only when the current evaluation network shows a sustained and stable advantage is its parameter updated to the target network; if the evaluation strategy fluctuates or degrades, the evaluation network state is reset with the target network parameters to prevent the strategy from falling into a local optimum or frequent oscillations.

[0130] In this embodiment of the present invention, the update formula of the DSR-DQN algorithm is:

[0131] ,

[0132] The loss function of the DSR-DQN algorithm is defined as follows:

[0133] ,

[0134] in, represents the weight parameters of the convolutional neural network model, Representation sample The sampling weight is allocated based on the experience pool source. The target Q value is:

[0135] ,

[0136] After obtaining the loss function of the DSR-DQN algorithm, the gradient descent algorithm can be used directly to solve the convolutional neural network model loss function. The weight parameter .

[0137] According to the update formula of the DSR-DQN algorithm, it can be seen that in the original Q-learning algorithm, the same parameter model is used to predict the Q value and the target Q value. In order to avoid the possibility of increasing mode oscillation and divergence due to the increase of the target Q value when the predicted Q value increases, the DSR-DQN algorithm of the embodiment of the present invention uses the old network parameters. To evaluate the state Q value of the next time step of the experience sample, and only update the old network parameters at discrete multi-step intervals , provides a stable training target for the network to be fitted and gives sufficient training time, thereby better controlling the estimation error. The DSR-DQN algorithm uses two convolutional neural networks for learning: a predicted evaluation network Evaluate the value function of the current state-action pair; target network Used to generate the target value (target Q), the algorithm loss function updates the parameters in the prediction network ,Every After iterations, consider whether to predict the parameters of the network based on the reward trend and dynamic threshold judgment mechanism. Parameters copied to the target network middle.

[0138] In an embodiment of the present invention, determining whether to update the weight parameters between the evaluation network and the target network according to the average reward value of the evaluation network includes:

[0139] 51) Obtain the average reward value of the evaluation network in the last N steps, and perform fitting based on the average reward value to obtain the reward change trend of the evaluation network;

[0140] Specifically, the reward trend determination mechanism determines whether the performance of the current evaluation network strategy is continuously improving or fluctuating in the short term, preventing immature strategies from overwriting the target network. This mechanism collects the average reward of the last N steps:

[0141] ,

[0142] Use the first-order difference mean to fit the reward change trend:

[0143] .

[0144] 52) determining the status of the evaluation network based on the reward change trend of the evaluation network;

[0145] It should be understood that the state of the evaluation network may be determined specifically based on the reward change trend of the evaluation network obtained by the above calculation.

[0146] Specifically, determining the state of the evaluation network according to the reward change trend of the evaluation network includes:

[0147] 521) comparing the reward change trend of the evaluation network with a preset stable positive threshold;

[0148] 522) If the reward change trend of the evaluation network is greater than the preset stable positive threshold, then the state of the evaluation network is determined to be stable;

[0149] 523) If the reward change trend of the evaluation network is not greater than the preset stable positive threshold, the state of the evaluation network is determined to be unstable.

[0150] In an embodiment of the present invention, if , then the evaluation strategy is considered to be stable and the state of the evaluation network is determined to be stable. Indicates a preset stable positive threshold, under which accidental fluctuations of the evaluation network can be avoided. Otherwise, it is determined that the state of the evaluation network is unstable.

[0151] 53) If the evaluation network is determined to be stable, determine whether the average reward of the target network meets the requirements of the dynamic threshold factor;

[0152] In an embodiment of the present invention, determining whether the average reward of the target network meets the requirements of the dynamic threshold factor includes:

[0153] 531) Determine a dynamic threshold factor based on the reward change trend of the evaluation network, wherein the calculation formula of the dynamic threshold factor is:

[0154] ,

[0155] in, represents the dynamic threshold factor, Indicates the reward change trend of the evaluation network, represents the adjustable factor, Indicates the normalization progress of training;

[0156] 532) determining an average reward change trend of the target network, and comparing the average reward change trend of the target network with the dynamic threshold factor;

[0157] 533) If the average reward change trend of the target network is greater than the dynamic threshold factor, it is determined that the average reward of the target network meets the requirements of the dynamic threshold factor; otherwise, it does not meet the requirements of the dynamic threshold factor.

[0158] In this embodiment of the present invention, the calculation formula for the average reward of the target network is: , Represents the average reward corresponding to the current target network.

[0159] It should be understood that comparing the reward trend of the evaluation network with the dynamic threshold factor, i.e., implementing a dynamic threshold determination mechanism, can dynamically adjust the threshold for replacing the target network, avoiding both premature and delayed replacement. The dynamic threshold factor represents the proportional increase in the current round's reward relative to the target network. This dynamic threshold determination mechanism can inhibit immature strategies from replacing the target network in the early stages of training, improving strategy stability in the early stages of learning. It can also accelerate evolution while maintaining stability in the middle stages of training. In the later stages of training, as strategies converge, it is important to avoid performance degradation due to misjudgments.

[0160] 54) If the average reward of the target network meets the requirements of the dynamic threshold factor, the weight parameters of the evaluation network are updated to the target network;

[0161] Based on the above judgment and comparison, it can be known that network parameter replacement is performed if and only if the following conditions are met:

[0162] .

[0163] 55) If it is determined that the state of the evaluation network is unstable, or the average reward of the target network does not meet the requirements of the dynamic threshold factor, the weight parameters of the target network are updated to the evaluation network.

[0164] 6) Evaluate whether to terminate the training process based on the results of the most recent N cycles, and when it is determined that the training is terminated, determine the input control parameters based on the weight parameter update results between the evaluation network and the target network.

[0165] Specifically, the recent The average round reward of the training result evaluation is used to set the threshold , if the average reward is greater than , the training ends.

[0166] It should be understood that the DSR-DQN algorithm introduces a target network to keep the target Q value unchanged over a period of time, which to a certain extent reduces the correlation between the predicted Q value and the target Q value, reduces the possibility of loss value oscillation and divergence during training, and thus improves the stability of the algorithm.

[0167] The DSR-DQN algorithm uses weighted parameters The deep convolutional neural network is used as the network model of the action value function, and the convolutional neural network model is used Action-value function Perform a simulation, namely:

[0168] .

[0169] Then, the objective function is defined as the loss function of the deep convolutional neural network using mean square error:

[0170] ,

[0171] Among them, the parameters and Represents the state and action of the next time step. As can be seen from the above formula, the expected reward value that needs to be updated in the algorithm is mainly used as the target value for training. In the DSR-DQN algorithm, the target network is used to predict the target expected reward value, and the prediction network is used to predict the current expected reward value. Finally, the mean squared error is used to calculate the time difference error in the algorithm. Next, the gradient of each parameter of the deep convolutional neural network model is calculated according to the above formula:

[0172] ,

[0173] Finally, the objective function of the convolutional neural network model is optimized by using the mini-batch stochastic gradient descent algorithm, and the network is used to calculate , and get the optimal action value.

[0174] To test the actual heading auto-disturbance rejection parameter optimization capability of the heading control method for the trenching and cable-laying robot according to the present invention, a simulation was performed in MATLAB. The heading PID parameters tuned by the DSR-DQN algorithm were used in the hydrodynamic model control of the trenching and cable-laying robot to analyze the control effect. Under the same desired heading, the step response performance of the DSR-DQN algorithm was compared with that of the manually tuned algorithm. The resulting heading control data is shown in Table 1:

[0175] Table 1 Effects of different setting methods

[0176]

[0177] It can be seen from the comparative data that the heading control method of the trenching and cable-laying robot in the embodiment of the present invention can adjust better control parameters. Its adjustment results are significantly better than the manual method in terms of rise time, overshoot, steady-state error and fluctuation performance indicators, and it eliminates the tedious process of manual adjustment.

[0178] In order to test the advantages of the dual-factor adaptive integral limiting PID algorithm, the control effects of the dual-factor adaptive integral limiting PID and the ordinary integral limiting PID were compared at different speeds after control parameter training, as shown in Table 2:

[0179] Table 2 Comparison of effects of different controllers

[0180]

[0181] Comparing the control performance data shows that at a speed of 0.5m / s, the control effects of the two controllers are similar, but the dual-factor adaptive integral limiter PID control effect is more stable and has stronger adaptability when the speed changes. As the speed increases, the dual-factor adaptive integral limiter PID reward value is stable and even shows a slight upward trend. The reward value of the ordinary limiter PID is unstable and decreases significantly with increasing speed. Therefore, the dual-factor adaptive integral limiter PID has better control effect than the ordinary limiter PID.

[0182] Therefore, the heading control method of the trenching and cable-laying robot provided in the embodiment of the present invention improves the performance of the controller through the dual-factor adaptive integral limiting PID, and the control effect of the simulated heading is better; through the parameter self-tuning technology, the heading control parameters are optimized, the heading control effect of the trenching and cable-laying robot is improved, and the problem of difficulty in optimizing the controller parameters is greatly improved, providing strong support for the underwater operation posture control of the trenching and cable-laying robot, and improving the underwater operation efficiency of the trenching and cable-laying robot.

[0183] In summary, the heading control method of the trenching and cable-laying robot provided by the present invention has the following advantages:

[0184] 1) A dual-factor adaptive integral limitation approach based on heading deviation and current speed dynamically adjusts the degree of integral participation by introducing a speed variable. This approach strengthens integral suppression at high speeds to prevent oscillation and overshoot caused by excessive integrals. At low speeds, the integral effect is moderately released to improve steady-state accuracy and tracking performance. This approach automatically balances response speed and steady-state error under varying speeds and deviations, significantly improving the stability and adaptability of the trenching and cable-laying machine's heading control. It outperforms traditional integral processing approaches based on fixed rules or single-factor control.

[0185] 2) Based on the three-experience pool experience replay mechanism, the traditional single experience pool is expanded to three experience pools: an excellent experience pool, a time-decay pool, and a recent high-reward pool. The excellent experience pool focuses on retaining high-quality, repeatedly verified samples with high long-term value for fine-grained strategy optimization. The time-decay experience pool stores all interaction data between the trenching and cable-laying robots and the environment, and introduces a time-decay mechanism to reduce the sampling probability of outdated experience, ensuring sample diversity while avoiding interference from historical low-quality data. The recent high-reward pool is used to temporarily store newly generated high-reward samples. By setting verification periods and reward thresholds to control their promotion mechanism, it captures potential strategy shift signals and strengthens the utilization of short-term high-performance experience.

[0186] 3) A target network parameter replacement method based on a reward trend and dynamic threshold judgment mechanism. By introducing the trend of multi-step reward mean changes, the current strategy optimization direction is judged, and the replacement threshold is dynamically adjusted in combination with the complexity of the environment. Only when the current evaluation network shows a sustained and stable advantage is its parameter updated to the target network. If the evaluation strategy fluctuates or degrades, the evaluation network state is reset with the target network parameters to prevent the strategy from falling into a local optimum or frequent oscillations, thereby effectively controlling the quality of strategy updates, avoiding ineffective exploration, and improving training efficiency and strategy convergence speed.

[0187] As another embodiment of the present invention, a heading control device 100 for a trenching and cable laying robot is provided, which is used to implement the heading control method of the trenching and cable laying robot described above, wherein: Figure 4 Shown, including:

[0188] An acquisition module 110 is configured to acquire a real-time heading angle of the trenching and cable-laying robot, and obtain heading error information based on the real-time heading angle and a desired heading angle;

[0189] a calling and control module 120 for calling an optimized PID controller and inputting the heading error information into the optimized PID controller to obtain a heading control strategy, wherein the optimized PID controller is capable of comprehensively calculating the heading error information according to input control parameters to obtain the heading control strategy, wherein the input control parameters are obtained by performing optimization training based on a reinforcement learning algorithm based on a three-pool structure, wherein the three-pool structure includes an excellent experience pool, a time decay pool, and a recent high reward pool, and the input control parameters include a differential parameter, an integral parameter, and a proportional parameter, and the integral parameter includes a dual-factor parameter consisting of a heading deviation and a current speed;

[0190] The sending module 130 is used to send the heading control strategy to the heading actuator of the trenching and cable-laying robot, and the heading actuator can adjust the heading angle according to the heading control strategy.

[0191] The heading control device of the trenching and cable-laying robot provided by the present invention optimizes the PID controller and introduces a dual-factor adaptive integral limitation mechanism, which adaptively adjusts the integral limitation according to dynamic quantities such as the current speed and error of the equipment, thereby optimizing the oscillation caused by integral saturation and the steady-state error accuracy problem at different speeds. By introducing speed variables, the degree of integral participation is dynamically adjusted, and integral suppression is strengthened during high-speed navigation to prevent oscillation and overshoot caused by excessive integral. The integral effect is appropriately released during low-speed operation to improve steady-state accuracy and tracking performance, so that the response speed and steady-state error can be automatically balanced under different speeds and different deviation conditions, which significantly improves the stability and adaptability of the trenching and cable-laying machine heading control, and is superior to the traditional fixed rule or single-factor control integral processing method; in addition, the optimization training process for the input control parameters is based on expanding the traditional single experience pool to three experience pools, namely, an excellent experience pool, a time decay pool, and a recent high reward pool. The excellent experience pool focuses on retaining high-quality samples that have been repeatedly verified and have high long-term value for fine-tuning strategies; the time decay experience pool stores all the interaction data between the trenching and cable-laying robots and the environment, and introduces a time decay mechanism to reduce the sampling probability of outdated experience, ensuring sample diversity while avoiding interference from historical low-quality data; the recent high reward pool is used to temporarily store newly generated high-reward samples, and controls its promotion mechanism by setting a verification period and reward threshold, capturing potential strategy change signals, and strengthening the use of short-term excellent experience. Therefore, the heading control device of the trenching and cable-laying robot provided by the present invention is both efficient and intelligent, and can achieve better control effects and stronger environmental adaptability.

[0192] Regarding the specific working principle of the heading control device of the trenching and cable-laying robot provided by the present invention, reference can be made to the description of the heading control method of the trenching and cable-laying robot in the previous text, which will not be repeated here.

[0193] As another embodiment of the present invention, a trenching and cable-laying robot is provided, which includes the heading control device of the trenching and cable-laying robot mentioned above.

[0194] The trenching and cable-laying robot provided by the present invention can achieve better control effects and stronger environmental adaptability due to the adoption of the heading control device of the trenching and cable-laying robot mentioned above.

[0195] Regarding the specific working principle of the trenching and cable-laying robot provided by the present invention, please refer to the description of the heading control method of the trenching and cable-laying robot in the previous article, which will not be repeated here.

[0196] It will be understood that the above embodiments are merely exemplary embodiments for illustrating the principles of the present invention, and the present invention is not limited thereto. Those skilled in the art will appreciate that various modifications and improvements can be made without departing from the spirit and substance of the present invention, and such modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A heading control method for a trenching and cable laying robot, characterized in that: include: Acquire a real-time heading angle of the trenching and cable-laying robot, and obtain heading error information based on the real-time heading angle and the expected heading angle; Calling an optimized PID controller and inputting the heading error information into the optimized PID controller to obtain a heading control strategy, wherein the optimized PID controller is capable of performing a comprehensive calculation on the heading error information according to input control parameters to obtain the heading control strategy, wherein the input control parameters are obtained by performing optimization training according to a reinforcement learning algorithm based on a three-pool structure, wherein the three-pool structure includes an excellent experience pool, a time decay pool, and a recent high reward pool, and the input control parameters include a differential parameter, an integral parameter, and a proportional parameter, and the integral parameter includes a dual-factor parameter consisting of a heading deviation and a current speed; Sending the heading control strategy to a heading actuator of the trenching and cable-laying robot, wherein the heading actuator is capable of adjusting the heading angle according to the heading control strategy; The input control parameters are obtained by optimizing the training according to the reinforcement learning algorithm based on the three-pool structure, and include: Obtain multiple actual heading angles of the trenching and cable-laying robot within a preset time period; Determine heading control state data based on multiple actual heading angles within a preset time period; Inputting the heading control state data into a deep convolutional neural network model based on a three-pool structure for training, and obtaining input control parameters based on the learning results of both the evaluation network and the target network in the deep convolutional neural network model; Among them, the expression of the optimized PID controller is: ， in, represents the output of the optimized PID controller, Indicates the current heading error information. Indicates the heading error information at the previous moment, represents the scale parameter, represents the integral parameter, represents the differential coefficient parameter, and the integral parameter The expression is: ， in, , , represents the deviation suppression factor, represents the speed suppression factor; Among them, the excellent experience pool is used to retain sample data whose estimated cumulative reward value reaches a preset threshold; the time decay pool is used to store all the interaction data between the trenching and cable laying robot and the environment, and reduce the sampling probability of outdated experience according to the time decay mechanism; the recent high reward pool is used to temporarily store recently generated high reward samples; in the process of optimizing the reinforcement learning algorithm based on the three-pool structure, each sample maintains an estimated cumulative reward : ， Let the mean of the cumulative rewards in the current environment be and standard deviation When a sample meets the following conditions, it is considered to have long-term value and is deposited into the excellent experience pool: ； If the timely reward meets the following conditions and is considered to have good short-term effects, it will be deposited into the recent high reward pool: ， in, Indicates the upper quantile value set as the reward distribution during training; All samples are initially stored in the time decay pool by default, and each sample records a timestamp , its sampling probability decays according to the following formula: ， in, Represents the attenuation factor, which controls the speed of filtering out old samples; During experience sampling, random sampling is performed from three experience pools. The network model is trained based on experience.

2. The heading control method of the trenching and cable laying robot according to claim 1, characterized in that: The heading control state data is input into a deep convolutional neural network model based on a three-pool structure for training, and input control parameters are obtained based on the learning results of both the evaluation network and the target network in the deep convolutional neural network model, including: Initialize the three-pool structure in the deep convolutional neural network model and initialize the weight parameters of the evaluation network and the target network; Inputting the heading control state data into a deep convolutional neural network model, obtaining attitude step response data corresponding to the heading control state data, and repeating the cycle; The heading control state data and attitude step response data in each cycle are stored as training experience data in a time-decay experience pool, high-quality samples with long-term value are stored in an excellent experience pool, and recent high-value samples are stored in a recent high-reward experience pool; For each cycle, empirical data from the three-pool structure are randomly selected according to the sampling ratio to perform gradient calculation to update the weight parameters of the evaluation network; Determine whether to update the weight parameters between the evaluation network and the target network based on the average reward value of the evaluation network; Whether to terminate the training process is evaluated based on the results of the most recent N cycles, and when it is determined that the training is terminated, the input control parameters are determined based on the weight parameter update results between the evaluation network and the target network.

3. The heading control method of the trenching and cable laying robot according to claim 2, characterized in that: According to the sampling ratio, the empirical data in the three-pool structure are randomly selected for gradient calculation to update the weight parameters of the evaluation network, including: Determine the total amount of experience data and the sampling ratio of the three experience pools; The empirical data in the three-pool structure are selected according to the total number of empirical data and the determined sampling ratios of the three empirical pools; The gradient is calculated based on the N selected empirical data, where the gradient calculation formula is: ， in, represents the network parameters, Represents the old network parameters, Represents the gradient of the current network parameters, represents the reward for the current time step, represents the discount factor, It represents the maximum Q value of all possible actions evaluated in the next state using the old network parameters, and represents the Q value estimate of the current network for the state and action; The weight parameters of the evaluation network are updated according to the results of the gradient calculation.

4. The heading control method of the trenching and cable laying robot according to claim 2, characterized in that: Determine whether to update the weight parameters between the evaluation network and the target network based on the average reward value of the evaluation network, including: Get the average reward value of the last N steps of the evaluation network, and fit it based on the average reward value to obtain the reward change trend of the evaluation network; Determining a status of an evaluation network based on a reward change trend of the evaluation network; If the evaluation network is determined to be stable, then determine whether the average reward of the target network meets the requirements of the dynamic threshold factor; If the average reward of the target network meets the requirements of the dynamic threshold factor, the weight parameters of the evaluation network are updated to the target network; If it is determined that the state of the evaluation network is unstable, or the average reward of the target network does not meet the requirements of the dynamic threshold factor, the weight parameters of the target network are updated to the evaluation network.

5. The heading control method of the trenching and cable laying robot according to claim 4, characterized in that: Determining the state of the evaluation network according to the reward change trend of the evaluation network includes: Comparing the reward change trend of the evaluation network with a preset stable positive threshold; If the reward change trend of the evaluation network is greater than the preset stable positive threshold, determining that the state of the evaluation network is stable; If the reward change trend of the evaluation network is not greater than the preset stable positive threshold, it is determined that the state of the evaluation network is unstable.

6. The heading control method of the trenching and cable laying robot according to claim 4, characterized in that: Determine whether the average reward of the target network meets the requirements of the dynamic threshold factor, including: The dynamic threshold factor is determined according to the reward change trend of the evaluation network, wherein the calculation formula of the dynamic threshold factor is: ， in, represents the dynamic threshold factor, Indicates the reward change trend of the evaluation network, represents the adjustable factor, Indicates the normalization progress of training; Determining an average reward change trend of the target network, and comparing the average reward change trend of the target network with the dynamic threshold factor; If the average reward change trend of the target network is greater than the dynamic threshold factor, it is determined that the average reward of the target network meets the requirements of the dynamic threshold factor; otherwise, it does not meet the requirements of the dynamic threshold factor.

7. A heading control device for a trenching and cable-laying robot, used to implement the heading control method for a trenching and cable-laying robot according to any one of claims 1 to 6, characterized in that: include: An acquisition module is used to obtain the real-time heading angle of the trenching and cable laying robot, and obtain heading error information based on the real-time heading angle and the expected heading angle; a calling and control module, configured to call an optimized PID controller and input the heading error information into the optimized PID controller to obtain a heading control strategy, wherein the optimized PID controller is capable of performing a comprehensive calculation on the heading error information according to input control parameters to obtain the heading control strategy, wherein the input control parameters are obtained by performing optimization training based on a reinforcement learning algorithm based on a three-pool structure, wherein the three-pool structure includes an excellent experience pool, a time decay pool, and a recent high reward pool, and the input control parameters include a differential parameter, an integral parameter, and a proportional parameter, and the integral parameter includes a dual-factor parameter consisting of a heading deviation and a current speed; The sending module is used to send the heading control strategy to the heading actuator of the trenching and cable-laying robot, and the heading actuator can adjust the heading angle according to the heading control strategy.

8. A trenching and cable laying robot, characterized in that: Including the heading control device of the trenching and cable-laying robot as described in claim 7.

Images