An intelligent control system for cutter suction dredger based on reinforcement learning

Abstract

This invention relates to the field of intelligent control for cutter suction dredgers. It proposes a reinforcement learning-based intelligent control system for cutter suction dredgers. The system comprises an operation optimization system, a cutter suction dredger intelligent control module, a process control unit, and an intelligent control subsystem. The intelligent control subsystem includes a traverse control subsystem, a mud pump control subsystem, a cutter control subsystem, a gate entry control subsystem, and a bridge control subsystem. The system utilizes only real-world ship data for self-learning model training. This system can quickly and effectively learn and achieve intelligent optimization goals under the complex environmental conditions of a real ship, reducing labor costs while improving dredging efficiency.

Description

[0001] This application is a division of Chinese patent application "An intelligent control system and method for a cutter suction dredger based on reinforcement learning", application number 2022101371338. Technical Field

[0002] The present invention belongs to the field of intelligent control of cutter suction dredgers, and in particular relates to an intelligent control system and control method based on reinforcement learning, which can perform self-learning and complete intelligent control of the cutter suction dredger, and belongs to the field of artificial intelligence and control technology. Background Art

[0003] Dredgers are essential equipment in water conservancy dredging. Cutter suction dredgers perform both dredging and slurry transportation in one operation, offering wide adaptability and economical efficiency. They play a crucial role in projects such as waterway dredging, port construction, and marine development.

[0004] The operating procedures of a cutter suction dredger are cumbersome, its working principles are complex, and its working environment is diverse. The dynamic characteristics of the actual dredging operation process are very complex, and the characteristics of the dredging equipment also vary significantly with the operating status and working location. The construction parameters in dredging projects can only be formulated for a specific dredger equipment and operating environment, with a narrow scope of applicability and no corresponding mathematical model. During the dredging operation, the main operating parameters of the dredger are flexibly determined by the operator based on his or her own experience, trial excavation conditions, and the actual operating results of the dredger. Due to the large number of influencing factors in the actual construction process and the complex interrelationships, the experience and theoretical level of dredging operators vary greatly. The actual output of manual operations is often lower than the designed output, and the dredging operation is inefficient, the degree of automation is low, and it is highly dependent on experience. Therefore, it is urgent to improve the intelligence level of dredging operations and use artificial intelligence technology to achieve intelligent optimization control of dredging vessels. Summary of the Invention

[0005] The purpose of this invention is to address the problems of the existing technology and propose an intelligent control system and method for a cutter suction dredger based on reinforcement learning. By controlling the cutter suction dredger's transverse subsystem, mud pump subsystem, cutter subsystem, bridge subsystem, and inlet control system, appropriate controllers are designed for different control subsystems. Reinforcement learning is then used to recommend intelligent and autonomous optimization of real-time construction parameters to achieve control of dredging process actions and important process parameters. This system eliminates the need for the driver to manually operate various control handles, greatly reducing the labor intensity of the staff and playing a positive role in improving the intelligence level of dredgers, increasing production efficiency, and reducing production costs.

[0006] The purpose of the invention is achieved through the following technical solutions:

[0007] An intelligent control method for a cutter suction dredger based on reinforcement learning, characterized by comprising the following steps:

[0008] S1. Use sensors to collect data from the cutter suction vessel during construction to form a raw database of the dredging operation process;

[0009] S2. Preprocess the raw data to filter out valid data within the lateral shift period and filter the data using a Kalman filter.

[0010] S3. Use the information gain method to select control parameters that are closely related to production and can be manually adjusted;

[0011] Information gain refers to the increase in benefits after adding information, or in other words, the reduction in uncertainty after adding information. The splitting information (also known as the node splitting information metric) is the information entropy of the samples at the node, which considers the number and size of branches when splitting an attribute. The information gain ratio is the ratio of information gain to splitting information, such that the importance of each attribute decreases as the splitting information increases. The parameters selected using the information gain ratio are traverse speed, mud pump speed, cutter speed, step distance, and bridge depth, which correspond to the cutter suction vessel's traverse subcontrol system, mud pump subcontrol system, cutter control system, entry subcontrol system, and bridge control system, respectively.

[0012] S4. After selecting the control variables, define the reinforcement learning action interval as [traverse speed, mud pump speed, auger speed, bridge depth, step distance]. This defines the variables that the agent can control in reinforcement learning.

[0013] S5. The reinforcement learning action interval is a continuous action space and meets the requirements of completeness, efficiency, and legality;

[0014] Among them, completeness refers to functional completeness (having basic functions) and timeliness completeness (fast response speed and short decision-making cycle under the premise of having basic functions);

[0015] Efficiency refers to the simplicity and efficiency of the action space, which can effectively reduce the difficulty of training and improve algorithm performance;

[0016] Legality refers to setting actions that are not allowed to occur in a certain state;

[0017] S6. Using the action intervals of reinforcement learning as input variables, an ELM (Extreme Learning Machine) neural network was used to build a production prediction model for cutter suction dredgers;

[0018] The input layer vector and output layer vector of ELM neural network are:

[0019] X = {traverse speed, mud pump speed, cutter speed, bridge depth, stepping distance};

[0020] Y = {instantaneous output};

[0021] Set the default number of neurons in the hidden layer to 30; set the activation function expression as follows:

[0022]

[0023] The advantage of ELM is that it is fast to calculate, and the random initialization of parameters makes ELM have good generalization;

[0024] S7. The reward function is the core of reinforcement learning. The learning goal of the reinforcement learning agent is to maximize the expected cumulative reward. The reward function expression is designed as follows:

[0025] R=OD c *DP f

[0026] Where R represents the reward value obtained by the agent when performing an action, that is, the evaluation of the current action;

[0027] O represents the instantaneous production of the cutter suction hopper predicted by the ELM neural network model. The higher the instantaneous production, the higher the reward value obtained by the system.

[0028] D is the Euclidean distance deviation between the control parameter at this moment and the control parameter at the previous moment. This variable is added to limit the change amplitude between two input signals and prevent the control parameter from jumping violently.

[0029] D c It is the multiplication coefficient of D, which can be adjusted according to the actual engineering conditions. The default value is 100.

[0030] P f Defined as a penalty function with a default value of 400. If the system control variable exceeds the limit, the agent will be immediately penalized. Adding a penalty function for control variable exceeding the limit can ensure that the optimal construction parameters recommended by reinforcement learning are within a safe range.

[0031] S8. The final reward of a reinforcement learning agent is observed only after multiple actions. For each action selected, the agent is divided into exploration (evenly distributing trial opportunities among all possible actions) and exploitation (taking only the action with the highest average reward from known experience). The distribution of action probabilities in this invention is based on the Boltzmann distribution, as follows:

[0032]

[0033] Where R(i) is the average reward after the current action is completed. The value of τ can be customized in the program. The smaller the τ is, the higher the probability that the action with the highest average reward will be selected. When τ approaches 0, it tends to be only exploited, and when τ approaches infinity, it tends to be only explored. The default value of τ is usually between 0.05 and 0.10.

[0034] S9. According to the current state S t , the reinforcement learning agent performs the current action A t Acting on the environment, the probability of performing different actions is P(k) calculated in the previous step; then the environment feedback corresponds to the state value S t and compare it with the reward value R t Feedback is given to the agent, and the environment moves to the next state S t+1 , and give a reward value R t+1 ; This gives the action state sequence of reinforcement learning:

[0035] τ={S1,A1,S2,A2,S3,A3,…,S n ,A n}

[0036] Where S1, S2, S3, ..., S n is the state sequence of the agent, A1, A2, A3, …, A n is the action sequence of the agent;

[0037] S10. The environment model of reinforcement learning consists of three parts: action space, state transition, and reward and punishment function. After building the environment model through the above steps, the reinforcement learning agent takes the random action A given in the previous step. i-1 Execute instructions and feedback status information S i-1 , gradually obtain the optimal strategy π through interactive learning with the reinforcement learning environment i As learning progresses, the probability of good actions being selected gradually increases, and bad actions are gradually eliminated, so the reward value R obtained by the machine i The gradual increase indicates that the agent gradually obtains the optimal construction parameters during the self-learning process;

[0038] S11. Set the maximum number of learning times for machine learning; if the cumulative reward value increases and gradually stabilizes, the learning termination condition is met; if not, return to step S6 and continue the cycle; if the maximum number of learning times is reached, end the reinforcement learning algorithm, adjust the parameters and restart learning; after meeting the learning termination condition, the action A obtained F The optimal dredging parameters (including traverse speed, mud pump speed, cutter speed, bridge depth and step distance) are found by machine learning.

[0039] S12. Set the control parameters of the dredger subsystem in the five control systems, namely the lateral movement control system, mud pump control system, cutter control system, bridge control system, and entry control system, and control the mechanical equipment through PLC so that the cutter suction dredger can perform intelligent dredging operations based on the optimal dredging parameters obtained by reinforcement learning.

[0040] A cutter suction dredger intelligent control system based on reinforcement learning is characterized by comprising an operation optimization system, a cutter suction dredger intelligent control module, a process control unit, and an intelligent control subsystem; the intelligent control subsystem includes a traverse control subsystem, a mud pump control subsystem, a cutter control subsystem, a dredge entry control subsystem, and a bridge control subsystem; the operation optimization system repeatedly optimizes control parameters online and iteratively based on intelligent optimization through reinforcement learning through the above steps; the intelligent control subsystem controls the dredging process with the optimized control parameters, executes the optimal dredging action, and ensures that the actual output of the cutter suction dredger is stably maintained at the optimal output value. The control process of the cutter suction dredger intelligent control module includes the following conventional steps (the process and each step itself are existing technologies): start the mud pump, start the cutter, lower the bridge, traverse, before reaching the trolley limit, traverse left and right, reach the trolley limit, traverse to the middle, end traverse left and right, advance the trolley, end traverse to the middle, retreat the trolley, change piles after the dredging period is completed, dredging is completed, raise the bridge, close the cutter, and close the mud pump. Each process control unit cooperates with the operation optimization system to form a top-down functionally complete control structure to realize the intelligent operation of the dredging operation process.

[0041] The present invention only requires real ship data to perform self-learning model training. It can quickly and effectively learn and achieve intelligent optimization goals under the complex environmental conditions of the real ship, thereby reducing labor costs and improving dredging construction efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 This is a control system structure block diagram of an intelligent control system and method for a cutter suction dredger based on reinforcement learning according to the present invention;

[0043] Figure 2 This is a schematic diagram of the structure of the intelligent and autonomous optimization of dredging parameters of a cutter suction dredger based on reinforcement learning according to the present invention;

[0044] Figure 3 This is a functional architecture diagram of an intelligent control system for a cutter suction dredger based on reinforcement learning and a method thereof;

[0045] Figure 4 This is a diagram of the lateral control subsystem of an intelligent control system and method for a cutter suction dredger based on reinforcement learning according to the present invention;

[0046] Figure 5This is a diagram of a mud pump control subsystem of an intelligent control system and method for a cutter suction dredger based on reinforcement learning according to the present invention;

[0047] Figure 6 This is a diagram of the cutter control subsystem of an intelligent control system and method for a cutter suction dredger based on reinforcement learning according to the present invention;

[0048] Figure 7 This is a diagram of the bridge control subsystem of an intelligent control system and method for a cutter suction dredger based on reinforcement learning according to the present invention;

[0049] Figure 8 This is a diagram of the entry control subsystem of an intelligent control system and method for a cutter suction dredger based on reinforcement learning according to the present invention;

[0050] Figure 9 This is a comparison chart of the output between the reinforcement learning control results and the actual manual operation obtained through the control simulation test. DETAILED DESCRIPTION

[0051] The technical solution provided by this application will be further described below in conjunction with specific embodiments and accompanying drawings. The advantages and features of this application will become more apparent with reference to the following description.

[0052] It should be noted that the embodiments of the present application have better feasibility and are not intended to limit the present application in any form. The technical features or combinations of technical features described in the embodiments of the present application should not be considered in isolation, and they can be combined with each other to achieve better technical effects. The scope of the preferred embodiments of the present application may also include other implementations, and this should be understood by those skilled in the art in the technical field to which the embodiments of the present application belong.

[0053] The present invention provides an intelligent control method for a cutter suction dredger based on reinforcement learning, such as Figure 1 The following steps are involved:

[0054] S1. Use sensors to collect data from the cutter suction dredging vessel construction process to form an original database of the dredging operation process.

[0055] S2. Perform data preprocessing on the original data, filter out the valid data within the lateral movement period, and use the Kalman filter method to filter the data.

[0056] S3. Use the information gain rate method to select control parameters that are closely related to production and can be manually controlled. Information gain refers to how much profit can be increased after adding information, that is, how much uncertainty can be reduced after adding information; the split information amount (also known as the node split information metric) is the information entropy of the sample on the node, which considers the number and size information of the branches when the attribute is split; the information gain rate is the ratio of information gain to split information amount, so that the importance of each attribute decreases as the split information amount increases. The information gain rate method is often used to judge the importance of variables. The parameters selected using the information gain rate are transverse speed, mud pump speed, auger speed, step distance and bridge depth, which correspond to the transverse sub-control system, mud pump sub-control system, auger sub-control system, entry sub-control system and bridge control system of the cutter suction dredger, respectively.

[0057] S4. After selecting the control variables, define the reinforcement learning action interval as [traverse speed, mud pump speed, auger speed, bridge depth, step distance]. This defines the variables that the agent can control in reinforcement learning.

[0058] S5. The reinforcement learning action range is a continuous action space and meets the requirements of completeness, efficiency, and legality. Completeness refers to functional completeness (having basic functions) and timeliness completeness (fast response speed and short decision-making cycle under the premise of having basic functions). Efficiency refers to the simplicity and efficiency of the action space, which can effectively reduce the difficulty of training and improve algorithm performance. Legality refers to setting actions that are not allowed to occur under specific conditions. For example, when a cutter suction boat moves across the left / right line, the lateral movement speed should not be too fast.

[0059] S6. Using the action interval of reinforcement learning as the input variable, the ELM (Extreme Learning Machine) neural network is used to build a production prediction model for cutter suction dredgers. The input layer vector and output layer vector of the ELM neural network are:

[0060] X = {traverse speed, mud pump speed, cutter speed, bridge depth, stepping distance};

[0061] Y = {instantaneous output}.

[0062] Set the default number of neurons in the hidden layer to 30. Set the activation function expression as follows:

[0063]

[0064] The advantage of ELM is that it is fast to calculate, and the random initialization of parameters makes ELM have good generalization;

[0065] S7. The reward function is the core of reinforcement learning. The learning goal of the reinforcement learning agent is to maximize the expected cumulative reward. The reward function expression is designed as follows:

[0066] R=OD c *DP f

[0067] Where R represents the reward value obtained by the agent when performing an action, that is, the evaluation of the quality of the current action.

[0068] O represents the instantaneous output of the cutter suction dredge predicted by the ELM neural network model. The higher the instantaneous output, the higher the reward value the system receives.

[0069] D is the Euclidean distance deviation between the control parameter at this moment and the control parameter at the previous moment. This variable is added to limit the change amplitude between the two input signals and prevent the control parameter from jumping drastically.

[0070] D c It is the multiplication coefficient of D, which can be adjusted according to the actual engineering conditions. The default value is 100.

[0071] P f Defined as a penalty function with a default value of 400, if the system control variable exceeds the limit, the agent will be penalized immediately. Adding a penalty function for control variable exceeding the limit can ensure that the optimal construction parameters recommended by reinforcement learning are within a safe range.

[0072] S8. The final reward of a reinforcement learning agent is observed only after multiple actions. For each action selected, the agent is divided into exploration (evenly distributing the opportunity to try each possible action) and exploitation (only taking the action with the highest average reward from known experience). The distribution of action probabilities in this invention is based on the Boltzmann distribution, as follows:

[0073]

[0074] Where R(i) is the average reward after completing the current action. The program can customize the value of τ. The smaller τ is, the more likely actions with high average rewards are selected. As τ approaches 0, the system tends to focus on exploitation, while as τ approaches infinity, the system tends to focus on exploration. The default value of τ is typically between 0.05 and 0.10.

[0075] S9. According to the current state S t , the reinforcement learning agent performs the current action A t Acting on the environment, the probability of performing different actions is P(k) calculated in the previous step. Then the environment feedback corresponds to the state value S t and compare it with the reward value R t Feedback is given to the agent, and the environment moves to the next state S t+1 , and give a reward value R t+1 . This gives the action state sequence of reinforcement learning:

[0076] τ={S1,A1,S2,A2,S3,A3,…,S n ,A n}

[0077] Where S1, S2, S3, ..., S n is the state sequence of the agent, A1, A2, A3, …, A n is the action sequence of the agent, such as Figure 2 .

[0078] S10. The environment model of reinforcement learning consists of three parts: action space, state transition and reward and punishment function. After building the environment model through the above steps, the reinforcement learning agent takes the random action A given in the previous step. i-1 Execute instructions and feedback status information S i-1 , gradually obtain the optimal strategy π through interactive learning with the reinforcement learning environment i As learning progresses, the probability of good actions being selected gradually increases, and bad actions are gradually eliminated, so the reward value R obtained by the machine is i It gradually increases, which means that the intelligent agent gradually obtains the optimal construction parameters during the self-learning process.

[0079] S11. Set the maximum number of learning times for machine learning. If the cumulative reward value increases and gradually stabilizes, the learning termination condition is met. If not, return to step 6 and continue the cycle. If the maximum number of learning times is reached, the reinforcement learning algorithm ends. At this time, consider adjusting the parameters and restarting the learning. After the learning termination condition is met, the action A obtained F The optimal dredging parameters (including traverse speed, mud pump speed, cutter speed, bridge depth and step distance) are found by machine learning.

[0080] S12. In the traverse control system, mud pump control system, cutter control system, bridge control system, entry control system ( Figure 4 This is the lateral control subsystem diagram, Figure 5 This is the mud pump control subsystem diagram, Figure 6 This is the reamer control subsystem diagram, Figure 7 This is the bridge control subsystem diagram, Figure 8 The five control systems (Figure 1) set the various control parameters in the dredger subsystem and control the mechanical equipment through PLC, so that the cutter suction dredger can perform intelligent dredging operations based on the optimal dredging parameters obtained by reinforcement learning.

[0081] A cutter suction dredger intelligent control system based on reinforcement learning includes an operation optimization system, a cutter suction dredger intelligent control module, a process control unit, and an intelligent control subsystem; the intelligent control subsystem includes: a traverse control subsystem, a mud pump control subsystem, a auger control subsystem, a gate entry control subsystem, and a bridge control subsystem; the operation optimization system repeatedly optimizes control parameters online and iteratively based on intelligent optimization through reinforcement learning through the above steps; the intelligent control subsystem controls the dredging process with the optimized control parameters, executes the optimal dredging action, and ensures that the actual output of the cutter suction dredger is stably maintained at the optimal output value. The control process of the cutter suction dredger intelligent control module includes the following conventional steps (the process and each step itself are existing technologies): start the mud pump, start the auger, lower the bridge, traverse, before reaching the trolley limit, traverse left and right, reach the trolley limit, traverse to the middle, end traverse left and right, advance the trolley, end traverse to the middle, retreat the trolley, change piles after the dredging is completed, dredging is completed, raise the bridge, close the auger, and close the mud pump. Each process control unit cooperates with the operation optimization system to form a top-down functionally complete control structure to realize the intelligent operation of the dredging operation process.

[0082] The present invention has the following advantages and effects:

[0083] (1) An ELM neural network and reinforcement learning algorithm were developed for the dredging process of a cutter suction dredger. Using a self-designed reinforcement learning reward function, autonomous learning and autonomous optimization under multi-constraint conditions in the nonlinear complex system engineering of the cutter suction dredger were achieved.

[0084] (2) The control system of the cutter suction dredger based on reinforcement learning can realize the intelligent control of the cutter suction dredger's transverse movement, cutter, entry, bridge and mud pump control systems, as well as the combination of multiple control systems, while ensuring the safety of construction.

[0085] (3) The system does not require the completion operator to operate various control handles. It can perform dredging operations flexibly and quickly by relying on machine learning methods, thereby reducing the labor intensity of dredging operators and improving dredging construction efficiency;

[0086] (4) The parameters optimized by reinforcement learning can quickly respond to dynamic environmental changes. The underlying reason is that the intelligent agent can evaluate the current state and find a strategy that maximizes future returns, thereby flexibly adjusting the control parameters to maximize expected returns. In contrast, operator decisions rely on data from the operation panel and lack future predictions.

[0087] Figure 9 In order to obtain a comparison chart of the output between the reinforcement learning control results and the actual manual operation through control simulation experiments, it is proved that the instantaneous output controlled by the optimal parameters of reinforcement learning is higher than that of manual operation, and the instantaneous output value can be maintained at a high point, which can effectively improve the dredging output of the dredger.

Claims

1. An intelligent control system for a cutter suction dredger based on reinforcement learning. The cutter suction dredger uses sensors to collect data during the construction process of the cutter suction dredger to form an original database of the dredging operation process; characterized in that: Through the ELM neural network and reinforcement learning algorithm of the dredging operation process, using the independently designed reinforcement learning reward function, autonomous learning and autonomous optimization under multi-constraint conditions in the nonlinear complex system engineering of the cutter suction dredger are achieved; It includes an operation optimization system, a cutter suction dredger intelligent control module, a process control unit and an intelligent control subsystem; The operation optimization system runs an intelligent control method for a cutter suction dredger, which performs intelligent optimization based on reinforcement learning and iteratively optimizes control parameters online; The control process of the intelligent control module of the cutter suction dredger includes the following conventional steps: start the mud pump, start the auger, put the bridge, move horizontally, if the trolley limit is not reached, move horizontally to the left and right, reach the trolley limit, move horizontally to the middle, end the left and right movement, move the trolley in, move horizontally to the middle, retract the trolley, change piles after the dredging is completed, dredging is completed, raise the bridge, close the auger, and close the mud pump; The process control unit and the operation optimization system cooperate with each other to form a top-down functionally complete control structure to realize the intelligent operation of the dredging operation process; The intelligent control subsystem includes: a traverse control subsystem, a mud pump control subsystem, a reamer control subsystem, a customs entry control subsystem and a bridge control subsystem; The control parameters of the cutter suction dredger subsystem are set in the five control subsystems, namely the traverse control subsystem, the mud pump control subsystem, the cutter control subsystem, the bridge control subsystem, and the access control subsystem. The traverse control subsystem, the mud pump control subsystem, the cutter control subsystem, the access control subsystem, and the bridge control subsystem control mechanical equipment through PLC, so that the cutter suction dredger can perform intelligent dredging operations based on the optimal dredging parameters obtained by reinforcement learning. The intelligent control subsystem controls the dredging process with the control parameters optimized by the operation optimization system through the cutter suction dredger intelligent control method, executes the best dredging action, and makes the actual output of the cutter suction dredger stably maintained at the optimal output value.

2. The intelligent control system for a cutter suction dredger based on reinforcement learning according to claim 1, characterized in that: The intelligent control method for the cutter suction dredger comprises the following steps: Perform data preprocessing on the original data, filter out the valid data within the lateral movement period, and use the Kalman filter method to filter the data; Furthermore, the information gain method was used to select the control parameters that are closely related to the output and can be manually adjusted. The parameters selected by the information gain method are the traverse speed, mud pump speed, cutter speed, step distance and bridge depth, which correspond to the traverse control subsystem, mud pump control subsystem, cutter control subsystem, entry control subsystem and bridge control subsystem of the cutter suction dredger respectively. Furthermore, after selecting the control variables, the reinforcement learning action interval is defined as [traverse speed, mud pump speed, auger speed, bridge depth, step distance], which defines the variables controlled by the agent in reinforcement learning. Furthermore, the action interval of reinforcement learning is used as the input variable, and the ELM neural network is used to build a production prediction model for the cutter suction dredger; the input layer vector and output layer vector of the ELM neural network are: X = {traverse speed, mud pump speed, cutter speed, bridge depth, step distance}; Y = {instantaneous production}; Set the default number of neurons in the hidden layer to 30; Set the activation function The expression is as follows: ; Design reward function expression: in Represents the reward value obtained by the agent when performing an action, that is, the evaluation of the current action; represents the instantaneous output of the cutter suction dredger predicted by the ELM neural network model. The higher the instantaneous output, the higher the reward value obtained by the system. The Euclidean distance deviation between the control parameter at this moment and the control parameter at the previous moment is added to limit the change range between the two input signals and prevent the control parameter from jumping violently. for The multiplication coefficient is adjusted according to the actual engineering conditions. The default value is ; Defined as a penalty function, the default value is ,If the system control variable exceeds the limit, the agent will be immediately given a penalty value; Adding a penalty function for exceeding the control variable limit can ensure that the optimal construction parameters recommended by reinforcement learning are within a safe range; Furthermore, the final reward of the reinforcement learning agent is observed after multiple actions. For each current action selection, it is divided into exploration and exploitation. Exploration is to evenly distribute the trial opportunities to each executed action, and exploitation is to only take the action with the highest average reward obtained from known experience. The distribution of action probabilities is based on the Boltzmann distribution, and the equation is as follows: In the formula The average reward after the current action is completed; customized in the program Value size, setting The smaller it is, the higher the probability that the action with high average reward will be selected. Approaching When τ approaches infinity, it tends to only explore. Value to between; Furthermore, according to the current status , the reinforcement learning agent performs the current action Acting on the environment, the probability of performing different actions is calculated in the previous step ; Then the environment feedback corresponds to the state value and compare it with the reward value Feedback is given to the agent, while the environment moves to the next state , and give reward value ; This results in the action state sequence of reinforcement learning: In the formula is the state sequence of the agent, is the action sequence of the agent; The environment model of reinforcement learning consists of three parts: action space, state transition, and reward and punishment function; After building the environment model through the above steps, the reinforcement learning agent takes random actions according to the previous step. Execute instructions and provide status information , gradually obtain the optimal strategy through interactive learning with the reinforcement learning environment As learning progresses, the probability of good actions being selected gradually increases, and bad actions are gradually eliminated, so the reward value obtained by the machine The gradual increase indicates that the agent gradually obtains the optimal construction parameters during the self-learning process; Furthermore, the maximum number of learning times for machine learning is set; if the cumulative reward value increases and gradually stabilizes, the learning termination condition is met; if not, the process returns to the ELM neural network and uses the action interval of reinforcement learning as the input variable to continue the cycle; if the maximum number of learning times is reached, the reinforcement learning algorithm is terminated, and the parameters are adjusted and learning is restarted; After the learning termination condition is met, the action obtained , including traverse speed, mud pump speed, cutter speed, bridge depth and step distance, which are the optimal dredging parameters found by machine learning.

Images