Unmanned ship intelligent control method based on improved low-entropy strategy and network structure

Abstract

The invention relates to an unmanned ship intelligent control method based on an improved low entropy strategy and a network structure, and the method comprises the following steps: setting an action space and a state space of an unmanned ship, and setting a reward function according to a multi-task target demand of the unmanned ship, and an unmanned ship controller combining a low-entropy motion selection mechanism and an improved LSTM network is adopted to perform intelligent control until a target point is reached, the intelligent control process of the unmanned ship is completed, and the unmanned ship controller comprises an Actor network, a Critic network and the improved LSTM network. Compared with the prior art, the unmanned ship has the advantages that the multi-task target requirement can be met in a complex water area, intelligent control is achieved, and the like.

Description

Technical Field

[0001] The invention relates to the field of intelligent control and ship and ocean engineering technology, and in particular to an unmanned boat intelligent control method based on an improved low entropy strategy and a network structure. Background Art

[0002] In recent years, with the rapid development of unmanned intelligent control technology, surface unmanned boats have received more and more attention. Compared with drones and unmanned vehicles, the research on unmanned boats is still in a rapid development stage. As a small surface operation task platform with autonomous planning and autonomous navigation capabilities, unmanned boats have nonlinear motion in complex and changing water environments, and the interference they are subject to is uncertain. Therefore, the autonomous navigation and intelligent control technology of unmanned boats is particularly important.

[0003] Patent CN114077258B discloses a method for controlling the posture of an unmanned boat based on the reinforcement learning PPO2 algorithm. The method uses the PPO2 algorithm to train the posture controller of the unmanned boat by setting the action space and state space, setting the reward function, and designing the deep neural network architecture, and can adaptively adjust its posture. Patent CN115454092B discloses a method and system for planning the path of an unmanned boat based on the improved RRT algorithm. The method uses the Collision function to detect the collision of obstacles on the new node Qnew; the collision function is used to detect the collision of obstacles on the new node Qnew. According to the terminal connection probability q, the termination detection method of the algorithm is selected. The path is traced back from the terminal point Qgoal to the starting point Qstart in the random tree T. The path is traced back from the terminal point Qgoal to the starting point Qstart in the random tree T, which can reduce the path planning time of the unmanned boat. Patent application CN117991780A discloses a method for optimizing the path of an unmanned boat based on the GWO algorithm. The method includes: selecting a mixture of a great circle route and a constant direction line as the initial; constructing the objective function; optimizing the GWO algorithm, which can achieve global optimal path planning, etc.

[0004] However, the machine learning method used in the above method is not only difficult to control the unmanned boat to achieve the comprehensive autonomous navigation requirements of safety, speed and energy saving in complex marine environments, but also cannot take into account the multi-task target requirements of the unmanned boat at the same time.

[0005] Therefore, there is an urgent need for an improved intelligent control method for unmanned boats that can meet the needs of multiple mission objectives of unmanned boats in complex marine environments and achieve the optimal intelligent control effect at the same time. Summary of the invention

[0006] The purpose of the present invention is to provide an unmanned boat intelligent control method based on improved low entropy strategy and network structure to achieve multi-task goals in complex waters.

[0007] The purpose of the present invention can be achieved by the following technical solutions:

[0008] An unmanned boat intelligent control method based on improved low entropy strategy and network structure includes the following steps: Set the action space and state space of the unmanned boat, set the reward function according to the multi-task target requirements of the unmanned boat, and use the unmanned boat controller that combines the low entropy action selection mechanism and the improved LSTM network for intelligent control until the unmanned boat reaches the target point, completing the intelligent control process of the unmanned boat; The unmanned boat controller includes an Actor network, a Critic network and an improved LSTM network. The process of intelligent control of the unmanned boat controller includes: A batch of data is collected from the experience replay pool, input into the improved LSTM network for processing, and then the control action is output through the Actor network, and the low entropy action selection mechanism is used to select the low entropy action to obtain the control output , to control the operation of the unmanned boat, wherein the experience replay pool is formed by interacting with the environment and combining the action space, state space and reward function; Collect a batch of data from the experience replay pool, input it into the Critic network for value estimation, and guide the Actor network to output control actions; Repeat the above control process until the unmanned boat reaches the target point.

[0009] Furthermore, the action space of the unmanned boat includes the control rate of the left motor of the unmanned boat , right motor control rate , distance to target point and the distance to obstacles , the state space includes the unmanned boat linear speed , angular velocity , distance to target point , Distance from obstacles , left motor control rate and right motor control rate .

[0010] Furthermore, the reward function is expressed as: , in: R 1 is the reward function for the unmanned boat to reach the target point, and the expression is: , R 2 is the reward function of the distance of the unmanned boat away from obstacles, and the expression is: , R3 is the reward function for the control of the linear velocity and angular velocity of the unmanned boat, and the expression is: , R 4 is the reward function of the unmanned boat navigation time control, and its expression is: , In the formula, R is the total reward function, is the weight of the reward for the unmanned boat to reach the target point in the total reward, It is the reward for the unmanned boat to reach the target point. is the weight of the reward for the distance of the unmanned boat away from obstacles in the total reward, It is the distance when the unmanned boat is within the set distance of the obstacle. It is the reward for the unmanned boat to collide with an obstacle. is the weight of the reward for the stability of the unmanned boat's linear speed in the total reward, is the current linear speed of the unmanned boat, is the linear speed of the unmanned boat at the previous moment, is the weight of the reward of the current linear speed of the unmanned boat in the total reward, is the weight of the reward for the angular velocity stability of the unmanned boat in the total reward, is the current angular velocity of the unmanned boat, is the angular velocity of the unmanned boat at the previous moment, is the weight of the reward for the distance between the unmanned boat and the target point in the total reward; It is the reward for the distance between the unmanned boat and the target point.

[0011] Furthermore, the Actor network is a dual-strategy network, including an Actor_1 subnetwork and an Actor_2 subnetwork, and the Actor_1 subnetwork and the Actor_2 subnetwork interact with the environment in parallel and output control actions.

[0012] Furthermore, the Critic network includes a Critic_1 subnetwork and a Critic_2 subnetwork, and the Critic_1 subnetwork and the Critic_2 subnetwork perform the value estimation in parallel, wherein the value estimation result includes a state value function and the state-action value function , the expressions are: , , In the formula, It is in strategy Next action Seek hope, is the temperature coefficient, For strategy In Status Next select action The logarithmic probability of It is the transfer function Next moment status Seek hope, is in state Next action Instant rewards, is the discount factor, It is the estimated state value of the state at the next moment.

[0013] Furthermore, the operation expression for selecting a low entropy action using the low entropy action selection mechanism is: , , In the formula, For the status Next Strategy The entropy of To select in state Next Strategy The minimum entropy value in It is in strategy Next action expectations, It’s a strategy In Status Next select action The logarithmic probability of It’s a strategy In Status Next select action The weight of is the action with the lowest entropy value, is to choose the strategy with the lowest entropy value Selected Action .

[0014] Furthermore, the improved LSTM network improves the updating method of the cell state in the input gate calculation process, and the updated expression of the improved cell state is: , , In the formula, The cell state of the current time step for the improved LSTM network structure, is the input of the forget gate, is the cell state at the previous moment, is the output of the input gate, is the hyperbolic tangent function, is the weight matrix of the hidden state, which is updated by the back-propagation algorithm. is the hidden state at the previous moment, is the input of the current time step, is the bias term, which is updated by the back-propagation algorithm. is the memory matrix of the previous moment, for Function operation, is the memory matrix of the previous moment.

[0015] Furthermore, the step of guiding the Actor network to output a control action refers to updating the network parameters based on the value evaluation result to select the optimal control action. The specific steps include: For each batch of data, after being processed by the improved LSTM network and Actor network, the network parameters of the unmanned boat controller are updated in combination with the value evaluation results, wherein the expression of the loss function of the network parameter update is: , In the formula, is the neural network parameter The loss function is are neural network parameters; is from the experience replay pool, i.e. The sample data obtained by sampling is is the state of the current time step, is the action at the current time step, Is to perform an action Instant rewards, It is the state of the next moment; is the replay pool from the experience, i.e. sampling And in the strategy Seek expectations, It is all the data stored in the experience replay pool. It is the action of the next moment. is in state Lower Parameters Strategy , is the current time step value, is the discount factor, It's the next moment Smaller value The expected return of the function, is the temperature coefficient, For strategy In Status Next select action The logarithmic probability of .

[0016] Furthermore, the objective function updated by the unmanned boat controller is: , In the formula, is the objective function of policy optimization, Is to obey the strategy The state-action group under the joint distribution of state and action The target value of is expected, It is in strategy The joint distribution of states and actions under ; is the discount factor, is the state of the current time step, is the action at the current time step, Is to perform an action Instant rewards, is the temperature coefficient, It’s a strategy In Status The entropy under .

[0017] Furthermore, the training process of the unmanned boat controller includes: 1) Add wind and flow interference to the environment, where the expression of wind interference in the environment is: , The expression of flow interference in the environment is: , In the formula, Because of wind interference, For flow interference, is the wind coefficient, is the heading angle of the unmanned boat, is the wind speed, is the water flow coefficient, is the angle between the water flow direction and the heading of the unmanned boat, is the water flow velocity; 2) Set the training cycle N; 3) Interact with the environment and put the interaction information into the experience replay pool in chronological order; 4) Determine whether the experience replay pool is full. If so, extract a batch of data from the experience replay pool to iteratively update the network parameters. If not, execute step 2); 5) Determine whether the set training cycle N has been reached, If yes, then further determine whether the multi-objective requirements are met. If yes, then output the control result to obtain the control effect. If no, then execute step 3); If not, execute step 3) until the training cycle N is reached and the multi-objective requirements are met, and the training process is completed.

[0018] Compared with the prior art, the present invention has the following beneficial effects:

[0019] (1) The present invention is based on the self-learning ability of the reinforcement learning framework, designs a reward function for the unmanned boat to meet the multi-objective requirements, so as to realize the coordinated control of the multi-task objectives of the unmanned boat, and introduces an improved LSTM network structure in the reinforcement learning framework to capture the dependency of data, thereby improving the efficiency of samples, and uses a low-entropy action selection mechanism in the reinforcement learning framework to select low-entropy actions, thereby solving the problems of algorithm instability and difficulty in convergence caused by blindly exploring more actions due to over-estimation of entropy values, and improving the information processing ability of the unmanned boat in complex waters. Therefore, the present invention can realize the intelligent control of the unmanned boat to meet the multi-task objectives in complex waters.

[0020] (2) The present invention integrates spatiotemporal data using an improved LSTM network structure and improves the cell state update method in the input gate through a memory matrix, so that the improved LSTM network structure can better utilize historical data information, further improve sample efficiency, and enhance the autonomous control capability of the unmanned boat.

[0021] (3) In the Actor-Critic network structure of the present invention, reinforcement learning is performed by executing Actor_1 and Actor_2, Critic_1 and Critic_2 in parallel, which can interact and learn with the environment at the same time, accelerate the learning process, and reduce the strategy fluctuation caused by the randomness or instability of a single Actor and enhance robustness. The Actor-Critic network structure of the present invention is more suitable for complex waters, and by combining a low-entropy action selection mechanism, the unmanned boat controller further improves its autonomous control and decision-making performance, as well as faster and safer path planning and obstacle avoidance capabilities.

[0022] (4) The present invention also takes into account the wind and flow interference in the actual environment. It can automatically adjust the control strategy by interacting with the wind and flow interference in the environment, so that the unmanned boat can quickly adapt to environmental changes in complex waters and improve mission efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a schematic diagram of the method flow of the present invention; Figure 2 It is a schematic diagram of the principle of the present invention. DETAILED DESCRIPTION

[0024] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.

[0025] This example provides an intelligent control method for unmanned boats based on improved low entropy strategy and network structure. Figure 1 As shown, the following steps are included:

[0026] Step S1: Set the action space and state space of the unmanned boat.

[0027] Specifically include the following: S101, set the unmanned boat action space, including: unmanned boat left motor control rate , right motor control rate , the distance from the target point , and the distance to the obstacle ; Control rate of the left motor of the unmanned boat and right motor control rate The action spaces are , the distance from the target point The action space is , distance from obstacles The action space is .

[0028] S102, setting the state space of the unmanned boat, including: the linear speed of the unmanned boat , angular velocity , the distance from the target point , distance from obstacles , left motor control rate and right motor control rate ; Unmanned boat linear speed The state space of , angular velocity The state space of , left motor control rate and right motor control rate The state space is , the distance from the target point The state space of , distance from obstacles The state space of .

[0029] Step S2: Set a reward function according to the multi-task target requirements of the unmanned boat.

[0030] Specifically include the following:

[0031] S201, set the reward function for the unmanned boat to reach the target point. In order to make the unmanned boat reach the target point, the reward function of the task goal 1 is R 1 is set to: , in, is the weight of the reward for the unmanned boat reaching the target point in the total reward; It is the reward for the unmanned boat to reach the target point.

[0032] S202, set the reward function for the distance of the unmanned boat away from obstacles. In order to keep the unmanned boat away from obstacles for more than 50 meters, the reward function of the second task objective is R 2Settings are: , in, is the weight of the reward for mission objective 2 in the total reward; It is the distance when the unmanned boat is within 50m of the obstacle; It is the reward for the unmanned boat colliding with an obstacle.

[0033] S203, set the reward function of the unmanned boat linear speed and angular speed control. In order to make the unmanned boat control as stable as possible, the reward function of the task goal three is R 3 settings are: , in, is the weight of the reward for the stability of the unmanned boat's linear speed in the total reward; is the current linear speed of the unmanned boat, is the linear speed of the unmanned boat at the previous moment; is the weight of the reward for the angular velocity stability of the unmanned boat in the total reward; is the current angular velocity of the unmanned boat, is the angular velocity of the unmanned boat at the previous moment.

[0034] S204, setting the reward function of the unmanned boat navigation time control, in order to make the unmanned boat reach the target point quickly, the reward function of the task goal four is R 4 settings are: , in, is the weight of the reward of the current linear speed of the unmanned boat in the total reward; is the current linear speed of the unmanned boat; is the weight of the reward for the distance between the unmanned boat and the target point in the total reward; It is the reward for the distance between the unmanned boat and the target point.

[0035] S205, set the multi-task target requirement reward function for the unmanned boat to reach the target point, the distance away from obstacles, the linear speed, the angular speed and the sailing time, and set the reward function as follows: , in, R is the total reward required by the multi-task objective; is the weight of the reward for the unmanned boat to reach the target point in the total reward, which is set to 0.3; It is the reward for the unmanned boat to reach the target point; is the weight of the reward for the distance between the unmanned boat and the obstacle in the total reward, which is set to 0.2; It is the distance when the unmanned boat is within 50m of the obstacle; It is the reward for the collision between the unmanned boat and the obstacle; is the weight of the reward for the stability of the unmanned boat's linear speed in the total reward, which is set to 0.1; is the current linear speed of the unmanned boat; is the linear speed of the unmanned boat at the previous moment; is the weight of the reward of the current linear speed of the unmanned boat in the total reward, which is set to 0.1; is the weight of the reward for the angular velocity stability of the unmanned boat in the total reward, which is set to 0.1; is the current angular velocity of the unmanned boat; is the angular velocity of the unmanned boat at the previous moment; is the weight of the reward for the distance between the unmanned boat and the target point in the total reward, which is set to 0.2; It is the reward for the distance between the unmanned boat and the target point.

[0036] Step S3: combining the low entropy action selection mechanism and the improved LSTM network structure to improve the unmanned boat intelligent control method.

[0037] Specifically include the following:

[0038] S301. Combined with the low entropy action selection mechanism, the unmanned boat intelligent control method is improved, and the low entropy action selection mechanism is set to: , , in, is in state Next Strategy Entropy of Is selected in the state Next Strategy The smallest entropy value in ; It is in strategy Next action expectations; It’s a strategy In Status Next select action The logarithmic probability of It’s a strategy In Status Next select action The weight of The action with the lowest entropy value; is to choose the strategy with the lowest entropy value Selected Action .

[0039] S302. Combined with the improved LSTM network structure, the unmanned boat intelligent control method is improved. The input gate calculation process is improved through the improved LSTM network structure. The specific settings are: , , in, It is the cell state of the improved LSTM network structure at the current time step; is the input of the forget gate; is the cell state at the previous moment; is the output of the input gate; is the hyperbolic tangent function; is the weight matrix of the hidden state, which is updated by the back-propagation algorithm; is the hidden state at the previous moment; is the input of the current time step; is the bias term, which is updated by the back-propagation algorithm; is the memory matrix of the current time step; yes Function operation; is the memory matrix of the previous moment.

[0040] S303, such as Figure 2 As shown in the figure, the unmanned boat intelligent controller includes the Actor network structure, the Critic network structure and the improved LSTM network structure. The Actor network structure includes the Actor_1 subnetwork and the Actor_2 subnetwork. Each subnetwork is executed in parallel and has a three-layer structure. According to the requirements of the unmanned boat intelligent controller, the input layer is designed to have 4 nodes, the hidden layer is designed to have 256 nodes, and the output layer is designed to have 2 nodes. The left motor control rate is collected from the experience playback pool. , right motor control rate , the distance from the target point , and the distance to the obstacle After being processed by the improved LSTM network structure, the control action is output through Actor_1 and Actor_2, and the entropy value of each control action is compared, and the low entropy action is selected to obtain the control output After that, it needs to be denormalized to get the actual speed of the motor and the control rate of the left motor of the unmanned boat. and right motor control rate The Critic network structure includes the Critic_1 subnetwork and the Critc_2 subnetwork. Each subnetwork is executed in parallel and has a three-layer structure. The input layer has 6 nodes, the hidden layer has 256 nodes, and the output layer has 1 node. The linear speed of the unmanned boat is collected from the experience playback pool. , angular velocity , the distance from the target point , distance from obstacles , left motor control rate and right motor control rate , output state value function and the state-action value function , used to update the parameters of the Actor network so that the Actor network can learn better strategies.

[0041] After the unmanned boat intelligent controller collects a batch of data, its critic network structure outputs the state value function and the state-action value function , the formula is as follows: , , in, It is in strategy Next action Seek expectations; is the temperature coefficient, which is generally 0.2; For strategy In Status Next select action The logarithmic probability of It is the transfer function Next moment status Seek expectations; is in state Next action Instant rewards; is the discount factor, usually 0.99; It is the estimated state value of the state at the next moment.

[0042] After the unmanned ship intelligent controller collects a batch of data, it is input into the improved LSTM network node for processing, and then input into the Actor network structure to update the parameters of the Actor_Critic network structure. Its loss function is as follows: , in, is the neural network parameter The loss function of are neural network parameters; is from the experience replay pool, i.e. The sample data obtained by sampling is is the state of the current time step, is the action at the current time step, Is to perform an action Instant rewards, It is the state of the next moment; is the replay pool from the experience, i.e. sampling And in the strategy Seek expectations; It is all the data stored in the experience replay pool; It is the action of the next moment; is in state Lower Parameters Strategy ; is the current time step value; is the discount factor, usually 0.99; It's the next moment Smaller value The expected return of the function; is the temperature coefficient, which is generally 0.2; For strategy In Status Next select action The logarithmic probability of .

[0043] Finally, the objective function updated by the unmanned boat intelligent controller is: , in, is the objective function of policy optimization; Is to obey the strategy The state-action group under the joint distribution of state and action Find the expected target value of ; It is in strategy The joint distribution of states and actions under ; is the discount factor, usually 0.99; is the state of the current time step, is the action at the current time step; Is to perform an action Instant rewards; is the temperature coefficient, which is generally 0.2; It’s a strategy In Status The entropy under .

[0044] Step S4: Based on the improved unmanned boat intelligent control method, the unmanned boat intelligent controller is trained.

[0045] Specifically include the following:

[0046] S401. Add wind and flow interference to the environment. Wind interference in the environment The formula is: , in, is the wind coefficient, which is usually in the range of 1.0 to 1.5, depending on the vessel and wind strength; is the heading angle of the unmanned boat; It's the wind speed.

[0047] Environmental flow disturbance The formula is: , in, is the current coefficient, usually in the range of 0.01 to 0.05, depending on the size of the vessel and the current; It is the angle between the water flow direction and the heading of the unmanned boat; is the water flow velocity; S402, setting the number of training cycles N; S403, the unmanned boat intelligent controller based on the improved unmanned boat intelligent control method interacts with the environment and stores the interaction information in the experience playback pool in chronological order; S404, determine whether the experience replay pool is full; if not, continue to execute step S402; if so, take out the data to iterate the neural network parameters;

[0048] S405, determine whether the set number of training cycles N is reached; if not, continue to execute step S403; if yes, observe the control effect of the unmanned boat intelligent controller;

[0049] Step S5, determine whether the trained unmanned boat intelligent controller meets the multi-task target requirements; if not, continue to execute step S4; if so, end and save the model and neural network parameters for the improved unmanned boat intelligent controller to call.

[0050] like Figure 2 As shown, it is a schematic diagram of the principle of the present invention. In a specific implementation, the unmanned boat intelligent controller selects a low entropy action to control the left motor of the unmanned boat. , right motor control rate Output, control the unmanned boat in the actual sea conditions, and obtain the status information of the unmanned boat , and finally the unmanned boat status information Input to the unmanned boat intelligent controller; after the above-mentioned specific implementation steps S1-S5, continue to select low entropy actions, and the new unmanned boat state information Feedback is given to the unmanned boat intelligent controller to continuously optimize the intelligent control effect.

[0051] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.

[0052] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The solutions in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and interpreted scripting language JavaScript, etc.

[0053] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0054] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0055] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0056] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0057] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An intelligent control method for an unmanned boat based on an improved low entropy strategy and a network structure, characterized in that: The following steps are involved: Set the action space and state space of the unmanned boat, set the reward function according to the multi-task target requirements of the unmanned boat, and use the unmanned boat controller that combines the low entropy action selection mechanism and the improved LSTM network for intelligent control until the unmanned boat reaches the target point, completing the intelligent control process of the unmanned boat; The unmanned boat controller includes an Actor network, a Critic network and an improved LSTM network. The process of intelligent control of the unmanned boat controller includes: A batch of data is collected from the experience replay pool, input into the improved LSTM network for processing, and then the control action is output through the Actor network, and the low entropy action selection mechanism is used to select the low entropy action to obtain the control output , to control the operation of the unmanned boat, wherein the experience replay pool is formed by interacting with the environment and combining the action space, state space and reward function; Collect a batch of data from the experience replay pool, input it into the Critic network for value estimation, and guide the Actor network to output control actions; Repeat the above control process until the unmanned boat reaches the target point.

2. According to claim 1, the unmanned boat intelligent control method based on improved low entropy strategy and network structure is characterized in that: The action space of the unmanned boat includes the control rate of the left motor of the unmanned boat , right motor control rate , distance to target point and the distance to obstacles , the state space includes the unmanned boat linear speed , angular velocity , distance to target point , Distance from obstacles , left motor control rate and right motor control rate .

3. According to claim 1, the unmanned boat intelligent control method based on improved low entropy strategy and network structure is characterized in that: The expression of the reward function is: ， in: R 1 is the reward function for the unmanned boat to reach the target point, and the expression is: , R 2 is the reward function of the distance of the unmanned boat away from obstacles, and the expression is: , R 3 is the reward function for the control of the linear velocity and angular velocity of the unmanned boat, and the expression is: ， R 4 is the reward function of the unmanned boat navigation time control, and its expression is: , In the formula, R is the total reward function, is the weight of the reward for the unmanned boat to reach the target point in the total reward, It is the reward for the unmanned boat to reach the target point. is the weight of the reward for the distance of the unmanned boat away from obstacles in the total reward, It is the distance when the unmanned boat is within the set distance of the obstacle. It is the reward for the collision between the unmanned boat and the obstacle. is the weight of the reward for the stability of the unmanned boat's linear speed in the total reward, is the current linear speed of the unmanned boat, is the linear speed of the unmanned boat at the previous moment, is the weight of the reward of the current linear speed of the unmanned boat in the total reward, is the weight of the reward for the angular velocity stability of the unmanned boat in the total reward, is the current angular velocity of the unmanned boat, is the angular velocity of the unmanned boat at the previous moment, is the weight of the reward for the distance between the unmanned boat and the target point in the total reward; It is the reward for the distance between the unmanned boat and the target point.

4. The unmanned boat intelligent control method based on improved low entropy strategy and network structure according to claim 1 is characterized in that: The Actor network is a dual-strategy network, including an Actor_1 subnetwork and an Actor_2 subnetwork. The Actor_1 subnetwork and the Actor_2 subnetwork interact with the environment in parallel and output control actions.

5. The unmanned boat intelligent control method based on improved low entropy strategy and network structure according to claim 1 is characterized in that: The Critic network includes a Critic_1 subnetwork and a Critc_2 subnetwork, and the Critic_1 subnetwork and the Critc_2 subnetwork perform the value estimation in parallel, wherein the value estimation result includes a state value function and the state-action value function , the expressions are: ， ， In the formula, It is in strategy Next action Seek hope, is the temperature coefficient, For strategy In Status Next select action The logarithmic probability of It is the transfer function Next moment status Seek hope, is in state Next action Instant rewards, is the discount factor, It is the estimated state value of the state at the next moment.

6. The unmanned boat intelligent control method based on improved low entropy strategy and network structure according to claim 1 is characterized in that: The operation expression for selecting a low entropy action using the low entropy action selection mechanism is: ， ， In the formula, For the status Next Strategy The entropy of To select in state Next Strategy The minimum entropy value in It is in strategy Next action expectations, It’s a strategy In Status Next select action The logarithmic probability of It’s a strategy In Status Next select action The weight of is the action with the lowest entropy value, is to choose the strategy with the lowest entropy value Selected Action .

7. The unmanned boat intelligent control method based on improved low entropy strategy and network structure according to claim 1 is characterized in that: The improved LSTM network improves the updating method of the cell state in the input gate calculation process. The updated expression of the improved cell state is: ， ， In the formula, The cell state of the current time step for the improved LSTM network structure, is the input of the forget gate, is the cell state at the previous moment, is the output of the input gate, is the hyperbolic tangent function, is the weight matrix of the hidden state, which is updated by the back-propagation algorithm. is the hidden state at the previous moment, is the input of the current time step, is the bias term, which is updated by the back-propagation algorithm. is the memory matrix of the previous moment, for Function operation, is the memory matrix of the previous moment.

8. The unmanned boat intelligent control method based on improved low entropy strategy and network structure according to claim 1 is characterized in that: The guiding of the Actor network output control action refers to updating the network parameters based on the value evaluation result to select the optimal control action. The specific steps include: For each batch of data, after being processed by the improved LSTM network and Actor network, the network parameters of the unmanned boat controller are updated in combination with the value evaluation results, wherein the expression of the loss function of the network parameter update is: ， In the formula, is the neural network parameter The loss function is are neural network parameters; is from the experience replay pool, i.e. The sample data obtained by sampling in is the state of the current time step, is the action at the current time step, Is to perform an action Instant rewards, It is the state of the next moment; is the replay pool from the experience, i.e. sampling And in the strategy Seek expectations, It is all the data stored in the experience replay pool. It is the action of the next moment. is in state Lower Parameters Strategy , is the current time step value, is the discount factor, It's the next moment Smaller value The expected return of the function, is the temperature coefficient, For strategy In Status Next select action The logarithmic probability of .

9. The unmanned boat intelligent control method based on improved low entropy strategy and network structure according to claim 1 is characterized in that: The objective function updated by the unmanned boat controller is: ， In the formula, is the objective function of policy optimization, Is to obey the strategy The state-action group under the joint distribution of state and action The target value of is expected, It is in strategy The joint distribution of states and actions under ; is the discount factor, is the state of the current time step, is the action at the current time step, Is to perform an action Instant rewards, is the temperature coefficient, It’s a strategy In Status The entropy under .

10. The unmanned boat intelligent control method based on improved low entropy strategy and network structure according to claim 1 is characterized in that: The training process of the unmanned boat controller includes: 1) Add wind and flow interference to the environment, where the expression of wind interference in the environment is: ， The expression of flow interference in the environment is: ， In the formula, Because of wind interference, For flow interference, is the wind coefficient, is the heading angle of the unmanned boat, is the wind speed, is the water flow coefficient, is the angle between the water flow direction and the heading of the unmanned boat, is the water flow velocity; 2) Set the training cycle N; 3) Interact with the environment and put the interaction information into the experience replay pool in chronological order; 4) Determine whether the experience replay pool is full. If so, extract a batch of data from the experience replay pool to iteratively update the network parameters. If not, execute step 2); 5) Determine whether the set training cycle N has been reached, If yes, then further determine whether the multi-objective requirements are met. If yes, then output the control result to obtain the control effect. If no, then execute step 3); If not, execute step 3) until the training cycle N is reached and the multi-objective requirements are met, and the training process is completed.

Images