A control method for unmanned sailboat based on deep reinforcement learning algorithm

Abstract

The present invention discloses a control method of an unmanned sailboat based on a deep reinforcement learning algorithm. The method includes collecting the position coordinate information of the unmanned sailboat, calculating the speed of the unmanned sailboat coordinate system based on the position coordinate information of the unmanned sailboat, collecting the linear acceleration and angular velocity of the three axes of the unmanned sailboat, and calculating the roll angle of the unmanned sailboat at the same time, inputting the current state quantity and action quantity into a trained neural network model to predict the rudder angle and sail angle at the next moment; and controlling the rudder angle and sail angle predicted by the sail steering engine and the rudder steering engine of the unmanned sailboat. The present invention only needs to collect the current sail angle, rudder angle and state data to perform a reinforcement learning method for subsequent sail control and rudder control decision-making, does not need to monitor the wind speed and wind direction in real time, can effectively resist the external interference of random wind field fluctuations, saves the assembly of anemometers, reduces the impact of environmental changes on the control algorithm that relies on wind speed and wind direction, and improves the overall integration of the sailboat.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned sailboat control, and in particular to an unmanned sailboat control method based on a deep reinforcement learning algorithm. Background Art

[0002] The development of robotics and artificial intelligence technologies has gradually led ocean exploration to an unmanned and intelligent level. Unmanned sailboats are a new type of long-endurance marine robot driven by marine renewable energy, breaking through the bottleneck of limited traditional energy supply. Against the backdrop of the growing demand for observation of large-scale environmental elements at the sea-air interface, unmanned sailboats can better solve the problem of scarcity of mesoscale and sub-mesoscale observation data.

[0003] At present, the method of using artificial intelligence technology (mainly deep reinforcement learning technology) to realize the control decision of unmanned sailboats basically relies on the velocity prediction program (VPP). The original design and application of VPP is to help crew members better drive sailboats. By establishing a mechanical model of the hull and the sail (including a fluid mechanics model and an aerodynamic model), a set of speed pole diagrams that ensure stable sailing are generated. When the true wind direction and true wind speed are obtained, the crew can find the corresponding maximum stable sailing speed through the pole diagram and make a decision. In the application of deep reinforcement learning, VPP provides the agent with possible action choices in the next time step. Deep reinforcement learning optimizes the action selection of the entire motion process through the Markov decision process, thereby obtaining the optimal sailing decision from the starting point to the end point. This method provides great help for the sailing and racing of unmanned sailboats. However, VPP has accuracy requirements for the input of the true wind direction and true wind speed, and the actual measurement often has errors. Therefore, it is also necessary to manually design a set of programs for correcting or predicting wind speed and direction, which increases the complexity of the entire control system to a certain extent.

[0004] In addition, some traditional control methods also need to detect wind speed and direction as necessary parameters. For example, people use fuzzy control to improve the autonomous navigation adaptability of unmanned sailboats in complex and unknown environments. Unmanned sailboats can select the output sail angle and rudder angle through a manually set wind field domain value. Compared with the numerical calculation dynamics part, the wind direction and wind speed have an accurate correspondence with the sail angle and rudder angle. Fuzzy control helps unmanned sailboats to travel like "manned ships". However, the division of domains in fuzzy control often requires a lot of "expert experience", and it is cumbersome to formulate a set of fuzzy control sets that can meet all situations. Summary of the invention

[0005] The purpose of the present invention is to provide an unmanned sailboat control method based on a deep reinforcement learning algorithm in view of the shortcomings of the prior art.

[0006] To achieve the above object, the present invention provides an unmanned sailboat control method based on a deep reinforcement learning algorithm, comprising:

[0007] Collect the location coordinate information of the unmanned sailboat ,in, is the horizontal coordinate of the unmanned sailboat, is the ordinate of the unmanned sailboat, is the speed of the unmanned sailboat, is the yaw angle of the unmanned sailboat;

[0008] Based on the location coordinate information of the unmanned sailboat Calculate the speed of the unmanned sailboat coordinate system as:

[0009]

[0010] in, is the moving speed of the unmanned sailboat in the forward and backward directions, is the lateral moving speed of the unmanned sailboat;

[0011] Collect the three-axis linear acceleration of the unmanned sailboat and the three-axis angular velocity ,in, and They represent the linear acceleration and angular velocity of the unmanned sailboat in the X-axis direction, and They represent the linear acceleration and angular velocity of the unmanned sailboat in the Y-axis direction, and Respectively represent the linear acceleration and angular velocity of the unmanned sailboat in the Z-axis direction; and let , , and calculate the rolling angle of the unmanned sailboat ;

[0012] The , , , , , , , as the current state quantity, and input the current rudder angle and sail angle as the current action quantity into a trained neural network model, wherein the neural network model predicts the rudder angle and sail angle at the next moment according to the current state quantity and action quantity;

[0013] The sail steering gear and rudder steering gear controlling the unmanned sailboat work according to the rudder angle and sail angle predicted by the neural network model.

[0014] Furthermore, the neural network model includes a first Critic network, a second Critic network and an Actor network, wherein the first Critic network and the second Critic network are used to output a Q value according to a current state quantity and an action quantity respectively, and the Q value is used to influence the decision of the Actor network, and the Actor network is used to output a mean and a standard deviation of a rudder angle and a sail angle according to the current state quantity and the Q value.

[0015] Furthermore, the first Critic network includes a first input module and a second input module, the first input module and the second input module are respectively used to receive the current state quantity and action quantity of the input, the first input module and the second input module are respectively connected to the first fully connected module and the second fully connected module, the first fully connected module and the second fully connected module both use 256 neurons to be fully connected with the state quantity and the action quantity respectively to generate 256 features respectively, the first fully connected module and the second fully connected module are both connected to the splicing module, the splicing module is used to connect the features of the state quantity and the action quantity in series to form 512 feature combinations, and it is connected to the first activation module , a third fully-connected module, a second activation module and a fourth fully-connected module are connected in series in sequence, the first activation module is used to activate the features after series connection to generate 512 non-negative features and pass them into the third fully-connected module, the third fully-connected module is used to use 256 neurons to fully connect with 512 features to obtain 256 high-level features to represent the high-level feature relationship between state quantity features and action quantity features, the second activation module is used to activate the high-level features output by the third fully-connected module to generate 256 non-negative features based on the 256 high-level features, and the fourth fully-connected module is used to calculate the Q value based on the 256 non-negative features generated by the second activation module.

[0016] Furthermore, the second Critic network has the same structure as the first Critic network, but its first fully connected module and the second fully connected module both use 128 neurons to generate corresponding features.

[0017] Further, the Actor network includes a third input module, the third input module is used to receive the current state quantity input, and the third input module is connected to the fifth fully connected module, the fifth fully connected module uses 256 neurons to fully connect with the state quantity to generate 256 features, and the third activation module is connected to the third activation module, the third activation module is used to activate the features output by the fifth fully connected module to generate 256 non-negative features and pass them into the sixth fully connected module and the seventh fully connected module respectively;

[0018] The sixth fully connected module uses 256 neurons to fully connect with the non-negative features output by the third activation module to generate 256 primary features of the mean of the amount of motion. The sixth fully connected module is connected in series with the fourth activation module, the eighth fully connected module, the fifth activation module and the ninth fully connected module in sequence. The fourth activation module is used to activate the 256 primary features of the mean of the amount of motion. The eighth fully connected module uses 128 neurons to fully connect with the primary features activated by the fourth activation module to generate 128 high-level features of the mean of the amount of motion. The fifth activation module is used to activate the 128 high-level features of the mean of the amount of motion. The ninth fully connected module uses 2 neurons to fully connect with the 128 high-level features of the mean of the amount of motion after being activated by the fifth activation module to obtain the mean of the rudder angle and the sail angle, and outputs them as the predicted rudder angle and sail angle at the next moment.

[0019] The seventh fully connected module uses 256 neurons to fully connect with 256 non-negative features output by the third activation module to generate 256 primary features of the standard deviation of the amount of motion. The seventh fully connected module is connected in series with the sixth activation module, the tenth fully connected module, the seventh activation module, the eleventh fully connected module and the eighth activation module in sequence. The sixth activation module is used to activate the 256 primary features of the standard deviation of the amount of motion. The tenth fully connected module uses 128 neurons to fully connect with the 256 primary features of the standard deviation of the amount of motion to form 128 high-level features of the standard deviation of the amount of motion. The seventh activation module is used to activate the 128 high-level features of the standard deviation of the amount of motion. The eleventh fully connected module uses 2 neurons to fully connect with the 128 high-level features of the activated standard deviation of the amount of motion to obtain the standard deviation of the rudder angle and the sail angle. The eighth activation module is used to activate the standard deviation of the rudder angle and the sail angle to obtain the non-negative standard deviation of the rudder angle and the sail angle.

[0020] Furthermore, it also includes: realizing dynamic estimation and correction of system state based on state error Kalman filter algorithm.

[0021] Furthermore, the neural network model is trained in the following manner:

[0022] The relative wind speed and relative wind direction data are collected by the aircraft in the target sea area, and the relative wind speed and relative wind direction are converted into absolute wind speed and absolute wind direction;

[0023] Design a simulated wind speed function and a simulated wind direction function according to the absolute wind speed and the absolute wind direction;

[0024] A reward function is designed to optimize the learning effect. The reward function is designed as:

[0025]

[0026]

[0027] in, is the calculated reward value, is the coordinate of the destination of the unmanned sailboat, is the predicted value of the distance to the end point calculated based on the current state, is the huge reward for reaching the end point, t is the real-time simulation time under a single simulation, It includes penalties for large changes in sail angle and rewards for keeping the sail angle constant. It includes penalties for large changes in the rudder angle and rewards for keeping the rudder angle constant;

[0028] The neural network model is trained based on the simulated wind speed function, the simulated wind direction function and the reward function.

[0029] Furthermore, the simulated wind speed function and the simulated wind direction function are respectively:

[0030]

[0031]

[0032] in, is the simulated wind speed at one sampling time, is the given median wind speed, is the wind speed at the last sampling time, is the standard normal distribution function, is the simulated wind direction at one sampling time, is the given median wind direction, is the wind speed at the previous sampling time.

[0033] Beneficial effects: 1. The present invention only needs to collect the current sail angle, rudder angle and status data to carry out the reinforcement learning method for subsequent sail control and rudder control decision-making, and does not need to monitor the wind speed and wind direction in real time;

[0034] 2. The present invention proposes a piecewise progressive reward function optimization idea, which improves the balance between navigation efficiency and energy required for navigation in the training results while ensuring the training effect;

[0035] 3. The present invention can effectively resist external interference caused by random wind field fluctuations;

[0036] 4. The present invention can save the installation of anemometers for unmanned sailboats, reduce the impact of environmental changes on the control algorithm that relies on wind speed and direction, and improve the overall integration of the sailboat. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1is a schematic diagram of the structure of a Critic network according to an embodiment of the present invention;

[0038] Figure 2 is a schematic diagram of the structure of the Actor network in an embodiment of the present invention;

[0039] Figure 3 It is a schematic diagram of the measured absolute wind direction;

[0040] Figure 4 It is a schematic diagram simulating the absolute wind direction;

[0041] Figure 5 It is a schematic diagram of the measured absolute wind speed;

[0042] Figure 6 It is a schematic diagram of simulating absolute wind speed. DETAILED DESCRIPTION

[0043] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. The embodiments are implemented based on the technical solution of the present invention. It should be understood that these embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention.

[0044] like Figures 1 to 6 As shown, an embodiment of the present invention provides an unmanned sailboat control method based on a deep reinforcement learning algorithm, comprising:

[0045] Collect the location coordinate information of the unmanned sailboat ,in, is the horizontal coordinate of the unmanned sailboat, is the ordinate of the unmanned sailboat, is the speed of the unmanned sailboat, is the yaw angle of the unmanned sailboat. Specifically, the position coordinate information of the unmanned sailboat It can be collected by a positioning device installed on an unmanned sailboat.

[0046] Based on the location coordinate information of the unmanned sailboat Calculate the speed of the unmanned sailboat coordinate system as:

[0047]

[0048] in, is the moving speed of the unmanned sailboat in the forward and backward directions, is the lateral moving speed of the unmanned sailboat,

[0049] Collect the three-axis linear acceleration of the unmanned sailboat and the three-axis angular velocity ,in, and They represent the linear acceleration and angular velocity of the unmanned sailboat in the X-axis direction, and They represent the linear acceleration and angular velocity of the unmanned sailboat in the Y-axis direction, and They represent the linear acceleration and angular velocity of the unmanned sailboat in the Z-axis direction. , , and calculate the rolling angle of the unmanned sailboat It should be noted that the linear acceleration of the three axes of the unmanned sailboat and the three-axis angular velocity The data can be collected by the inertial sensor installed on the unmanned sailboat. The coordinate axis of the inertial sensor needs to be adjusted to be consistent with the coordinate axis of the sailboat, so that , .

[0050] Will , , , , , , , These eight parameters are used as the current state quantities, and the current rudder angle and sail angle are used as the current action quantities and input into the trained neural network model. The neural network model predicts the rudder angle and sail angle at the next moment based on the current state quantities and action quantities.

[0051] The above is the acquisition of the initial state of the sailboat. Before inputting into the trained neural network model, it is also preferred to use the state error Kalman filter algorithm (ESKF) combined with the data collected by the positioning device and the inertial sensor to realize the dynamic estimation and correction of the system state. The state transfer model and the state update equation correspond to two main steps in the ESKF: prediction and update. The state transfer model and the state update equation are explained below:

[0052] State transition model (prediction step):

[0053] Simple dynamic equations: Assume that at a small time step Inside

[0054] Latitude and longitude at the time for:

[0055]

[0056]

[0057] Roll angle at time and yaw angle They are:

[0058]

[0059]

[0060] , , , In simple cases, the state transition model is expressed as follows:

[0061]

[0062] in, is the state vector,

[0063]

[0064] is the state transfer matrix, is the control input matrix, is the measurement matrix of the inertial sensor, is the process noise.

[0065] State update equation (update step):

[0066] The state update equation incorporates new observations to correct the predicted state estimate:

[0067] The data collected by the positioning device is used as the observation :

[0068]

[0069]

[0070] in, is the observation matrix, is the observation noise.

[0071]

[0072] Calculate Kalman gain for:

[0073]

[0074] in, is the error covariance matrix, is the observation noise covariance, T is the transpose of the matrix;

[0075] Update the status to:

[0076]

[0077]

[0078]

[0079] in, For the updated status, is the Jacobian matrix of the state transfer function, which represents the sensitivity of the state model to the current state. is the process noise covariance matrix, which represents the uncertainty of process noise. is the identity matrix, the base Represents the state at time k predicted by the state transition model at time k-1;

[0080]

[0081] After each state correction, the updated covariance matrix represents the uncertainty of the revised state estimate, and the covariances of all state variables are adjusted accordingly.

[0082] The time steps in the state transfer model are expressed as k+1 and k because the state transfer model is a prediction model and an estimate of the future. The state update equation time is expressed as k and k-1 because the state update equation is a correction to the current state estimate.

[0083] The sail steering gear and rudder steering gear controlling the unmanned sailboat work according to the rudder angle and sail angle predicted by the neural network model.

[0084] The neural network model of the embodiment of the present invention includes a first critic network, a second critic network and an actor network. The first critic network and the second critic network are used to output a Q value according to the current state quantity and action quantity respectively. The Q value is used to influence the decision of the actor network, specifically referring to the expected reward of taking the input action under the input state. The actor network is used to output the mean and standard deviation of the rudder angle and the sail angle according to the current state quantity and Q value.

[0085] See Figure 1, the first Critic network includes a first input module 1 and a second input module 2, the first input module 1 and the second input module 2 are respectively used to receive the current state quantity and action quantity of the input. The first input module 1 and the second input module 2 are respectively connected to the first fully connected module 3 and the second fully connected module 4, and the first fully connected module 3 and the second fully connected module 4 use 256 neurons to fully connect with the state quantity and the action quantity respectively to generate 256 features respectively. The first fully connected module 3 and the second fully connected module 4 are both connected to the splicing module 5, and the splicing module 5 is used to connect the features of the state quantity and the action quantity in series to form 512 feature combinations. The splicing module 5 is connected in series with the first activation module 6, the third fully connected module 7, the second activation module 8 and the fourth fully connected module 9 in sequence. Among them, the first activation module 6 is used to activate the features after the series connection to generate 512 non-negative features and pass them into the third fully connected module 7, and the third fully connected module 7 is used to use 256 neurons to fully connect with 512 features to obtain 256 high-level features to represent the high-level feature relationship between the state quantity features and the action quantity features. The second activation module 8 is used to activate the high-level features output by the third fully connected module 7 to generate 256 non-negative features based on the 256 high-level features. The fourth fully connected module 9 is used to calculate the Q value based on the 256 non-negative features generated by the second activation module 8.

[0086] The second Critic network of the embodiment of the present invention has the same structure as the first Critic network, but its first fully connected module and second fully connected module both use 128 neurons to generate corresponding features. In the two Critic networks, different numbers of neurons are used for the initial feature value generation, which will produce different evaluation values ​​for the state quantity and action quantity, and can better improve the overall training effect.

[0087] See also Figure 2The Actor network of the embodiment of the present invention includes a third input module 10, which is used to receive the current state quantity input, and is connected to the fifth fully connected module 11, which uses 256 neurons to fully connect with the state quantity to generate 256 features, and the fifth fully connected module 11 is connected to the third activation module 12, and the third activation module 12 is used to activate the features output by the fifth fully connected module 11 to generate 256 non-negative features and pass them to the sixth fully connected module 13 and the seventh fully connected module 14 respectively. The sixth fully connected module 13 uses 256 neurons to fully connect with the non-negative features output by the third activation module 12 to generate 256 primary features of the action quantity mean. The sixth fully connected module 13 is connected in series with the fourth activation module 15, the eighth fully connected module 16, the fifth activation module 17 and the ninth fully connected module 18 in sequence. Among them, the fourth activation module 15 is used to activate the 256 primary features of the mean of the amount of motion, and the eighth fully connected module 16 uses 128 neurons to fully connect with the primary features activated by the fourth activation module 15 to generate 128 high-level features of the mean of the amount of motion. The fifth activation module 17 is used to activate the 128 high-level features of the mean of the amount of motion, and the ninth fully connected module 18 uses 2 neurons to fully connect with the 128 high-level features of the mean of the amount of motion after being activated by the fifth activation module 17 to obtain the mean of the rudder angle and the sail angle, and output them as the predicted rudder angle and sail angle at the next moment.

[0088] The seventh fully connected module 14 uses 256 neurons to fully connect with the 256 non-negative features output by the third activation module 12 to generate 256 primary features of the standard deviation of the amount of motion. The seventh fully connected module 14 is connected in series with the sixth activation module 19, the tenth fully connected module 20, the seventh activation module 21, the eleventh fully connected module 22 and the eighth activation module 23. Among them, the sixth activation module 19 is used to activate the 256 primary features of the standard deviation of the amount of motion, and the tenth fully connected module 20 uses 128 neurons to fully connect with the 256 primary features of the standard deviation of the amount of motion to form 128 high-level features of the standard deviation of the amount of motion. The seventh activation module 21 is used to activate the 128 high-level features of the standard deviation of the amount of motion. The eleventh fully connected module 22 uses 2 neurons to fully connect with the 128 high-level features of the activated standard deviation of the amount of motion to obtain the standard deviation of the rudder angle and the sail angle. The eighth activation module 23 is used to activate the standard deviation of the rudder angle and the sail angle to obtain the non-negative standard deviation of the rudder angle and the sail angle. The standard deviation of the output is used to measure the uncertainty of the output action. It should be noted that the influence of the Q value on the actor network is a prior art, which is specifically reflected in the sixth fully connected module 13 and the seventh fully connected module 14 and the modules connected to the lower side thereof, thereby affecting the calculation process of the mean and standard deviation of the rudder angle and the sail angle, which will not be repeated here.

[0089] The Soft Actor-Critic used in the present invention is a model-free, online, non-strategy reinforcement learning method. The introduction of the concept of entropy can help the intelligent agent better explore the environment and avoid falling into the local optimal solution. The present invention does not modify the underlying algorithm logic, so the calculation process will not be elaborated in detail.

[0090] The above neural network model is preferably trained in the following way:

[0091] The relative wind speed and relative wind direction data are collected by the aircraft in the target sea area, and the relative wind speed and relative wind direction are converted into absolute wind speed and absolute wind direction. The absolute wind direction and absolute wind speed obtained are as follows: Figure 3 and Figure 4 Then, the simulated wind speed function and the simulated wind direction function are designed according to the absolute wind speed and the absolute wind direction. The simulated wind speed function and the simulated wind direction function are respectively:

[0092]

[0093]

[0094] in, is the simulated wind speed at one sampling time, is the given median wind speed, is the wind speed at the last sampling time, is the standard normal distribution function, is the simulated wind direction at one sampling time, is the given median wind direction, is the wind speed at the last sampling time. Finally, we get Figure 5 and Figure 6 The simulated wind direction and wind speed images shown are similar to the real wind direction and wind speed. Without considering the mean value (in the actual navigation process, the mean value of a certain period of time is similar, and there are few cases of sudden drastic changes), it can be seen from the fluctuation range (wind direction changes by about 30°, wind speed fluctuates by 4m / s) and the randomness of the fluctuation that the simulated wind field has a certain effect that fits the actual situation. In addition, in the calculation part of the state of the unmanned sailboat, the present invention adopts a mature four-degree-of-freedom simulation model, which can obtain the state of the unmanned sailboat under continuous motion, while maintaining the reliability of the simulation data, reducing the degrees of freedom with a smaller impact, and improving the efficiency of the calculation.

[0095] Due to the complexity and variability of the marine environment, it is impossible to accurately replicate the real environment. Therefore, a model-free reinforcement learning framework was selected in the previous method. This framework allows the unmanned sailboat to learn directly from the feedback of the environment and has a higher ability to adapt to complex environments. In this type of framework, we use a progressive reward function design to optimize the learning effect. The reward function is designed as:

[0096]

[0097]

[0098] in, is the calculated reward value, is the coordinate of the destination of the unmanned sailboat, The predicted value of the distance to the end point is calculated based on the current state. A sampling time of 1 second is selected in a single simulation. The predicted value is used to predict the position of the next second to help guide the unmanned sailboat to the target point. Choosing a negative value can better avoid the unmanned sailboat from obtaining positive rewards aimlessly for a long time, which may lead to training failure. The huge reward for reaching the end point is calculated only once after the single simulation is terminated after reaching the target point. t is the real-time simulation time under a single simulation. The use of negative values ​​here is similar to the fixed value of negative numbers set in the traditional reward function, which is used to assist in improving the speed of the unmanned sailboat. However, compared with the traditional fixed value, the simulation time as a constant can avoid the exploration trap in the early stage of a single simulation, and at the same time increase the convergence of the late stage of a single simulation. It includes penalties for excessive changes in the sail angle and rewards for keeping the sail angle unchanged. In actual angle control, the sail angle change should be less than 13° / s. Excessive angle changes during training are considered unreasonable. Keeping the sail angle unchanged can reduce the energy consumption of the unmanned sailboat and achieve a balance between fast sailing and energy saving. It includes penalties for excessive changes in the rudder angle and rewards for keeping the rudder angle unchanged. In actual angle control, the rudder angle change should be less than 30° / s. Excessive angle changes during training are considered unreasonable. Keeping the rudder angle unchanged can reduce the energy consumption of the unmanned sailboat and achieve a balance between fast sailing and energy saving. The constraints of the sail and rudder angles can make the training results more in line with the actual situation, and in some states, stable sail and rudder angles can make sailing smoother (because the sailboat itself is affected by both the sail and the rudder. Once one of the quantities changes frequently and drastically, the other quantity also needs to change a lot to achieve balance while maintaining stability. In actual sailing / artificial sailing experience, choosing to change within a small and gentle range will bring greater profits). In addition, the constraints of the sail and rudder angles should not be added to the reward function too early. In the initial exploration process, over-emphasizing the angle will reduce the training effect and even lead to non-convergence.

[0099] The neural network model is trained based on the simulated wind speed function, simulated wind direction function and reward function. Specifically, since the present invention adopts a model-free training framework, after the simulation environment is designed in advance, only the target point data needs to be provided to let the unmanned sailboat train how to reach it quickly. The neural network was trained in a relatively downwind area, and relatively good results were obtained after about 20,000 simulation cycle iterations (including basic reward iterations and advanced reward iterations), and the predetermined tasks could be completed under different initial headings.

[0100] The above is only a preferred embodiment of the present invention. It should be noted that for ordinary technicians in this technical field, other parts not specifically described belong to the prior art or common knowledge. Several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered as the protection scope of the present invention.

Claims

1. A control method for an unmanned sailboat based on deep reinforcement learning algorithm. It is characterized in that include: Collect the location coordinate information of the unmanned sailboat ,in, is the horizontal coordinate of the unmanned sailboat, is the ordinate of the unmanned sailboat, is the speed of the unmanned sailboat, is the yaw angle of the unmanned sailboat; Based on the location coordinate information of the unmanned sailboat Calculate the speed of the unmanned sailboat coordinate system as: ； in, is the moving speed of the unmanned sailboat in the forward and backward directions, is the lateral moving speed of the unmanned sailboat; Collect the three-axis linear acceleration of the unmanned sailboat and the three-axis angular velocity ,in, and They represent the linear acceleration and angular velocity of the unmanned sailboat in the X-axis direction, and They represent the linear acceleration and angular velocity of the unmanned sailboat in the Y-axis direction, and Respectively represent the linear acceleration and angular velocity of the unmanned sailboat in the Z-axis direction; and let , , and calculate the rolling angle of the unmanned sailboat ; The , , , , , , , as the current state quantity, and input the current rudder angle and sail angle as the current action quantity into a trained neural network model, wherein the neural network model predicts the rudder angle and sail angle at the next moment according to the current state quantity and action quantity; The sail steering gear and rudder steering gear of the unmanned sailboat are controlled to work according to the rudder angle and sail angle predicted by the neural network model; The neural network model is trained in the following way: The relative wind speed and relative wind direction data are collected by the aircraft in the target sea area, and the relative wind speed and relative wind direction are converted into absolute wind speed and absolute wind direction; Design a simulated wind speed function and a simulated wind direction function according to the absolute wind speed and the absolute wind direction; A reward function is designed to optimize the learning effect. The reward function is designed as: ； ； in, is the calculated reward value, is the coordinate of the destination of the unmanned sailboat, is the predicted value of the distance to the end point calculated based on the current state, is the huge reward for reaching the end point, t is the real-time simulation time under a single simulation, It includes penalties for large changes in sail angle and rewards for keeping the sail angle constant. It includes penalties for large changes in the rudder angle and rewards for keeping the rudder angle constant; The neural network model is trained based on the simulated wind speed function, the simulated wind direction function and the reward function.

2. According to claim 1, a method for controlling an unmanned sailboat based on a deep reinforcement learning algorithm, It is characterized in that The neural network model includes a first critic network, a second critic network and an actor network. The first critic network and the second critic network are used to output a Q value according to the current state quantity and action quantity respectively. The Q value is used to influence the decision of the actor network. The actor network is used to output the mean and standard deviation of the rudder angle and the sail angle according to the current state quantity and the Q value.

3. According to claim 2, a method for controlling an unmanned sailboat based on a deep reinforcement learning algorithm, It is characterized in that The first Critic network includes a first input module and a second input module, the first input module and the second input module are respectively used to receive the current state quantity and action quantity of the input, the first input module and the second input module are respectively connected to the first fully connected module and the second fully connected module, the first fully connected module and the second fully connected module both use 256 neurons to be fully connected with the state quantity and the action quantity respectively to generate 256 features respectively, the first fully connected module and the second fully connected module are both connected to the splicing module, the splicing module is used to connect the features of the state quantity and the action quantity in series to form 512 feature combinations, and it is connected to the first activation module, the first activation module, the second ... The three fully-connected modules, the second activation module and the fourth fully-connected module are connected in series in sequence. The first activation module is used to activate the features after series connection to generate 512 non-negative features and pass them into the third fully-connected module. The third fully-connected module is used to use 256 neurons to fully connect with the 512 features to obtain 256 high-level features to represent the high-level feature relationship between the state quantity features and the action quantity features. The second activation module is used to activate the high-level features output by the third fully-connected module to generate 256 non-negative features based on the 256 high-level features. The fourth fully-connected module is used to calculate the Q value based on the 256 non-negative features generated by the second activation module.

4. According to claim 3, a method for controlling an unmanned sailboat based on a deep reinforcement learning algorithm, It is characterized in that The second Critic network has the same structure as the first Critic network, but its first fully connected module and second fully connected module both use 128 neurons to generate corresponding features.

5. According to claim 3, a method for controlling an unmanned sailboat based on a deep reinforcement learning algorithm, It is characterized in that The Actor network includes a third input module, the third input module is used to receive the current state quantity input, and the third input module is connected to the fifth fully connected module, the fifth fully connected module uses 256 neurons to fully connect with the state quantity to generate 256 features, and the third activation module is connected to the third activation module, the third activation module is used to activate the features output by the fifth fully connected module to generate 256 non-negative features and respectively pass them into the sixth fully connected module and the seventh fully connected module; The sixth fully connected module uses 256 neurons to fully connect with the non-negative features output by the third activation module to generate 256 primary features of the mean of the amount of motion. The sixth fully connected module is connected in series with the fourth activation module, the eighth fully connected module, the fifth activation module and the ninth fully connected module in sequence. The fourth activation module is used to activate the 256 primary features of the mean of the amount of motion. The eighth fully connected module uses 128 neurons to fully connect with the primary features activated by the fourth activation module to generate 128 high-level features of the mean of the amount of motion. The fifth activation module is used to activate the 128 high-level features of the mean of the amount of motion. The ninth fully connected module uses 2 neurons to fully connect with the 128 high-level features of the mean of the amount of motion after being activated by the fifth activation module to obtain the mean of the rudder angle and the sail angle, and outputs them as the predicted rudder angle and sail angle at the next moment. The seventh fully connected module uses 256 neurons to fully connect with 256 non-negative features output by the third activation module to generate 256 primary features of the standard deviation of the amount of motion. The seventh fully connected module is connected in series with the sixth activation module, the tenth fully connected module, the seventh activation module, the eleventh fully connected module and the eighth activation module in sequence. The sixth activation module is used to activate the 256 primary features of the standard deviation of the amount of motion. The tenth fully connected module uses 128 neurons to fully connect with the 256 primary features of the standard deviation of the amount of motion to form 128 high-level features of the standard deviation of the amount of motion. The seventh activation module is used to activate the 128 high-level features of the standard deviation of the amount of motion. The eleventh fully connected module uses 2 neurons to fully connect with the 128 high-level features of the activated standard deviation of the amount of motion to obtain the standard deviation of the rudder angle and the sail angle. The eighth activation module is used to activate the standard deviation of the rudder angle and the sail angle to obtain the non-negative standard deviation of the rudder angle and the sail angle.

6. According to claim 1, a method for controlling an unmanned sailboat based on a deep reinforcement learning algorithm, It is characterized in that Also includes: Dynamic estimation and correction of system state are achieved based on the state error Kalman filter algorithm.

7. According to claim 1, a method for controlling an unmanned sailboat based on a deep reinforcement learning algorithm, It is characterized in that The simulated wind speed function and simulated wind direction function are respectively: ； ； in, is the simulated wind speed at one sampling time, is the given median wind speed, is the wind speed at the last sampling time, is the standard normal distribution function, is the simulated wind direction at one sampling time, is the given median wind direction, is the wind speed at the previous sampling time.

Images