A method for obstacle avoidance and navigation control of unmanned sailboats based on deep reinforcement learning algorithms

Abstract

This invention discloses an obstacle avoidance navigation control method for unmanned sailboats based on a deep reinforcement learning algorithm. The method includes collecting state data of the unmanned sailboat and the position information of obstacles in the environment, and combining this with the current path endpoint information to construct the current state vector; acquiring historical data within a set time period, and concatenating the state vectors and action results from the historical data according to time steps to form historical sequence information; inputting the current state vector and historical sequence information into a trained unmanned sailboat obstacle avoidance model, so that the unmanned sailboat obstacle avoidance model outputs action results based on the current state vector and historical sequence information. This invention uses historical sequences to assist in decision-making based on the current state, enabling navigation tasks in all wind directions without relying on wind field information. It can directly output sail and rudder control decisions to complete local obstacle avoidance navigation of the unmanned sailboat, possessing potential for field deployment and providing positive technical guidance for the autonomous navigation of unmanned sailboats.

Description

Technical Field

[0001] This invention relates to the field of obstacle avoidance technology for unmanned sailboats, and specifically to a method for obstacle avoidance navigation control of unmanned sailboats based on deep reinforcement learning algorithms. Background Technology

[0002] Unmanned sailboats, as highly automated navigation platforms, are widely used in fields such as marine research, environmental monitoring, and disaster relief. Compared with traditional powered unmanned surface vessels (USVs), unmanned sailboats rely on wind power for navigation, offering lower energy consumption and longer range. However, the instability and directional nature of wind pose more complex challenges to the control systems of unmanned sailboats, especially when performing autonomous obstacle avoidance in the ever-changing marine environment, where the need for such capabilities is particularly urgent.

[0003] The GNC (guidance-navigation-control) system is a crucial system in unmanned sailboat navigation. The guidance system, responsible for path planning and target setting, acts as the highest decision-making layer and typically uses the potential field method to generate the desired course and trajectory in obstacle avoidance scenarios. The navigation system obtains the real-time state of the unmanned sailboat based on sensors and related algorithms. The control system, based on the real-time state of the navigation system and the desired state of the guidance system, controls the sails and rudder to ensure the unmanned sailboat navigates along the desired path. The difference between obstacle avoidance scenarios and normal navigation lies primarily in the guidance system's need to perform entirely new navigation planning based on incoming obstacle information, and the need to maintain a sufficient distance from obstacles at sea (due to significant inertia). In the field of unmanned sailboats, current practices involve planning a path first and then tracking it for autonomous navigation. This aligns with the mainstream research and applications of autonomous driving in automobiles. This path planning + path tracking approach offers stronger verifiability and is easier to deploy in practice. Path planning algorithms typically use, for example... Dijkstra's method is used for global path planning, combined with methods such as potential field method and dynamic window method (DWA) to achieve local path planning. The path tracking part is achieved by adjusting the sail and rudder angles through LOS (line-of-sight) and backstepping to achieve heading alignment.

[0004] In recent years, researchers have used artificial intelligence methods to address the shortcomings of traditional algorithms and have achieved reliable validation. However, there has been almost no research on one-piece end-to-end models. The difference between a one-piece end-to-end model and the aforementioned path planning + path tracing approach lies in unifying planning and tracing within a single model. The model can make corresponding sail and rudder control decisions based on the current data state. For unmanned sailboats and other maritime vehicles, the complexity and factors involved in surface navigation are far less than those for autonomous vehicles. A one-piece end-to-end model can actually demonstrate its simplicity and significantly reduce the demand for various technologies. Summary of the Invention

[0005] The purpose of this invention is to address the shortcomings of existing technologies by providing an obstacle avoidance navigation control method for unmanned sailboats based on deep reinforcement learning algorithms.

[0006] To achieve the above objectives, this invention provides a method for obstacle avoidance navigation control of unmanned sailboats based on deep reinforcement learning algorithms, comprising: Collect the status data of the unmanned sailboat and the location information of obstacles in the environment, and combine them with the current path endpoint information to form the current state vector; Acquire historical data within a set time period, and concatenate the state vectors and action results in the historical data according to time steps to form historical sequence information; The current state vector and historical sequence information are input into the trained unmanned sailboat obstacle avoidance model so that the unmanned sailboat obstacle avoidance model outputs action results based on the current state vector and historical sequence information. The action results include changes in sail angle and rudder angle.

[0007] Furthermore, the unmanned sailboat's status data includes the unmanned sailboat's x-coordinate, y-coordinate, roll angle, yaw angle, forward and backward speed, lateral speed, x-axis rotation speed, z-axis rotation speed, sail angle, rudder angle, decrease in distance relative to the endpoint per unit sampling time, and decrease in projected distance relative to the endpoint per unit sampling time; the location information of obstacles in the environment includes the x-coordinate of the nearest obstacle, the y-coordinate of the nearest obstacle, the x-coordinate of the second nearest obstacle, and the y-coordinate of the second nearest obstacle; the current path endpoint information includes the endpoint x-coordinate and endpoint y-coordinate.

[0008] Furthermore, the unmanned sailboat obstacle avoidance model includes an action network, which includes: The first fully connected layer is used to receive the current state vector as input and extract instantaneous state features from the current state vector; The second fully connected layer is used to receive the input historical sequence information and extract historical feature sequences from the historical sequence information; A first LSTM network is used to extract hidden state features across time steps from the historical feature sequence and obtain temporal decision features through normalization processing. The first splicing module is used to splice the instantaneous state features and the temporal decision features to obtain a joint feature vector; The third fully connected layer is used to generate high-level decision features based on the joint feature vector; The first output mapping module is used to output action results based on the high-level decision characteristics.

[0009] Furthermore, the first fully connected layer extracts instantaneous state features in the following way: ; in, These are the instantaneous state features extracted from the first fully connected layer. For activation function, Let this be the current state vector. This is the weight matrix of the first fully connected layer. This is the bias vector of the first fully connected layer. It is a real number.

[0010] Furthermore, the historical sequence information is represented as follows: ; in, This is the historical sequence information at the current moment. for The state vector at time t, , Let be the dimension of the state vector. for The result of the action at any moment, , As a dimension of action result, This represents the total number of steps in the historical timeline. The second fully connected layer extracts historical feature sequences in the following way: ; in, The historical feature sequence extracted from the second fully connected layer. This is the weight matrix of the second fully connected layer. This is the bias vector for the second fully connected layer.

[0011] Furthermore, the first LSTM network extracts temporal decision features in the following way: ; ; in, The temporal decision features extracted by the first LSTM network. For normalization operations, The hidden state features extracted by the first LSTM network. To extract features using a long short-term memory network.

[0012] Furthermore, the third fully connected layer generates high-level decision features in the following way: ; in, This is the weight matrix of the third fully connected layer. This is the joint feature vector formed by the first concatenation module. , This is the bias vector for the third fully connected layer.

[0013] Furthermore, the first output mapping module outputs the action results based on the high-level decision-making characteristics in the following way: Within the framework of the stochastic policy algorithm, the action distribution parameters are constructed based on the aforementioned high-level decision characteristics as follows: ; in, Let be the mean of the action distribution. This is the first weight matrix of the first output mapping module. The first bias vector of the first output mapping module; Based on the mean of the action distribution The generated action result is: ; in, This is the action result currently generated by the first output mapping module. The hyperbolic tangent activation function is used. The mean of the action distribution standard deviation The symbol for element-wise multiplication. For standard normally distributed noise, It follows a standard normal distribution.

[0014] Furthermore, the unmanned sailboat obstacle avoidance model also includes: The first evaluation network is used to generate a first score during the training phase based on the current state vector, action result, and historical sequence information. The second evaluation network is used during the training phase by introducing a different capacity configuration than the first evaluation network and generating a second score based on the current state vector, action result, and historical sequence information.

[0015] Furthermore, the unmanned sailboat obstacle avoidance model is trained based on a reward function R, which is expressed as: ; in, This is the dynamic weight adjustment coefficient. As a reward for sailing progress, Penalty for smooth movement Rewards for checkpoints along the way, A reward for reaching the finish line. For time penalty items, This is a penalty for obstacle avoidance.

[0016] Beneficial effects: This invention uses historical sequences to assist in decision-making based on the current state, enabling navigation in all wind directions without relying on wind field information. It can directly output sail and rudder control decisions to complete local obstacle avoidance navigation of unmanned sailboats, has the potential for field deployment, and provides positive technical guidance for the autonomous navigation of unmanned sailboats. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of the structure of an unmanned sailboat obstacle avoidance navigation control method based on a deep reinforcement learning algorithm according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the motion network of an unmanned sailboat obstacle avoidance model; Figure 3 This is a schematic diagram of the first evaluation network / second evaluation network of the unmanned sailboat obstacle avoidance model; Figure 4 This is a schematic diagram of the sailing trajectory of an unmanned sailboat in a downwind situation; Figure 5 Is with Figure 4 A schematic diagram showing the changes in sail angle corresponding to the navigation trajectory of an unmanned sailboat; Figure 6 Is with Figure 4 A schematic diagram showing the rudder angle changes corresponding to the navigation trajectory of an unmanned sailboat; Figure 7 This is a schematic diagram of the sailing trajectory of an unmanned sailboat in headwind conditions; Figure 8 Is with Figure 7 A schematic diagram showing the changes in sail angle corresponding to the navigation trajectory of an unmanned sailboat; Figure 9 Is with Figure 7 A schematic diagram showing the rudder angle changes corresponding to the navigation trajectory of an unmanned sailboat. Detailed Implementation

[0018] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. These embodiments are implemented based on the technical solutions of the present invention, and it should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention.

[0019] like Figure 1 As shown, this embodiment of the invention provides a method for obstacle avoidance navigation control of an unmanned sailboat based on a deep reinforcement learning algorithm, including: The system collects state data of the unmanned sailboat and the location information of obstacles in the environment, and combines this data with the current path endpoint information to construct the current state vector. Specifically, the state vector consists of 18 parameters. The unmanned sailboat's state data includes: its x-coordinate, y-coordinate, roll angle, yaw angle, forward / backward speed, lateral speed, x-axis rotational speed, z-axis rotational speed, sail angle, rudder angle, decrease in distance relative to the endpoint per unit sampling time, and decrease in projected distance relative to the endpoint per unit sampling time. The obstacle location information includes the x-coordinate, y-coordinate, second-nearest obstacle, and third-nearest obstacle positions of the observed obstacle. The current path endpoint information includes the endpoint's x-coordinate and y-coordinate.

[0020] Historical data within a set time period is acquired, and the state vectors and action results from the historical data are concatenated according to time steps to form historical sequence information. Specifically, the aforementioned historical data can be represented as follows: ; in, To obtain historical data, for The state vector at time t, , The dimension of the state vector is 18 in this embodiment of the invention. for The result of the action at any moment, , The dimension of the action result is 2 in this embodiment of the invention. , This represents the total number of steps in the historical timeline.

[0021] The above historical sequence information can be represented as: ; in, This represents the historical sequence information at the current moment.

[0022] The current state vector and historical sequence information are input into the trained unmanned sailboat obstacle avoidance model, enabling the model to output action results based on these information. These action results include changes in sail angle. and rudder angle change .

[0023] The aforementioned obstacle avoidance model for unmanned sailboats includes an actor network, whose role is to process the current state vector. and historical data The mapping is to the action result, and the overall mapping relationship can be simply represented as: ; in, This indicates that action results are generated by sampling from the distribution. For learnable parameters The policy distribution, representing the current state vector. +Historical Data Below, the result of the action The probability distribution, Learnable parameters for action networks The strategy function, To extract features using a long short-term memory network.

[0024] For details, see Figure 2 The aforementioned action network includes a first fully connected layer 11, a second fully connected layer 12, a first LSTM network 13, a first splicing module 14, a third fully connected layer 15, and a first output mapping module 16.

[0025] The first fully connected layer 11 is used to receive the input current state vector. and from the current state vector The instantaneous state features are extracted as follows: ; in, The instantaneous state features extracted from the first fully connected layer 11 For activation function, This is the weight matrix of the first fully connected layer 11. This is the bias vector of the first fully connected layer 11. It is a real number.

[0026] The second fully connected layer 12 is used to receive the input historical sequence information. and from historical sequence information Extracting historical feature sequences, as follows: ; in, The historical feature sequence extracted from the second fully connected layer 12. This is the weight matrix of the second fully connected layer. This is the bias vector of the second fully connected layer 12.

[0027] The first LSTM network 13 is used to extract historical feature sequences. We extract hidden state features across time steps and obtain temporal decision features through normalization, as follows: ; ; in, The temporal decision features extracted by the first LSTM network 13 For normalization operations, For the first LSTM network 13, from historical feature sequences The hidden state features extracted from it.

[0028] The first concatenation module 14 is used to concatenate the instantaneous state features with the temporal decision features to obtain a joint feature vector, as follows: ; in, This is the joint feature vector obtained by splicing the first splicing module 14.

[0029] The third fully connected layer 15 is used based on the joint feature vector. Generate high-level decision-making features, as follows: ; in, High-level decision features generated for the third fully connected layer 15 This is the weight matrix for the third fully connected layer 15. This is the bias vector for the third fully connected layer 15.

[0030] The first output mapping module 16 is used to map high-level decision-making characteristics. The output result of the action is as follows: Within the framework of stochastic policy algorithms, based on the characteristics of high-level decision-making... The parameters for constructing the action distribution are: ; in, Let be the mean of the action distribution. This is the first weight matrix of the first output mapping module 16. This is the first bias vector of the first output mapping module 16.

[0031] Based on the mean of action distribution The generated action result is: ; in, This is the action result currently generated by the first output mapping module 16. The hyperbolic tangent activation function is used. The mean of the action distribution standard deviation The symbol for element-wise multiplication. For standard normally distributed noise, It follows a standard normal distribution. Additionally, the first output mapping module 16 has two parallel line layers, one of which is used to output the mean of the action distribution. Another line layer is used to calculate the logarithmic standard deviation. for: ; in, The second weight matrix of the first output mapping module 16 This is the second bias vector for the first output mapping module 16. (Logarithmic standard deviation) Only used in training to guarantee the standard deviation. Non-negative and optimized for greater stability.

[0032] Furthermore, the unmanned sailboat obstacle avoidance model of this application also includes a first evaluation network and a second evaluation network. The first evaluation network is used to generate a first score during the training phase based on the current state vector, action result, and historical sequence information. The second evaluation network is used during the training phase by introducing a different capacity configuration than the first evaluation network, and generates a second score based on the current state vector, action result, and historical sequence information. The first and second evaluation networks have the same structure as the action network, but their parameters are initialized independently and trained separately to alleviate the overestimation problem of the value function; during policy update, the smaller of the two values ​​is used as the target estimate.

[0033] The input parameters for the first and second evaluation networks are the current state and action features. and historical sequence information Among them, the current state-action feature From the current state vector and action results This is formed by splicing together the above current state-action characteristics. See Figure 3 The first and second evaluation networks each use a fourth fully connected layer 21 for feature extraction to obtain high-dimensional features. for: ; in, This is the weight matrix of the fourth fully connected layer 21. This high-dimensional feature is the bias vector of the fourth fully connected layer 21. Used to describe the actions performed in the current state. The immediate value of related information.

[0034] For historical sequence information The features are extracted sequentially through the fifth fully connected layer 22 and the second LSTM network 23 to obtain the temporal decision features. The feature extraction methods of the fifth fully connected layer 22 and the second LSTM network 23 are the same as those of the second fully connected layer 12 and the first LSTM network 13, and will not be repeated here. The aforementioned high-dimensional features... and time-series decision characteristics The input is fed into the second splicing module 24, where it is spliced ​​along the feature dimension to form a joint evaluation feature. for: ; Then, the input is fed into the sixth fully connected layer 25 for nonlinear mapping to form high-level decision features before value assessment. It should be noted that the mapping method of the sixth fully connected layer 25 is the same as that of the third fully connected layer 15. However, since the sixth fully connected layer 25 of the first evaluation network contains 256 neurons (AND), while the sixth fully connected layer 25 of the second evaluation network contains 128 neurons, the high-level decision features generated by the first and second evaluation networks are different. The differences cause the second output mapping module 26 of the first and second evaluation networks to output the first score respectively. Second score By introducing different network capacity configurations into the two evaluation networks, the diversity of value estimation can be enhanced to some extent, further suppressing the problem of systematic overestimation.

[0035] The training method for the above-mentioned unmanned sailboat obstacle avoidance model is as follows: I. Environment and Task Setting Regarding the dynamic environment, the OU process is used to calculate changes in the wind field (wind direction and speed). In terms of task settings, this model primarily simulates situations with multiple obstacles that may occur during local navigation. Specific settings are as follows: 1. The starting point is set to (0,0), and the ending point is set to (500,0), with a distance of 500 meters between them. 2. Several obstacle points are randomly generated in the intermediate area between the starting and ending points (due to the actual distance between obstacles at sea and the limited maneuverability of the unmanned sailboat, the randomly generated obstacles are at least 120 meters apart). The unmanned sailboat needs to maintain a distance of at least 40 meters from the obstacles; if the distance is less than 40 meters, the task will be considered a failure. 3. For the coordinate information of the obstacles, a symmetrically distributed fan-shaped detection area is constructed in front of the current bow direction of the unmanned sailboat, using this as the central axis of the field of view. This detection field of view covers 45° to the left and right of the bow direction, forming a forward fan-shaped area with a total opening angle of 90°; radially, the detection range extends forward from the hull position to 150 m. Environmental information within this sector (such as the horizontal and vertical coordinates of obstacles) is considered valid perception information at the current moment. Distance calculations are performed on all perceived obstacles relative to the current position, and only the coordinates of the two closest obstacles are input into the state for model decision-making. 4. To improve the model's generality and training effectiveness, the overall environment will contain obstacles 70% of the time and will have no obstacles 30% of the time.

[0036] II. Reward Settings For reinforcement learning tasks, the reward setting largely determines the final training effect. The reward function used in this invention includes the following aspects: ;

[0037] in, This is the dynamic weight adjustment coefficient. The navigation progress reward mainly consists of the decrease in distance relative to the endpoint per unit sampling time, the decrease in projected distance relative to the endpoint per unit sampling time, the forward and backward speed of the unmanned sailboat, and the lateral speed of the unmanned sailboat. It is mainly used to guide the unmanned sailboat to move quickly toward the target point. As a penalty for smooth motion, the change in sail angle is used. and rudder angle change The design incorporates appropriate penalties based on changes in the sail and rudder angles, reducing the amplitude and frequency of adjustments during normal navigation and thus lowering energy consumption. For example, the change in sail angle at a given sampling moment... and rudder angle change If both are 0, then It will change from 1 to 1.2, further encouraging the model to maintain sail and rudder angles. By incorporating rewards for intermediate checkpoints and dividing the entire voyage into multiple segments, with rewards based on responses at each intermediate checkpoint, training effectiveness can be significantly improved. The reward for reaching the finish line plays a decisive role in the sailing direction of the unmanned sailboat. This is a time penalty to prevent unmanned sailboats from falling into unpredictable reward traps and to encourage them to complete tasks as quickly as possible. As an obstacle avoidance penalty, it draws on the use of repulsive force in the traditional potential field method, calculates the distance based on the coordinates of the obstacle in the input observation space, and this penalty only takes effect when the distance to the obstacle is less than 100 meters, and reaches its maximum when the distance is less than or equal to 50 meters.

[0038] III. Model Training Results The overall model reached a stable state after sampling 12.6 million data points (with only the most recent 3 million data points retained in the experience replay pool) and performing 336,000 iterations of the critic network and 84,000 iterations of the actor network. Obstacle avoidance tests were conducted for different wind directions, and the model achieved a task completion rate of over 95% in all cases.

[0039] IV. Implementation Cases In this implementation case, the total number of historical time steps T is a fixed value of 10. This historical sequence is used to characterize the continuous maneuvering trend and motion inertia of the unmanned sailboat during navigation.

[0040] In ocean navigation scenarios, the two nearest obstacles usually constitute the main constraints on the navigation safety of unmanned sailboats; by introducing the spatial position information of the two obstacles at the same time, the local optimal path problem caused by avoiding only a single obstacle can be effectively avoided.

[0041] By introducing the specific spatial coordinates of obstacles as state features and combining them with historical state-action sequences for temporal modeling, the unmanned sailboat can adjust its navigation strategy in advance based on the evolution trend of its own position and the position of obstacles. In this embodiment, no rule-based obstacle avoidance algorithm or explicit collision detection module is used. Instead, a reinforcement learning policy network learns the mapping relationship between the spatial distribution of obstacles and safe maneuvering through a large number of interactions, thereby achieving adaptive obstacle avoidance in complex environments.

[0042] Given the limited turning ability of unmanned sailboats, obstacles in this implementation case are set at least 150m apart, and the minimum obstacle avoidance range is a 40m radius area centered on the obstacle (entering this area during the test phase will be considered a mission failure). At least 5 obstacles are distributed in the area from the starting point to the end point.

[0043] The implementation case tested navigation missions under different wind directions, achieving near 100% obstacle avoidance accuracy in tailwind conditions (meaning point-to-point navigation based on obstacle avoidance, rather than simply avoiding individual obstacles). Specifically, as follows... Figure 4 As shown, Figure 4 This diagram illustrates the sailing trajectory of an unmanned sailboat in a downwind environment, with the changes in sail angle and rudder angle as shown below. Figure 5 and Figure 6 As shown. Due to the field of view, obstacle detection is gradually refined during the movement, resulting in a dynamic adjustment process during navigation, successfully avoiding the set danger radius. In a completely headwind environment, the difficulty of obstacle avoidance increases further because the sailboat needs to navigate at an angle to gain forward momentum. Tests showed an 85% success rate, which increased to 95% when the minimum obstacle avoidance radius was further reduced to 20m. Based on the above simulation tests, the model's obstacle avoidance capability in multi-obstacle environments was successfully verified. Specifically, as shown... Figure 7 As shown, Figure 7 This diagram illustrates the sailing trajectory of an unmanned sailboat in headwind conditions, with the changes in sail angle and rudder angle as shown below. Figure 8 and Figure 9 As shown, the unmanned sailboat successfully navigated between multiple obstacles and reached the target location, validating the model's applicability in complex environments.

[0044] The above description is merely a preferred embodiment of the present invention. It should be noted that for those skilled in the art, other parts not specifically described are existing technology or common knowledge. Several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for obstacle avoidance navigation control of an unmanned sailboat based on a deep reinforcement learning algorithm, characterized in that, include: Collect the status data of the unmanned sailboat and the location information of obstacles in the environment, and combine them with the current path endpoint information to form the current state vector; Acquire historical data within a set time period, and concatenate the state vectors and action results in the historical data according to time steps to form historical sequence information; The current state vector and historical sequence information are input into the trained unmanned sailboat obstacle avoidance model so that the unmanned sailboat obstacle avoidance model outputs action results based on the current state vector and historical sequence information. The action results include changes in sail angle and rudder angle.

2. The obstacle avoidance navigation control method for unmanned sailboats based on deep reinforcement learning algorithm according to claim 1, characterized in that, The unmanned sailboat's status data includes its x-coordinate, y-coordinate, roll angle, yaw angle, forward and backward speed, lateral speed, x-axis rotation speed, z-axis rotation speed, sail angle, rudder angle, decrease in distance relative to the endpoint per unit sampling time, and decrease in projected distance relative to the endpoint per unit sampling time; the location information of obstacles in the environment includes the x-coordinate of the nearest obstacle, the y-coordinate of the nearest obstacle, the x-coordinate of the second nearest obstacle, and the y-coordinate of the second nearest obstacle. The current path endpoint information includes the endpoint's x-coordinate and endpoint's y-coordinate.

3. The obstacle avoidance navigation control method for unmanned sailboats based on deep reinforcement learning algorithm according to claim 1, characterized in that, The unmanned sailboat obstacle avoidance model includes an action network, which includes: The first fully connected layer is used to receive the current state vector as input and extract instantaneous state features from the current state vector; The second fully connected layer is used to receive the input historical sequence information and extract historical feature sequences from the historical sequence information; A first LSTM network is used to extract hidden state features across time steps from the historical feature sequence and obtain temporal decision features through normalization processing. The first splicing module is used to splice the instantaneous state features and the temporal decision features to obtain a joint feature vector; The third fully connected layer is used to generate high-level decision features based on the joint feature vector; The first output mapping module is used to output action results based on the high-level decision characteristics.

4. The obstacle avoidance navigation control method for unmanned sailboats based on deep reinforcement learning algorithm according to claim 3, characterized in that, The first fully connected layer extracts instantaneous state features in the following way: ； in, The instantaneous state features extracted from the first fully connected layer. For activation function, Let this be the current state vector. This is the weight matrix of the first fully connected layer. This is the bias vector of the first fully connected layer. It is a real number.

5. The obstacle avoidance navigation control method for unmanned sailboats based on deep reinforcement learning algorithm according to claim 4, characterized in that, The historical sequence information is represented as follows: ； in, This is the historical sequence information at the current moment. for The state vector at time t, , Let be the dimension of the state vector. for The result of the action at any moment, , As a dimension of action result, This represents the total number of steps in the historical timeline. The second fully connected layer extracts historical feature sequences in the following way: ； in, The historical feature sequence extracted from the second fully connected layer. This is the weight matrix of the second fully connected layer. This is the bias vector for the second fully connected layer.

6. The obstacle avoidance navigation control method for unmanned sailboats based on deep reinforcement learning algorithm according to claim 5, characterized in that, The first LSTM network extracts temporal decision features in the following way: ； ； in, The temporal decision features extracted by the first LSTM network. For normalization operations, The hidden state features extracted by the first LSTM network. To extract features using a long short-term memory network.

7. The obstacle avoidance navigation control method for unmanned sailboats based on deep reinforcement learning algorithm according to claim 6, characterized in that, The third fully connected layer generates high-level decision features in the following way: ； in, This is the weight matrix of the third fully connected layer. This is the joint feature vector formed by the first concatenation module. , This is the bias vector for the third fully connected layer.

8. The obstacle avoidance navigation control method for unmanned sailboats based on deep reinforcement learning algorithm according to claim 7, characterized in that, The first output mapping module outputs the action results based on the high-level decision characteristics in the following way: Within the framework of the stochastic policy algorithm, the action distribution parameters are constructed based on the aforementioned high-level decision characteristics as follows: ； in, Let be the mean of the action distribution. This is the first weight matrix of the first output mapping module. This is the first bias vector of the first output mapping module; Based on the mean of the action distribution The generated action result is: ； in, This is the action result currently generated by the first output mapping module. The hyperbolic tangent activation function is used. The mean of the action distribution standard deviation The symbol for element-wise multiplication. For standard normally distributed noise, It follows a standard normal distribution.

9. The obstacle avoidance navigation control method for unmanned sailboats based on deep reinforcement learning algorithm according to claim 1, characterized in that, The unmanned sailboat obstacle avoidance model also includes: The first evaluation network is used to generate a first score during the training phase based on the current state vector, action result, and historical sequence information. The second evaluation network is used during the training phase by introducing a different capacity configuration than the first evaluation network and generating a second score based on the current state vector, action result, and historical sequence information.

10. The obstacle avoidance navigation control method for unmanned sailboats based on deep reinforcement learning algorithm according to claim 1, characterized in that, The unmanned sailboat obstacle avoidance model is trained based on a reward function R, which is expressed as: ； in, This is the dynamic weight adjustment coefficient. As a reward for sailing progress, Penalty for smooth movement Rewards for checkpoints along the way, A reward for reaching the finish line. For time penalty items, This is a penalty for obstacle avoidance.

Images