A method and system for autonomous navigation of unmanned boat based on reinforcement learning

Abstract

The present invention provides an unmanned boat autonomous navigation method and system based on reinforcement learning, which belongs to the field of navigation control technology based on artificial intelligence; first, a real underwater environment and an unmanned boat model are created by using Gazebo software for simulation. Secondly, a multi-reward mechanism fusion function including dense rewards, sparse rewards and reward penalties is designed to improve the operational consistency and efficiency of the unmanned boat in underwater autonomous navigation. Subsequently, an unmanned boat autonomous navigation model is constructed, and the unmanned boat autonomous navigation model is optimized in combination with deep Q learning and attention mechanism to improve the learning efficiency of the model and its robustness in complex environments. Through multi-stage training and supervised learning strategies, the training time is further shortened and the intelligence level is improved. Finally, the sonar radar and inertial navigation system are deployed on the unmanned boat with the optimized model to realize real-time autonomous intelligent navigation.

Description

Technical Field

[0001] The present invention belongs to the field of navigation control technology based on artificial intelligence, and in particular relates to an autonomous navigation method and system for an unmanned boat based on reinforcement learning. Background Art

[0002] Unmanned boats refer to underwater vehicles without human control. They use advanced sensors, navigation systems and artificial intelligence technologies to achieve autonomous navigation and mission execution. The autonomous navigation capability of unmanned boats is one of its core technologies, which not only involves real-time path planning and obstacle avoidance, but also includes decision-making and actions in complex environments. Unmanned boats can perform tasks such as marine environmental monitoring, search and rescue, and logistics transportation for a long time without human supervision. This technological advancement reduces the dependence on manual operation and reduces labor costs, while improving the accuracy of operations and the continuity of tasks. Especially for complex and dangerous environments, such as bad weather or hostile waters, unmanned boats can avoid putting people at risk and achieve more efficient resource utilization. In addition, the autonomous navigation system of unmanned boats can autonomously adjust the navigation path in the face of various dynamically changing environmental conditions, thereby increasing the probability of mission success. Therefore, the importance of autonomous navigation lies in improving the efficiency and safety of marine exploration and monitoring.

[0003] With the continuous development of technology, unmanned boats will play an increasingly important role in future ocean exploration and marine activities, and provide new solutions for the sustainable utilization of marine resources. The existing unmanned boat navigation methods mainly include the following two types: visual navigation method and fusion navigation technology.

[0004] Visual navigation method: The visual navigation method uses the camera and computer vision technology on the unmanned boat to capture and analyze environmental images in real time to identify and track surrounding landmarks, obstacles and other key features. Image processing algorithms process this data to generate a real-time map of the environment and use it for dynamic path planning. The unmanned boat can adjust its route to avoid obstacles according to environmental changes and achieve autonomous navigation. Visual navigation provides high-resolution environmental perception and can adapt to complex and dynamic water conditions, but this navigation method will be greatly affected in low-light environments and will not be able to ensure the normal navigation mission.

[0005] Fusion navigation technology: Fusion navigation technology provides comprehensive and accurate environmental perception by integrating data from sensors such as GPS and inertial navigation system (INS). This method uses data fusion algorithms (such as Kalman filter) to combine information from different sensors to generate more reliable positioning and navigation data, thereby improving the navigation accuracy and robustness of the unmanned boat in complex and changing environments. This technology can make up for the shortcomings of a single navigation system when the signal is unstable or the environmental conditions are not ideal, ensuring that the autonomous navigation of the unmanned boat is stable and reliable.

[0006] However, visual navigation relies on cameras and image processing algorithms, which may be limited in low light or strong light interference. At the same time, real-time processing requirements are high and may face challenges in complex environments. Fusion navigation technology provides accurate navigation by integrating multiple sensor data, but its system complexity and high cost, as well as data synchronization issues are its main shortcomings. Therefore, the present invention proposes an autonomous navigation method and system for unmanned boats based on reinforcement learning, which continuously optimizes the navigation strategy through the interaction between the intelligent agent and the environment, adapts to the dynamically changing environment, reduces the need for manual adjustment, and thus improves the navigation performance and robustness of the unmanned boat under various conditions. Summary of the invention

[0007] In view of the above problems, the first aspect of the present invention provides an unmanned boat autonomous navigation method based on reinforcement learning, comprising the following process:

[0008] Deploy the trained autonomous navigation model of the unmanned boat to the terminal of the unmanned boat; send the location information of the navigation target to the autonomous navigation model of the unmanned boat to carry out the autonomous navigation task;

[0009] Based on the sonar radar and inertial navigation system deployed on the surface of the unmanned boat, the surrounding obstacle information and its own status information are obtained in real time during the unmanned boat's navigation mission. The collection cycle is consistent with the data collection cycle during model training.

[0010] The collected data is input into the autonomous navigation model of the unmanned boat, and the model outputs the correct heading direction and navigation speed of the unmanned boat at each state moment;

[0011] The model output results are sent to the control system of the unmanned boat to adjust the real-time heading direction and navigation speed of the unmanned boat to complete the navigation task;

[0012] The unmanned boat autonomous navigation model is trained based on an underwater simulation environment, and a reward function integrating multiple reward mechanisms is designed, including sparse rewards, penalty negative rewards, and dense rewards; the sparse rewards are used to guide the unmanned boat to complete the navigation task, the penalty negative rewards are used to guide the unmanned boat to avoid collision with surrounding obstacles, and the dense rewards are used to guide the unmanned boat to approach the target and determine the real-time heading in real time; the unmanned boat autonomous navigation model is based on deep Q learning, and based on the acquired position information of itself , surrounding obstacle information And navigation target location information , output the optimal navigation speed corresponding to the unmanned boat reaching the navigation target point at this time and sailing direction .

[0013] Preferably, the unmanned boat autonomous navigation model is trained in a simulation environment; an unmanned boat model is created in the Gazabo software, and the features of the unmanned boat simulation model are calibrated with reference to the shape of the real unmanned boat, and the mass, hull density, center of gravity, and material parameters of the unmanned boat simulation model are designed with reference to the corresponding parameters of the real unmanned boat; at the same time, a simulated ocean environment is created using the Gazabo software, and underwater obstacles are added to the simulated ocean environment, including shipwrecks, reefs, and schools of fish, and ocean currents, turbulence, and submarine undercurrents are added to simulate the impact of water flow on the driving of the unmanned boat in real scenarios; in the simulation environment, a sonar sensor simulation model is added to the unmanned boat simulation model to collect distance information of surrounding obstacles from the unmanned boat in the simulation environment. , and add an inertial navigation simulation system consisting of a gyroscope and accelerometer to obtain real-time unmanned boat position information And the navigation target point location information , and combined with the real-time location information of the unmanned boat And navigation target point location information The absolute distance information of the unmanned boat relative to the navigation target point is calculated.

[0014] Preferably, the sparse reward is specifically that during the process of the unmanned boat trying various navigation methods to try to approach the navigation target point, if the real-time position of the unmanned boat is finally and the location of the navigation target point The absolute distance If the distance is less than 0.5m, the unmanned boat is considered to have successfully reached the target point, and a reward value is given to this navigation method. For navigation methods that fail to successfully reach the target point, no reward value is given.

[0015]

[0016] in, Represents the sparse module reward value. When the unmanned boat can successfully reach the target navigation point, its value is set to 20. If it fails to reach the target navigation point, its value is 0. Respectively represent the real-time position of the unmanned boat coordinate, Respectively represent the location of the unmanned boat navigation target point coordinate.

[0017] Preferably, the penalty negative reward is specifically that during the process of reinforcement learning training, when the unmanned boat collides with surrounding obstacles, a penalty is given to this navigation mode; when the distance between the unmanned boat and the surrounding obstacles is less than 0.01m, it is considered that the unmanned boat and the surrounding obstacles have collided, and when the distance with the surrounding obstacles is greater than 0.01m, it is considered that no collision has occurred;

[0018]

[0019] in, represents the penalty value, Represents the distance between the unmanned boat and surrounding obstacles, which is collected by sonar radar sensors.

[0020] Preferably, the intensive reward is to record the position of the unmanned boat in the previous state as , the position of the unmanned boat after the state is updated, that is, after the unmanned boat completes one step of operation, is recorded as , and combined with the location of the navigation target point Calculate the distance from the unmanned boat to the navigation target point in the previous state And the distance from the unmanned boat to the navigation target point after the status is updated :

[0021]

[0022]

[0023] They respectively represent the status update, that is, the position of the unmanned boat after the unmanned boat completes an operation. coordinate, Respectively represent the location of the unmanned boat navigation target point coordinates; when the unmanned boat is in the middle of the driving process, if its The value is less than The value of , that is, the actual position of the unmanned boat is closer to the final navigation target point, then the driving operation just performed by the unmanned boat is considered to be a correct operation, and a reward value is given to such an operation ;

[0024]

[0025] At the same time, during the driving process of the unmanned boat, in addition to optimizing the driving behavior of the unmanned boat by gradually approaching the navigation target point, the navigation direction of the unmanned boat should also be optimized. The navigation angle of the unmanned boat at this time is defined as , the actual angle between the position of the unmanned boat and the navigation target point is defined as , calculate the angle difference at this time :

[0026]

[0027] When the angle difference is smaller, it means that the navigation direction of the unmanned boat is more correct, and different reward values ​​are given to it according to different angle differences. :

[0028]

[0029] in, Represents the absolute value operation.

[0030] Preferably, the autonomous navigation model of the unmanned boat includes an information processing module and a behavior function module; the information processing module is used for processing input data during training, and the behavior function module first processes the data through two LSTM layers respectively. and Perform feature extraction, and then activate the features through the Softmax activation function to obtain intermediate features and ;

[0031]

[0032]

[0033] in, represent Operation function, represent Layer operation functions;

[0034] Then, the self-attention mechanism is used to and Perform automatic fusion to obtain fusion features , to be more relevant to specific navigation tasks:

[0035] Then, the fusion features are transformed through a fully connected layer, a BN normalization layer, and a Relu activation function. After further processing, the optimal sailing speed at this moment is obtained and sailing direction .

[0036] Preferably, when the unmanned boat is undergoing reinforcement learning training, the information processing module retains the information of the unmanned boat and surrounding obstacles during the environmental interaction process. , unmanned boat location information , navigation target location information , and use this information as the input variable of the subsequent behavior function module, and save the real-time navigation speed of the unmanned boat and sailing direction And the final reward value obtained during this learning process ;

[0037] When collecting information, the sampling period is set to 1s, that is, the unmanned boat will correspond to a new state every 1s during a complete environmental interaction process. For a complete environmental interaction process, the surrounding obstacle information collected , unmanned boat location information , and the speed of the unmanned boat and the direction of the unmanned boat Each is a specific sequence.

[0038] Preferably, a multi-stage training mechanism and a supervised learning strategy are integrated during training, including preliminary training, intensive training and optimization training;

[0039] The preliminary training is to record the overall data of 20 navigation processes , conduct training on the operation of unmanned boats;

[0040] The reinforcement training changes the number of collected data groups to 50 groups, and still only retains the navigation data of the unmanned boat with the top 5 total reward function values, and uses the retained data to train the reinforcement learning model and optimize its model parameters; repeat the iteration until the navigation success rate exceeds 100% in each data process. The basis for successful navigation is the real-time position of the unmanned boat. and the location of the navigation target point The absolute distance If it is less than 0.5m, the navigation is considered to be completed successfully;

[0041]

[0042] Respectively represent the real-time position of the unmanned boat coordinate, Respectively represent the location of the unmanned boat navigation target point coordinate;

[0043] The optimization training ensures that the unmanned boat has a faster overall navigation speed and a shorter travel distance during the navigation process. On the basis of the intensive training, 20 sets of navigation data of the unmanned boat are still collected. At this time, the 5 sets of navigation data with the shortest overall navigation time among the 20 sets of heading data are selected to carry out optimization training on the reinforcement learning model again, and the number of training iterations is set to 8 times.

[0044] The second aspect of the present invention provides an unmanned boat autonomous navigation system based on reinforcement learning, which applies the unmanned boat autonomous navigation method as described in the first aspect; it includes an information perception module, an unmanned boat control system, a memory and a power supply module;

[0045] The information perception module includes a sonar radar sensor, a gyroscope and an acceleration sensor, which are used to collect surrounding obstacle information and navigation status information of the unmanned boat in real time; the collected information is then sent to the unmanned boat autonomous navigation model as described in the first aspect for analysis and processing, and the output result is sent to the unmanned boat control system to complete the real-time adjustment of the navigation direction and speed of the unmanned boat;

[0046] They are all powered by a unified power supply module, and the navigation status information is saved in real time using the memory.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] (1) Improving the degree of automation and intelligence: This method can significantly improve the degree of automation and intelligence of unmanned boats in performing navigation tasks by designing an unmanned boat autonomous navigation method and system based on reinforcement learning. In addition, the multiple reward mechanisms designed in the present invention integrate reward functions, which can improve the training quality of the autonomous navigation model, clarify the goals of the navigation tasks, and optimize their specific driving details to enhance the degree of automation of the entire navigation process. At the same time, an autonomous navigation model integrating a deep neural network is designed, which can effectively optimize the real-time operation of the unmanned boat and ensure the correctness of the operation. Finally, the performance of the autonomous navigation model is further improved through a multi-stage training mechanism and a supervised learning strategy, thereby enhancing the degree of automation and intelligence of the unmanned boat in performing navigation tasks;

[0049] (2) Improving the accuracy of navigation operations: The present invention constructs a variety of reward functions with reward mechanisms, including sparse rewards, dense rewards, and penalty negative reward mechanisms. The reward function is the main optimization basis used in reinforcement learning. The reward function designed by the present invention can effectively improve the accuracy of navigation operations. To ensure the overall safety of the unmanned boat when performing navigation tasks, it will not collide with surrounding obstacles, thereby ensuring the smooth progress of the unmanned boat navigation task and reducing the loss of economic resources;

[0050] (3) Optimize navigation trajectory and improve the overall efficiency of navigation tasks: In the present invention, a multi-stage training mechanism is used during the actual training of unmanned boats to gradually improve the correctness of unmanned boat operation and its overall intelligence level, and supervised learning is introduced on the basis of the multi-stage training mechanism. After ensuring that the navigation task can be completed normally, the navigation trajectory and navigation time of the unmanned boat are further optimized, so that when the unmanned boat performs a specific unmanned navigation task, the overall time to complete the navigation task will be shortened. This can not only improve the efficiency of the entire navigation process, but also because the unmanned boat will choose a more economical route, the energy consumed during the navigation process will be further reduced;

[0051] (4) Improving the robustness of navigation operations: In the present invention, an autonomous navigation model of an unmanned boat that integrates a deep neural network is designed, and an attention mechanism is designed in the deep neural network to better adapt the unmanned boat to the actual driving environment. Using the model designed by the present invention, the normal implementation of navigation tasks can be guaranteed under different water conditions, and the navigation tasks can still be successfully completed in the presence of various interference factors underwater. Improve the adaptability of the unmanned boat to the environment and enhance its robustness in navigation operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 The present invention is a logical flow chart of the overall implementation of the unmanned boat autonomous navigation method.

[0053] Figure 2 This is a structural block diagram of the unmanned boat autonomous navigation system of the present invention.

[0054] Figure 3 This is a network structure diagram of the unmanned boat autonomous navigation model of the present invention.

[0055] Figure 4 This is a flow chart of the training of the autonomous navigation model of the unmanned boat of the present invention.

[0056] Figure 5 This is the application flow chart of the model after deployment.

[0057] Figure 6 This is a performance comparison chart of the algorithm model of the present invention.

[0058] Figure 7 This is a comparison chart of experimental results. DETAILED DESCRIPTION

[0059] The overall implementation logic of the present invention is as follows Figure 1 As shown in the figure. First, the Gazebo software is used to create a real underwater environment and unmanned boat model for simulation to reduce resource consumption and cost in actual training. Secondly, a multi-reward mechanism fusion function including dense rewards, sparse rewards and reward penalties is designed to improve the operational consistency and efficiency of the unmanned boat in underwater autonomous navigation. Subsequently, an unmanned boat autonomous navigation model is constructed, and this unmanned boat autonomous navigation model is optimized by combining deep learning and attention mechanism to improve the learning efficiency of the model and its robustness in complex environments. Then, through multi-stage training and supervised learning strategies, the training time is further shortened and the intelligence level is improved. Finally, the sonar radar and inertial navigation system are deployed on the unmanned boat with the optimized model to realize real-time autonomous intelligent navigation.

[0060] The overall structure of the unmanned boat autonomous navigation system in the present invention is as follows: Figure 2 As shown, it includes an information perception module, an unmanned boat control system, a memory and a power supply module;

[0061] The information perception module includes sonar radar sensors, gyroscopes, and accelerometers, which are used to collect information about surrounding obstacles and the navigation status of the unmanned boat in real time. The collected information is then sent to the trained unmanned boat autonomous navigation model for analysis and processing, and the output results are sent to the unmanned boat control system to complete the real-time adjustment of the unmanned boat's navigation direction and speed. The above systems and modules are all powered by a unified power supply module, and the navigation status information is saved in real time using a memory. The unmanned boat control system, memory, and power supply module are all conventional structures, and will not be described in detail here.

[0062] The invention will be further described below in conjunction with specific embodiments.

[0063] 1. Construction of reinforcement learning simulation environment

[0064] As a learning process that interacts with the environment in real time, a real and effective objective environment can enhance learning efficiency and improve model performance. Since the process of reinforcement learning is a process of continuous trial and error, continuous optimization and adjustment, training in a real marine environment will bring great resource consumption. Therefore, in the present invention, Gazebo software is used to create the real underwater environment and unmanned boat model required for reinforcement learning, which can greatly reduce the resources consumed by model training, ensure the coherence and consistency of model training, and reduce the cost of realizing the unmanned boat autonomous navigation task.

[0065] Construction of underwater simulation environment: Reinforcement learning is a learning method that requires constant interaction with the surrounding environment. Training in a real ocean environment will cause a huge waste of resources, and the continuity of training cannot be guaranteed. Therefore, the present invention uses Gazabo software to create a simulated ocean environment. At the same time, in order to further simulate the real underwater driving environment of unmanned boats, underwater obstacles such as shipwrecks, reefs, and schools of fish are added to the simulated ocean environment. In addition, in the established underwater simulation environment, ocean currents, turbulence, and submarine undercurrents are added to simulate the impact of water flow on the driving of unmanned boats in real scenarios;

[0066] Unmanned boat model design and calibration: Create an unmanned boat model in Gazabo software. To further reduce the difference between the unmanned boat simulation model and the real unmanned boat, the unmanned boat simulation model is calibrated using the real unmanned boat parameters. Specifically: the features of the unmanned boat simulation model are calibrated with reference to the shape of the real unmanned boat. At the same time, the mass, hull density, center of gravity, and material parameters of the unmanned boat simulation model are designed with reference to the corresponding parameters of the real unmanned boat.

[0067] Sensor model design: In the autonomous navigation system of the unmanned boat deployed by the present invention, it is necessary to deploy a sonar radar on the surface of the unmanned boat to collect real-time information about the surrounding environment, and deploy an inertial navigation system composed of a gyroscope and an accelerometer to realize the real-time positioning of the unmanned boat. Therefore, in the constructed simulation environment, a sonar sensor simulation model is also added to the unmanned boat simulation model to collect the distance information of the surrounding obstacles from the unmanned boat in the simulation environment. , and add an inertial navigation simulation system consisting of a gyroscope and accelerometer to obtain real-time unmanned boat position information And the navigation target point location information , and combined with the real-time location information of the unmanned boat And navigation target point location information The absolute distance information of the unmanned boat relative to the navigation target point is calculated.

[0068] Design of unmanned boat motion control module: In the actual process of autonomous navigation of unmanned boat, the unmanned boat mainly changes its navigation trajectory by changing its navigation direction and navigation speed. Therefore, in the motion control module designed for the unmanned boat simulation model of the present invention, the motion state of the unmanned boat is controlled by controlling the navigation direction and navigation speed of the unmanned boat;

[0069] Construction of reinforcement learning simulation environment: A simulation environment for reinforcement learning is constructed based on various designed simulation models and control modules. The driving conditions of the unmanned boat simulation model in the underwater simulation environment are used to replace the real driving conditions of the unmanned boat, thereby realizing the construction of the reinforcement learning simulation environment.

[0070] 2. Construction of reward function for multiple reward mechanism integration in reinforcement learning

[0071] In the training process of reinforcement learning, the design of the reward function plays a vital role in the training results of reinforcement learning. The present invention designs a reward function that integrates multiple reward mechanisms to train and guide the driving operation of the unmanned boat. Among them, sparse rewards can guide the unmanned boat to successfully complete the navigation task, and the penalty of negative rewards is mainly to avoid the collision of the unmanned boat with surrounding obstacles when performing the navigation task, while the dense reward is to improve the intelligent operation of the unmanned boat during the navigation process. The reward function used by the final model includes the above three reward mechanisms.

[0072] Sparse reward module design: For the unmanned boat navigation task, its ultimate goal is to enable the unmanned boat to reach the navigation target point. Therefore, in the process of interactive learning between the unmanned boat and the underwater environment (that is, the process of the unmanned boat trying various navigation methods to try to approach the navigation target point), if the real-time position of the unmanned boat is finally and the location of the navigation target point The absolute distance If the distance is less than 0.5m, it is considered that the unmanned boat has successfully reached the target point. At this time, a larger reward value is given to this navigation method, while no reward value is given to the navigation method that does not successfully reach the target point;

[0073]

[0074] in, It represents the sparse module reward value. When the unmanned boat can successfully reach the target navigation point, its value is set to 20. If it fails to reach the target navigation point, its value is 0. Respectively represent the real-time position of the unmanned boat coordinate, Respectively represent the location of the unmanned boat navigation target point coordinate;

[0075] Design of penalty negative reward module: In the unmanned boat autonomous navigation task, collision with obstacles will cause damage to the unmanned boat and lead to failure of the autonomous navigation task. This situation is an accident that should be avoided in the real unmanned boat navigation. Therefore, in the process of reinforcement learning training, when the unmanned boat collides with surrounding obstacles, this navigation mode is given a penalty. In the present invention, when the distance between the unmanned boat and the surrounding obstacles is less than 0.01m, it is considered that the unmanned boat and the surrounding obstacles have collided. When the distance with the surrounding obstacles is greater than 0.01m, it is considered that no collision has occurred.

[0076]

[0077] in, represents the penalty value, Represents the distance between the unmanned boat and surrounding obstacles, collected by sonar radar sensors;

[0078] Design of intensive reward module: In the process of interactive learning between the unmanned boat and the environment, the reward function module described helps to clarify the navigation task of the unmanned boat, that is, to sail smoothly to the navigation target point and not collide with surrounding obstacles during the navigation process. However, if only these two types of reward functions are relied upon, although the goal of the unmanned boat navigation task is clarified, there is a lack of guidance for the unmanned boat during the intermediate driving process, which will introduce too many random operations in the training process of reinforcement learning, and it is impossible to gradually learn and standardize the behavior of the unmanned boat, which ultimately leads to the consumption of a large amount of computing resources. Therefore, the present invention designs a intensive reward function to guide the learning of the specific navigation operations of the unmanned boat.

[0079] In the autonomous navigation task, the unmanned boat gradually approaches the navigation target point, which is the key to achieve smooth navigation. In the training process of reinforcement learning, the position point of the unmanned boat in the previous state is recorded as , the position of the unmanned boat after the state is updated, that is, after the unmanned boat completes one step of operation, is recorded as . And combined with the location of the navigation target point Calculate the distance from the unmanned boat to the navigation target point in the previous state And the distance from the unmanned boat to the navigation target point after the status is updated :

[0080]

[0081]

[0082] in, They respectively represent the status update, that is, the position of the unmanned boat after the unmanned boat completes an operation. coordinate, Respectively represent the location of the unmanned boat navigation target point Coordinates. If the unmanned boat is in the middle of the driving process, The value is less than The value of , that is, the actual position of the unmanned boat is closer to the final navigation target point, then the driving operation just performed by the unmanned boat is considered to be a correct operation, and a reward value is given to such an operation

[0083]

[0084] During the driving process of the unmanned boat, in addition to optimizing the driving behavior of the unmanned boat by gradually approaching the navigation target point, the navigation direction of the unmanned boat must also be optimized. The navigation angle of the unmanned boat at this time is defined as , the actual angle between the position of the unmanned boat and the navigation target point is defined as , calculate the angle difference at this time :

[0085]

[0086] in, Represents the absolute value operation. When the angle difference is smaller, it means that the navigation direction of the unmanned boat is more correct. Different reward values ​​are given to it according to different angle differences. ;

[0087]

[0088] Based on the designed reward function module, the reward values ​​obtained by all reward function modules are added together to obtain the final reward value used in the reinforcement learning training process. :

[0089] .

[0090] 3. Construction of autonomous navigation model of unmanned boat

[0091] In the autonomous navigation model of the unmanned boat designed by the present invention, the use of navigation state sequence data to predict the navigation action of the unmanned boat can further enhance the feature extraction capability of the data, thereby improving the robustness of the unmanned boat during actual driving, and ensuring that the unmanned boat can maintain the normal navigation task under various water conditions. The self-attention mechanism designed in the model can automatically fuse the features extracted from the obstacle sequence data and the features extracted from the absolute distance sequence data. Thus, when the unmanned boat carries out the actual navigation task, it helps the unmanned boat to make the best action selection according to the specific driving conditions, further improving the intelligence level of the unmanned boat operation.

[0092] Sensor information collection: In the present invention, sensors are used to obtain the status information of the unmanned boat and the interaction information with the surrounding environment. It should be noted that the following information collection is completed in a reinforcement learning simulation environment. In order to successfully complete the navigation task in the present invention, it is necessary to ensure that there is no collision with obstacles during the driving of the unmanned boat. Therefore, the sonar radar sensor is first used to record the distance information between the unmanned boat and the surrounding obstacles. , using the inertial navigation system to record the position information of the unmanned boat in real time , and the location information of the navigation target , and record the real-time speed of the unmanned boat and heading direction .

[0093] Information processing module design: When the unmanned boat is undergoing reinforcement learning training, the information about the unmanned boat and surrounding obstacles during the environmental interaction process (i.e., the trial and error process of the unmanned boat) is retained. , unmanned boat location information , navigation target location information , and use this information as the input variable of the subsequent deep neural network model, and save the real-time navigation speed of the unmanned boat and sailing direction And the final reward value obtained during this learning process It should be noted that when collecting information, the sampling period is set to 1 second, that is, the unmanned boat will correspond to a new state every 1 second during a complete environmental interaction process, which means that the position of the unmanned boat and the distance to the surrounding obstacles, as well as its own navigation speed and heading direction will correspond to a new update every 1 second. Therefore, for a complete environmental interaction process, the surrounding obstacle information collected , unmanned boat location information , and the speed of the unmanned boat and the direction of the unmanned boat Each is a specific sequence;

[0094] Design of behavioral function module based on deep neural network: In the unmanned boat autonomous navigation system designed by the present invention, the unmanned boat can combine its own position information , surrounding obstacle information And navigation target location information Get the optimal navigation speed corresponding to the unmanned boat reaching the navigation target point at this time and sailing direction Compared with the traditional table lookup method, the present invention uses a deep neural network to predict the optimal sailing speed corresponding to each time point. and sailing direction .

[0095] (1) Network input data: To predict the optimal navigation speed of the unmanned boat at time T and sailing direction The input data includes the surrounding obstacle information sequence data of 10 time points from time T-9 to time T. , and the absolute distance sequence data from the unmanned boat to the navigation target point at 10 time points from T-9 to T .

[0096] (2) Model architecture: Due to the surrounding obstacle information data And the absolute distance data from the unmanned boat to the navigation target point They are all sequence data. In the constructed behavior function module, the data are first processed by two LSTM layers respectively. and Perform feature extraction, and then activate the features through the Softmax activation function to obtain intermediate features and ;

[0097]

[0098]

[0099] in, represent Operation function, represent Layer operation functions;

[0100] When performing actual navigation operations, the absolute distance to the final navigation target point should be gradually reduced. However, when encountering obstacles, distance overhead will inevitably be incurred to avoid collision. Therefore, in order to ensure the safety of the unmanned boat's autonomous navigation mission and shorten the time of the entire navigation mission as much as possible, a self-attention mechanism is designed to and Perform automatic fusion to obtain fusion features , so as to be more appropriate to specific navigation tasks and improve the intelligence level of unmanned boats.

[0101]

[0102] Then, the fusion features are transformed through a fully connected layer, a BN normalization layer, and a Relu activation function. After further processing, the optimal sailing speed at this moment is obtained and sailing direction ;

[0103]

[0104] in, represents the Relu activation function, Represents the BN normalization layer operation function, Represents the fully connected layer operation function;

[0105] Based on the designed information perception module, the information collected each time the unmanned boat interacts with the environment is saved in the state information database, and the data in the state information database is used to train the behavior function based on the deep neural network to gradually optimize the behavior of the unmanned boat and improve its success rate in performing unmanned autonomous navigation tasks.

[0106] 4. Model Training

[0107] In the field of autonomous navigation of unmanned boats, the unmanned boat autonomous navigation reinforcement learning training method that integrates multi-stage training mechanism and supervised learning strategy has brought significant advantages. First of all, through phased training, this method can gradually refine the learning process of the unmanned boat, from mastering the basic navigation rules to gradually improving the ability to cope with complex environments, effectively avoiding the problems of unclear model learning direction and low efficiency that may be caused by traditional random sample training. This not only greatly shortens the training cycle, but also ensures that the model can obtain targeted optimization at each stage, thereby accelerating the overall learning process. This training mechanism can greatly improve the intelligence level of unmanned boats, enabling them to make smarter and more efficient decisions when facing complex and changeable marine environments. Thereby, the normal implementation of navigation tasks can be guaranteed in various marine environments. The flow chart is as follows Figure 4 shown.

[0108] 1. Initial model training:

[0109] Based on the established unmanned boat autonomous navigation model, in order for the unmanned boat to gradually learn how to complete navigation tasks autonomously, it is necessary to train its behavior function based on deep neural networks to gradually improve the correctness of the unmanned boat operation. In the early stage of model training, due to the low intelligence level of the unmanned boat, the process of its interactive learning with the environment to accumulate data should be relatively short. Because the intelligence level of the unmanned boat is relatively low at this time, the correctness of its operation is also relatively low. Therefore, after accumulating a small amount of data, the operation of the unmanned boat should be trained to improve the intelligence level of its operation. In the present invention, the initial number of interactions between the unmanned boat and the surrounding environment is set to 20 times (that is, the unmanned boat carries out 20 autonomous navigation attempts), and the overall data of these 20 navigation processes are recorded according to the described data recording method. , including the unmanned boat data recorded during each navigation That is, the state information of the unmanned boat at each time state includes its own position information , and surrounding obstacle information And navigation target location information and motion information including real-time speed and sailing direction ;

[0110]

[0111]

[0112] (2) Based on the established reward function, calculate the total reward function value of each process in the above process in turn, and only retain the data recorded in the navigation process with the top 5 total reward function values, and exclude other data. Then use the saved data to train the established deep neural network (behavior function module), and use the Adam optimizer to optimize the model parameters to improve the performance of the unmanned boat autonomous navigation reinforcement learning model;

[0113] (3) Based on the optimized reinforcement learning model for autonomous navigation of the unmanned boat, the unmanned boat interacts with the environment again and repeats the above process. The number of iterations is set to 5 times. At this time, a reinforcement learning model with a higher degree of intelligence is obtained.

[0114] 2. Model reinforcement training:

[0115] Based on the fact that the intelligence of the reinforcement learning model has been improved at this time, its navigation data contains more correct and efficient operations. At this time, in order to further improve the performance of the model, compared with the initial training, when collecting data, 20 groups of unmanned boat navigation data are collected each time, and the number of collected data groups is changed to 50 groups. And still only the unmanned boat navigation data with the top 5 total reward function values ​​are retained, and the retained data is used to train the reinforcement learning model and optimize its model parameters;

[0116] Repeat the above process until the navigation is successfully completed more than 49 times in each data process, that is, the navigation success rate exceeds It should be noted that the basis for successful navigation is still the real-time position of the unmanned boat. and the location of the navigation target point The absolute distance If it is less than 0.5m, the navigation is considered to be completed successfully;

[0117]

[0118] Respectively represent the real-time position of the unmanned boat coordinate, Respectively represent the location of the unmanned boat navigation target point coordinate;

[0119] Model optimization training: In the actual navigation task, on the basis of successfully completing the navigation task, the navigation time should be shortened as much as possible (this means that the overall navigation speed of the unmanned boat is faster during the navigation process, and the distance it travels is relatively short). Therefore, on the basis of the above training, in order to further improve the intelligence level of the unmanned boat during autonomous navigation and reduce the time spent on its navigation, 20 sets of navigation data of the unmanned boat are still collected. However, compared with the previous method of extracting data using the reward function value, this time, the 5 sets of navigation data with the least overall navigation time among the 20 sets of heading data are selected to optimize the reinforcement learning model again to further improve the intelligence level of its operation. The number of iterations of this training is set to 8 times.

[0120] 5. Practical Implementation and Deployment of Autonomous Navigation of Unmanned Boats

[0121] Based on the designed unmanned boat autonomous navigation model and the described multi-stage reinforcement learning supervised training strategy, an optimized unmanned boat autonomous navigation model is finally obtained. This optimized unmanned boat autonomous navigation model is deployed to the unmanned boat to help the unmanned boat realize intelligent and automated navigation. The specific implementation steps are as follows: Figure 5 As shown, including:

[0122] Hardware deployment: Sonar radar and an inertial navigation system based on gyroscopes and accelerometers are deployed on the surface of the unmanned boat to collect information about surrounding obstacles and its own location in real time while the unmanned boat is performing navigation tasks. During model training, the state of the sampling is in a cycle of 1s, so when the navigation task is actually performed, the state interval is also set to 1s;

[0123] Software deployment: deploy the trained autonomous navigation model of the unmanned boat to the terminal of the unmanned boat, and send the information collected by its external hardware to the automatic navigation model;

[0124] Autonomous navigation of unmanned boat: set the target position information, and based on the autonomous navigation model of the unmanned boat, obtain the most correct heading direction and navigation speed of the unmanned boat at each state moment (update the navigation status of the unmanned boat every second), and send this information to the control system of the unmanned boat to adjust the real-time heading direction and navigation speed of the unmanned boat, and finally successfully complete the navigation task.

[0125] Experimental results:

[0126] In order to verify the effectiveness of the method proposed in this paper, the following comparative experiment is designed: In the comparative experiment, the navigation error is used to measure the accuracy of the algorithm navigation. In this experiment, the calculation method of the navigation error is as follows: After the navigation is completed, the position point of the unmanned boat is defined as , and the navigation target point The straight-line distance is defined as , its navigation error Comparative experiments of three methods including the method proposed in this invention were carried out at five different navigation distances of 100m, 300m, 500m, 700m and 1000m. The experimental results are as follows: Figure 6 The results show that the method proposed by the present invention has a small navigation error at each navigation distance, which indicates that the navigation accuracy of this method is high. Not only is the navigation error small, Figure 7 As shown, this method has the least actual navigation mileage and the least time consumed in the overall navigation process, which proves that the method proposed in the present invention not only presents a good level in navigation accuracy but also in navigation efficiency.

[0127] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0128] Although the above describes the specific implementation methods of the present invention, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. An autonomous navigation method for an unmanned boat based on reinforcement learning, characterized in that: The process includes: Deploy the trained autonomous navigation model of the unmanned boat to the terminal of the unmanned boat; Send the location information of the navigation target to the autonomous navigation model of the unmanned boat to carry out the autonomous navigation task; Based on the sonar radar and inertial navigation system deployed on the surface of the unmanned boat, the surrounding obstacle information and its own status information are obtained in real time during the unmanned boat's navigation mission. The collection cycle is consistent with the data collection cycle during model training. The collected data is input into the autonomous navigation model of the unmanned boat, and the model outputs the correct heading direction and navigation speed of the unmanned boat at each state moment; The model output results are sent to the control system of the unmanned boat to adjust the real-time heading direction and navigation speed of the unmanned boat to complete the navigation task; The unmanned boat autonomous navigation model is trained based on an underwater simulation environment, and a reward function integrating multiple reward mechanisms is designed, including sparse rewards, penalty negative rewards, and dense rewards; the sparse rewards are used to guide the unmanned boat to complete the navigation task, the penalty negative rewards are used to guide the unmanned boat to avoid collision with surrounding obstacles, and the dense rewards are used to guide the unmanned boat to approach the target and determine the real-time heading in real time; the unmanned boat autonomous navigation model is based on deep Q learning, and based on the acquired position information of itself , surrounding obstacle information And navigation target location information , output the optimal navigation speed corresponding to the unmanned boat reaching the navigation target point at this time and sailing direction ; The sparse reward is specifically that when the unmanned boat tries various navigation methods to try to approach the navigation target point, if the real-time position of the unmanned boat is and the location of the navigation target point The absolute distance If the distance is less than 0.5m, the unmanned boat is considered to have successfully reached the target point, and a reward value is given to this navigation method. For navigation methods that fail to successfully reach the target point, no reward value is given. The penalty negative reward is specifically that during the process of reinforcement learning training, when the unmanned boat collides with surrounding obstacles, a penalty is given to this navigation mode; when the distance between the unmanned boat and the surrounding obstacles is less than 0.01m, it is considered that the unmanned boat and the surrounding obstacles have collided, and when the distance with the surrounding obstacles is greater than 0.01m, it is considered that no collision has occurred; The intensive reward is to record the position of the unmanned boat in the previous state during the training process of reinforcement learning. , the position of the unmanned boat after the state is updated, that is, after the unmanned boat completes one step of operation, is recorded as , and combined with the location of the navigation target point Calculate the distance from the unmanned boat to the navigation target point in the previous state And the distance from the unmanned boat to the navigation target point after the status is updated ; The intensive reward is to record the position of the unmanned boat in the previous state during the training process of reinforcement learning. , the position of the unmanned boat after the state is updated, that is, after the unmanned boat completes one step of operation, is recorded as , and combined with the location of the navigation target point Calculate the distance from the unmanned boat to the navigation target point in the previous state And the distance from the unmanned boat to the navigation target point after the status is updated ; When the unmanned boat is in the middle of the driving process, if its The value is less than If the actual position of the unmanned boat is closer to the final navigation target point, the driving operation just performed by the unmanned boat is considered to be a correct operation, and a reward value is given to such an operation. ; At the same time, during the driving process of the unmanned boat, in addition to optimizing the driving behavior of the unmanned boat by gradually approaching the navigation target point, the navigation direction of the unmanned boat should also be optimized. The navigation angle of the unmanned boat at this time is defined as , the actual angle between the position of the unmanned boat and the navigation target point is defined as , calculate the angle difference at this time; When the angle difference is smaller, it means that the navigation direction of the unmanned boat is more correct, and different reward values ​​are given to it according to different angle differences. ; The unmanned boat autonomous navigation model includes an information processing module and a behavior function module; the information processing module is used for input data processing during training, and the behavior function module first processes the data through two LSTM layers respectively. and Perform feature extraction, and then activate the features through the Softmax activation function to obtain intermediate features and ; in, represent Operation function, represent Layer operation functions; Then, the self-attention mechanism is used to and Perform automatic fusion to obtain fusion features , to be more relevant to specific navigation tasks: Then, the fusion features are transformed through a fully connected layer, a BN normalization layer, and a Relu activation function. After further processing, the optimal sailing speed at this moment is obtained and sailing direction .

2. The method for autonomous navigation of an unmanned boat based on reinforcement learning as claimed in claim 1, characterized in that: The unmanned boat autonomous navigation model is trained in a simulation environment; an unmanned boat model is created in the Gazabo software, and the features of the unmanned boat simulation model are calibrated with reference to the shape of the real unmanned boat, and the mass, hull density, center of gravity, and material parameters of the unmanned boat simulation model are designed with reference to the corresponding parameters of the real unmanned boat; at the same time, a simulated ocean environment is created using the Gazabo software, and underwater obstacles are added to the simulated ocean environment, including shipwrecks, reefs, and schools of fish, and ocean currents, turbulence, and submarine undercurrents are added to simulate the impact of water flow on the driving of the unmanned boat in real scenarios; in the simulation environment, a sonar sensor simulation model is added to the unmanned boat simulation model to collect distance information from surrounding obstacles to the unmanned boat in the simulation environment , and add an inertial navigation simulation system consisting of a gyroscope and accelerometer to obtain real-time unmanned boat position information And the navigation target point location information , and combined with the real-time location information of the unmanned boat And navigation target point location information The absolute distance information of the unmanned boat relative to the navigation target point is calculated.

3. The method for autonomous navigation of an unmanned boat based on reinforcement learning as claimed in claim 1, characterized in that: The sparse reward calculation formula is: in, Represents the sparse module reward value. When the unmanned boat successfully reaches the target navigation point, its value is set to 20. If it fails to reach the target navigation point, its value is 0. Respectively represent the real-time position of the unmanned boat coordinate, Respectively represent the location of the unmanned boat navigation target point coordinate.

4. The unmanned boat autonomous navigation method based on reinforcement learning as claimed in claim 1, characterized in that: The penalty negative reward calculation formula is: in, represents the penalty value, Represents the distance between the unmanned boat and surrounding obstacles, which is collected by sonar radar sensors.

5. The method for autonomous navigation of an unmanned boat based on reinforcement learning as claimed in claim 1, characterized in that: In the dense reward, the location of the navigation target point is combined Calculate the distance from the unmanned boat to the navigation target point in the previous state And the distance from the unmanned boat to the navigation target point after the status is updated , the calculation formula is: They respectively represent the status update, that is, the position of the unmanned boat after the unmanned boat completes an operation. coordinate, Respectively represent the location of the unmanned boat navigation target point coordinate; The reward value The calculation formula is: The angle difference The calculation formula is: The reward value The calculation formula is: in, Represents the absolute value operation.

6. The method for autonomous navigation of an unmanned boat based on reinforcement learning as claimed in claim 1, characterized in that: The information processing module retains the information of the unmanned boat and surrounding obstacles during the environmental interaction process when the unmanned boat is undergoing reinforcement learning training. , unmanned boat location information , navigation target location information , and use this information as the input variable of the subsequent behavior function module, and save the real-time navigation speed of the unmanned boat and sailing direction And the final reward value obtained during this learning process ; When collecting information, the sampling period is set to 1s. The unmanned boat will correspond to a new state every 1s during a complete environmental interaction process. For a complete environmental interaction process, the surrounding obstacle information collected , unmanned boat location information , and the speed of the unmanned boat and the direction of the unmanned boat Each is a specific sequence.

7. The method for autonomous navigation of an unmanned boat based on reinforcement learning as claimed in claim 1, characterized in that: Integrate multi-stage training mechanisms and supervised learning strategies during training, including preliminary training, enhanced training, and optimized training; The preliminary training is to record the overall data of 20 navigation processes , conduct training on the operation of unmanned boats; The reinforcement training changes the number of collected data groups to 50 groups, and still only retains the unmanned boat navigation data with the top 5 total reward function values, and uses the retained data to train the reinforcement learning model and optimize its model parameters; Repeat the iteration until the navigation success rate exceeds ; The basis for successful navigation is the real-time position of the unmanned boat. and the location of the navigation target point The absolute distance If it is less than 0.5m, the navigation is considered to be completed successfully; Respectively represent the real-time position of the unmanned boat coordinate, Respectively represent the location of the unmanned boat navigation target point coordinate; The optimization training ensures that the unmanned boat has a faster overall navigation speed and a shorter travel distance during the navigation process. On the basis of the intensive training, 20 sets of navigation data of the unmanned boat are still collected. At this time, the 5 sets of navigation data with the shortest overall navigation time among the 20 sets of heading data are selected to carry out optimization training on the reinforcement learning model again, and the number of training iterations is set to 8 times.

8. An unmanned boat autonomous navigation system based on reinforcement learning, characterized in that: The method for autonomous navigation of an unmanned boat as claimed in any one of claims 1 to 7 is applied; comprising an information perception module, an unmanned boat control system, a memory and a power supply module; The information perception module includes a sonar radar sensor, a gyroscope and an acceleration sensor, which are used to collect surrounding obstacle information and the navigation status information of the unmanned boat in real time; the collected information is then sent to the unmanned boat autonomous navigation model for analysis and processing, and the output result is sent to the unmanned boat control system to complete the real-time adjustment of the navigation direction and speed of the unmanned boat; They are all powered by a unified power supply module, and the navigation status information is saved in real time using the memory.

Images