Intelligent obstacle avoidance method for unmanned boat based on end-to-end deep reinforcement learning of radar images
The intelligent obstacle avoidance method for unmanned boats employs end-to-end deep reinforcement learning of radar images to enhance decision-making in complex environments, addressing inefficiencies in existing methods by utilizing a deep convolutional neural network for rapid and effective obstacle avoidance.
Patent Information
- Application Number
- CN202210906026.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-07-29
- Publication Date
- 2024-12-27
- Estimated Expiration
- 2042-07-29
AI Technical Summary
Existing obstacle avoidance methods for unmanned boats are inadequate in complex environments, suffer from high computational complexity, and struggle to meet real-time needs for high-speed navigation, leading to inefficient and ineffective obstacle avoidance.
An intelligent obstacle avoidance method using end-to-end deep reinforcement learning of radar images, which constructs a mathematical model of an under-actuated unmanned boat, incorporates a deep convolutional neural network with multi-feature scale fusion, and utilizes a reward mechanism to enable efficient decision-making in complex scenarios.
The method enables rapid and efficient obstacle avoidance by leveraging deep neural networks for comprehensive situation understanding and decision-making, reducing time complexity and improving obstacle avoidance efficiency in complex environments.
Smart Images
Figure CN115167447B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of unmanned boats, and in particular relates to an intelligent obstacle avoidance method for an unmanned boat based on end-to-end deep reinforcement learning of radar images. Background Art
[0002] At present, as a small surface mission platform, Unmanned Surface Vehicle (USV) has the advantages of high speed, intelligence, flexibility, concealment, low cost, and no casualties, and has important military and civilian value. Driven by wireless communication technology and navigation control technology, USV's path tracking, formation coordination and other technologies have made important breakthroughs. The autonomous obstacle avoidance capability of USV is the key to ensuring its safe navigation and smooth execution of tasks in complex marine environments. According to the task level, the obstacle avoidance problem can be divided into global path planning obstacle avoidance and local obstacle avoidance. Global path planning obstacle avoidance is to generate a continuous path to avoid obstacles according to task requirements when the obstacle map information of the sea area is known; local obstacle avoidance is usually based on the situation information obtained by the sensor in real time. When it is judged that there is a risk of collision, an avoidance path is temporarily generated, and the original navigation state is restored after the threat disappears.
[0003] Traditional local obstacle avoidance algorithms can be roughly divided into two categories based on technical routes: one is the geometric rule method, which simplifies obstacles and ships into convex bodies and considers kinematic constraints to calculate the upper and lower boundaries of feasible trajectories. Typical algorithms include speed obstacle method, artificial potential field method and dynamic virtual ship method. However, when encountering complex environments, the above methods are prone to fall into local optimality or no solution; the other is the path search method, which searches for a reasonable path through optimization, such as the dynamic window method and graph search algorithms such as A* algorithm and rapidly expanding random tree. The path search method is suitable for avoiding static or relatively low-speed obstacles, and has high computational complexity, making it difficult to meet the needs of real-time obstacle avoidance for high-speed unmanned boats. With the advancement of deep reinforcement learning (DRL) technology, a large number of research results have emerged in the fields of game AI, autonomous driving and robot control. This type of algorithm interacts a lot with the environment, learns and optimizes strategies through trial and error, has strong adaptive ability to complex environments, and opens up a new way to solve the obstacle avoidance problem. In response to the local obstacle avoidance problem of USV, prior art 1 uses deep Q learning (deep Q network, DQN) to achieve USV obstacle avoidance in fixed scenarios. Prior art 2 uses the DDPG algorithm to deal with the obstacle avoidance problem between USV formations, but the state input dimension of the deep network is related to the number of USV formations, and does not involve strategies for dealing with non-cooperative targets. Prior art 3 studies the dynamic obstacle situation, using map raster images as the state input of the deep network to enable the USV to successfully cross the obstacle area, but does not consider the constraints of the navigation path. Prior art 4 introduces a semi-Markav decision model and designs a navigation strategy switcher based on DQN, which can switch between the three navigation modes of left obstacle avoidance, right obstacle avoidance and keeping a straight line according to the state of the USV's surrounding environment. The navigation test proves the effectiveness of obstacle avoidance, but the obstacle avoidance control process and implementation mechanism of this method are relatively complex.
[0004] Through the above analysis, the problems and defects of the existing technology are as follows: the existing obstacle avoidance method cannot be applied to complex environments, the calculation complexity is high, it is difficult to meet the needs of real-time obstacle avoidance of high-speed unmanned boats, and it cannot effectively avoid obstacles, and the obstacle avoidance efficiency and obstacle avoidance control are poor. Summary of the invention
[0005] In view of the problems existing in the prior art, the present invention provides an intelligent obstacle avoidance method for an unmanned boat based on end-to-end deep reinforcement learning of radar images.
[0006] The present invention is implemented as follows: an unmanned boat intelligent obstacle avoidance method based on end-to-end deep reinforcement learning of radar images, the unmanned boat intelligent obstacle avoidance method based on end-to-end deep reinforcement learning of radar images comprises:
[0007] A mathematical model of an under-actuated unmanned boat is constructed. The relative position relationship between the obstacle and the unmanned boat and the information of the nearest encounter point are used to construct an obstacle track layer, a collision threat layer or other high-dimensional state features. A deep convolutional neural network with multi-feature scale fusion is constructed and trained in combination with the navigation state, path error or other low-dimensional states. The trained deep convolutional neural network with multi-feature scale fusion is used to perform intelligent obstacle avoidance for the unmanned boat.
[0008] Furthermore, the unmanned boat intelligent obstacle avoidance method based on end-to-end deep reinforcement learning of radar images includes the following steps:
[0009] Step 1: Acquire an unmanned boat radar echo image, and perform rasterization processing on the unmanned boat radar echo image and convert it into a grayscale image;
[0010] Step 2: construct an obstacle track layer and a collision threat layer to represent the track information of the obstacle relative to the USV and the predicted position and threat level of the collision;
[0011] Step 3: Construct low-dimensional state features; construct a reward function for real-time evaluation of the current state of the USV; construct and train a deep convolutional neural network for multi-feature scale fusion;
[0012] Step 4: Use the trained multi-feature scale fusion deep convolutional neural network to perform intelligent obstacle avoidance for the unmanned boat.
[0013] Further, the step 2 of constructing the obstacle track layer and the collision threat layer includes:
[0014] (1) Constructing the obstacle track layer: Taking the pixel point at the center of the layer as the origin, the horizontal rightward direction as the x-axis, and the vertical upward direction as the y-axis, establish the layer coordinate system {l}, determine the mapping matrix R(κ)=diag(κ,κ) from the hull coordinate system {b} to the layer coordinate system {l}, and determine the position layer representing the position of the obstacle in the layer coordinate system {l} According to the mapping relationship, the coordinates and radius of the obstacle position features at time t in the layer coordinate system {l} are determined as follows: and κr i , let the location layer The intensity value of all obstacle coverage areas is 255, and the obstacle track layer is obtained. The recursive expression is:
[0015]
[0016] Among them, κ represents the mapping coefficient, which is used to characterize the number of pixels in the layer coordinate system {l} corresponding to the unit distance of the hull coordinate system {b}; Represents the obstacle track layer; Represents the location layer; γs Represents the position attenuation coefficient; the clip function is used to limit the pixel intensity in the layer to the interval [0,255]; represents the position of the ith obstacle in the ship coordinate system {b}; r i represents the threat radius of the ith obstacle in the ship coordinate system {b}, i = 1, 2, …, n, and n represents the number of obstacles located in the situation awareness area observed by the USV radar at time t;
[0017] (2) By calculating the encounter position relationship and time margin between each obstacle and USV in the situation awareness area, a collision threat layer is constructed using the collision threat layer construction method based on the closest encounter distance DCPA and the closest encounter time TCPA:
[0018] Determine the collision threat layer L c is the relative position relationship of all obstacles at the closest encounter time in the layer coordinate system {l}, and the relative position of obstacles and USV at the closest encounter time According to the mapping relationship, the coordinates and radius of the relative encounter position feature of the i-th obstacle in the layer coordinate system {l} are and κr i , collision threat layer L c The intensity value of the area covered by the i-th encounter point in is:
[0019]
[0020] Among them, d i represents the DCPA of the ith obstacle relative to the USV in the ship coordinate system {b}; t i represents the TCPA of the ith obstacle relative to the USV in the ship coordinate system {b}; b i represents the CPA side angle at the closest encounter time; γ c represents the threat attenuation coefficient, when t i ≥0, the intensity value is closer to the maximum value 255; when t i When <0, the intensity value is 0; the collision threat layer L c The encounter point in the image is used to characterize the threat level of collision between the obstacle and the USV. When the encounter point is close to the center of the layer, it means that if the USV continues to maintain the original course, a collision will occur. The larger the intensity value of the encounter point, the shorter the expected time of collision and the more urgent the threat. When there is no encounter point with a high intensity value in the center of the layer, it is judged that the collision threat of the USV is low.
[0021] Furthermore, the low-dimensional state feature vector in step 3 is:
[0022]
[0023] Among them, S l represents the low-dimensional state feature vector; a t-1 Indicates the action output at the previous moment; represents the yaw angular velocity; the low-dimensional state feature vector includes the LOS guidance error, and the LOS guidance error includes the distance error information y e , and heading error information χ d ,
[0024] The reward function in step 3 is the position error reward r d , heading error reward r ψ and the heading stability reward r s composition:
[0025] r=w d r d +w ψ r ψ +w s r s ;
[0026]
[0027] The reward is described in the form of a negative exponential function, and the range of each sub-reward is limited to (-1, 1], k1, k2, k3 represent the adjustment coefficients of each sub-reward, σ represents the standard deviation of the USV heading over a period of time; r s represents the heading stability reward; w d 、w ψ and w s represents the weight coefficient;
[0028] When the USV goes out of bounds or collides, the mission is terminated and a termination reward is generated. Depending on the situation, the rewards obtained are as follows: When the USV exceeds the mission boundary, r = r o , r o Represents the out-of-bounds penalty reward, which is negative; when the USV collides with an obstacle, r = r c , r c Represents the collision penalty reward, which is negative; when the USV reaches the obstacle avoidance end area, k4 represents the adjustment coefficient, r e Represents the task completion reward, which is positive. The closer the arrival at the end zone is to the reference path, the higher the termination reward is.
[0029] Furthermore, the multi-feature scale fusion deep convolutional neural network in step 3 includes:
[0030] Input layer, used to input obstacle track layer L s With the collision threat layer Lc The composed dual channel image;
[0031] The convolution layer-pooling layer is used to compress the size of the feature tensor, expand the number of channels, and flatten it into a one-dimensional feature vector through multiple convolution-pooling processes;
[0032] The flattening layer is used to concatenate the low-dimensional state features and the one-dimensional feature vector to obtain a common feature vector;
[0033] The first strategy network layer is used to obtain the corresponding output using two independent fully connected networks;
[0034] The second strategy network layer is used to obtain the corresponding output using two independent fully connected networks;
[0035] The first value network layer is used to obtain the corresponding output using two independent fully connected networks;
[0036] The second value network layer is used to obtain the corresponding output using two independent fully connected networks;
[0037] At the same time, the multi-feature scale fusion deep convolutional neural network is extracted from the action set Select the action with the highest probability from The actions included are desired heading -10°, desired heading +10°, and desired heading unchanged.
[0038] Furthermore, the training of a deep convolutional neural network for multi-feature scale fusion includes:
[0039] Use TensorFlow as a deep learning training framework and input the high-dimensional state L of the network s and L c Use normalization processing; use standardization to process the low-dimensional state feature vector S l , using a proximal strategy optimization algorithm to train the multi-feature scale fusion deep convolutional neural network;
[0040] At the same time, during training, randomly select from the action set according to the probability distribution Select an action from the drop-down list to update the desired heading.
[0041] Another object of the present invention is to provide an unmanned boat intelligent obstacle avoidance system based on end-to-end deep reinforcement learning of radar images, which implements the unmanned boat intelligent obstacle avoidance method based on end-to-end deep reinforcement learning of radar images. The unmanned boat intelligent obstacle avoidance system based on end-to-end deep reinforcement learning of radar images comprises:
[0042] A perception module, which is used to comprehensively summarize and process reference path information, obstacle information and USV status information, and convert the reference path information, obstacle information and USV status information into high-dimensional image information and low-dimensional status information;
[0043] A decision module, used to output a desired heading based on state information using a deep neural network;
[0044] The execution module is used to output the control amount of the USV throttle and steering command according to the current USV navigation parameters and the expected heading, and update the state information according to the USV dynamics and kinematics model.
[0045] Another object of the present invention is to provide a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the unmanned boat intelligent obstacle avoidance method based on end-to-end deep reinforcement learning of radar images.
[0046] Another object of the present invention is to provide a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the unmanned boat intelligent obstacle avoidance method based on end-to-end deep reinforcement learning of radar images.
[0047] Another object of the present invention is to provide an information data processing terminal, which is used to implement the unmanned boat intelligent obstacle avoidance system based on end-to-end deep reinforcement learning of radar images.
[0048] In combination with the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions to be protected by the present invention are as follows:
[0049] First, in view of the technical problems existing in the above-mentioned prior art and the difficulty of solving the problems, the technical solutions to be protected by the present invention and the results and data during the research and development process are closely combined to analyze in detail and deeply how the technical solutions of the present invention solve the technical problems, and some creative technical effects brought about after solving the problems. The specific description is as follows:
[0050] The present invention improves the obstacle avoidance strategy of the USV in continuous interaction with the environment by establishing a mathematical model of the underactuated unmanned vehicle, a dynamic obstacle generation model and a reward mechanism, so that the USV can make effective decisions in complex scenarios and complete the obstacle avoidance task more quickly and efficiently.
[0051] The present invention provides a model-free end-to-end path planning algorithm, proposes two high-dimensional state features: a layer for representing past and present obstacle tracks and a layer for representing predicted future collision threats. The designed deep convolutional neural network with multi-feature scale fusion can directly output decisions, simplifying the rule-based decision-making process of traditional obstacle avoidance methods. The obstacle avoidance technical solution of the present invention fully exploits the situation understanding, perception and decision-making capabilities of deep neural networks in complex obstacle avoidance scenarios. Its time calculation complexity is independent of the number and distribution complexity of obstacles, and has engineering application value.
[0052] Second, considering the technical solution as a whole or from the perspective of the product, the technical effects and advantages of the technical solution to be protected by the present invention are described in detail as follows:
[0053] The deep reinforcement learning method proposed in the present invention can effectively learn obstacle avoidance strategies during the training phase, has the ability to understand obstacle environment information and make comprehensive decisions in complex situations, and can complete obstacle avoidance tasks quickly and efficiently.
[0054] Third, as auxiliary evidence of the inventiveness of the claims of the present invention, it is also reflected in the fact that the technical solution of the present invention fills the technical gap in the industry at home and abroad:
[0055] This invention proposes for the first time an end-to-end unmanned boat reinforcement learning obstacle avoidance method. Unlike traditional obstacle avoidance methods, this solution does not need to introduce complex obstacle avoidance rules, but instead constructs an information-rich situation image by secondary encoding the raw data obtained by sensors such as navigation radar / lidar. The present invention fully combines the advantages of convolutional neural networks in image feature extraction and deep reinforcement learning in sequence decision-making problems. It uses images and unmanned boat state vectors as the original input of the neural network, directly outputs the desired heading decision, and has a simple control process. This end-to-end obstacle avoidance control solution fills the technical gap in related fields. BRIEF DESCRIPTION OF THE DRAWINGS
[0056] Figure 1 It is a flow chart of an unmanned boat intelligent obstacle avoidance method based on end-to-end deep reinforcement learning of radar images provided by an embodiment of the present invention;
[0057] Figure 2 is a schematic diagram of an obstacle avoidance scenario at t=55 provided by an embodiment of the present invention;
[0058] Figure 3 is a schematic diagram of an obstacle avoidance scenario at t=80 provided by an embodiment of the present invention;
[0059] Figure 4 is a schematic diagram of an obstacle avoidance scenario at t=105 provided by an embodiment of the present invention;
[0060] Figure 5is a schematic diagram of an obstacle avoidance scenario at t=130 provided by an embodiment of the present invention;
[0061] Figure 6 is a position layer image at t=55 provided by an embodiment of the present invention;
[0062] Figure 7 is a position layer image at t=80 provided by an embodiment of the present invention;
[0063] Figure 8 is a position layer image at t=105 provided by an embodiment of the present invention;
[0064] Fig. 9 is a position layer image at t=130 provided by an embodiment of the present invention;
[0065] Fig.10 is an obstacle track layer image at t=55 provided by an embodiment of the present invention;
[0066] Fig.11 is an obstacle track layer image at t=80 provided by an embodiment of the present invention;
[0067] Fig.12 is the obstacle track layer image at t=105 provided by an embodiment of the present invention;
[0068] Fig.13 It is the obstacle track at t=130 provided by the embodiment of the present invention; layer image;
[0069] Fig.14 is a collision threat layer image at t=55 provided by an embodiment of the present invention;
[0070] Fig.15 is a collision threat layer image at t=80 provided by an embodiment of the present invention;
[0071] Fig.16 is a collision threat layer image at t=105 provided by an embodiment of the present invention;
[0072] Fig.17 is a collision threat layer image at t=130 provided by an embodiment of the present invention;
[0073] Fig.18 It is a schematic diagram of the structure of a deep convolutional neural network with multi-feature scale fusion provided by an embodiment of the present invention;
[0074] Fig.19 is a diagram of a deployment scheme of an obstacle avoidance system provided by an embodiment of the present invention;
[0075] Fig. 20 1 is a diagram of multi-ship collision avoidance simulation results of the DRL method for scenario 1 provided by an embodiment of the present invention;
[0076] Fig.21 This is a diagram of multi-ship collision avoidance simulation results of the artificial potential field method in scenario 1 provided by an embodiment of the present invention;
[0077] Fig. 22 is a diagram of multi-ship collision avoidance simulation results of the DRL method for scenario 2 provided by an embodiment of the present invention;
[0078] Fig.23 It is a diagram of multi-ship collision avoidance simulation results of the artificial potential field method in scenario 2 provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0079] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.
[0080] 1. Explanatory Examples In order to enable those skilled in the art to fully understand how to implement the present invention, this section provides an illustrative example that expands and describes the technical solution of the claims.
[0081] like Figure 1 As shown, the unmanned boat intelligent obstacle avoidance method based on end-to-end deep reinforcement learning of radar images provided by an embodiment of the present invention includes the following steps:
[0082] S101, acquiring an unmanned boat radar echo image, and performing rasterization processing on the unmanned boat radar echo image and converting it into a grayscale image;
[0083] S102, constructing an obstacle track layer and a collision threat layer to respectively represent the track information of the obstacle relative to the USV and the predicted position and threat level of the collision;
[0084] S103, construct low-dimensional state features; construct a reward function for real-time evaluation of the current state of the USV; construct and train a deep convolutional neural network for multi-feature scale fusion;
[0085] S104, using the trained multi-feature scale fusion deep convolutional neural network to perform intelligent obstacle avoidance for the unmanned boat.
[0086] The unmanned boat intelligent obstacle avoidance method based on end-to-end deep reinforcement learning of radar images provided by an embodiment of the present invention includes:
[0087] Step 1: rasterize the radar echo image of the unmanned boat and convert it into a grayscale image of 117×117 pixels. The image grayscale is positively correlated with the target echo intensity.
[0088] Step 2: Construct an obstacle track layer and a collision threat layer to represent the track information of the obstacle relative to the USV and the predicted location and threat level of the collision. (For detailed steps of the obstacle track layer and the collision threat layer, see steps 3 and 4).
[0089] Step 3, construction of obstacle track layer:
[0090] Assume that the number of obstacles in the situational awareness area observed by the USV radar at time t is n, and the position and threat radius of the ith obstacle are defined in the ship coordinate system {b} as follows: and r i , i=1,2,…,n. The layer coordinate system {l} is established with the pixel point at the center of the layer as the origin, the horizontal rightward as the x-axis, and the vertical upward as the y-axis. The mapping matrix R(κ)=diag(κ,κ) of the hull coordinate system {b} to the layer coordinate system {l} is defined, where κ is the mapping coefficient, which indicates the number of pixels in the layer coordinate system {l} corresponding to the unit distance of the hull coordinate system {b}. In the present invention, the value of κ is 1 / 8 pixel / meter.
[0091] Define location layer Represents the position of the obstacle in the layer coordinate system {l}. According to the mapping relationship, the coordinates and radius of the obstacle's position feature at time t in {l} are respectively and κr i , let the location layer The intensity value of all obstacle coverage areas is 255, and the obstacle track layer is defined as Its recursive expression is:
[0092]
[0093] In the formula, γ s is the position attenuation coefficient. The function of the clip function is to limit the pixel intensity in the layer to the interval [0,255]. The purpose of this formula is to generate the track characteristics of obstacles relative to the USV in the track layer.
[0094] Figures 2 to 5 The USV obstacle avoidance scenario at different times in the geographic coordinate system is shown. This scenario simulates the USV driving along the reference path and crossing four dynamic obstacles. The circle in the figure represents a dynamic obstacle, and the dotted rectangular box represents the USV's situational awareness area; Figures 6 to 9 The position layer image at the corresponding moment is displayed, which is consistent with the obstacle situation in the situation awareness area; Figures 10 to 13 The obstacle track layer image is displayed, and it can be seen that the intensity of the obstacle's historical track gradually decreases with time, achieving a display effect similar to radar afterglow. This layer can characterize the position characteristics and historical track of the obstacle relative to the USV.
[0095] Step 4: Collision threat layer feature construction:
[0096] Obstacle track layer L s It can provide the past and present characteristics of obstacles. Correspondingly, the collision threat layer aims to provide predictions of future collision threats. This section proposes a collision threat layer construction method based on DCPA and the closest point of approach (TCPA) by calculating the encounter position relationship and time margin between each obstacle and USV in the situational awareness area.
[0097] Let the DCPA of the ith obstacle relative to the USV in the ship coordinate system {b} be d i , TCPA is t i , the CPA side angle at the closest encounter is b i . Define the collision threat layer L c is the relative position relationship of all obstacles at the closest encounter time in the layer coordinate system {l}, and the relative position of obstacles and USV at the closest encounter time According to the mapping relationship, the coordinates and radius of the relative encounter position feature of the i-th obstacle in the layer coordinate system {l} are and κr i , collision threat layer L c The intensity value of the area covered by the i-th encounter point in is:
[0098]
[0099] In the formula, γ c is the threat attenuation coefficient, when t i ≥0, the relative distance between USV and obstacle gradually decreases, so when t i The closer to time 0, the greater the collision threat, and the closer the intensity value is to the maximum value of 255; when t i When L is less than 0, the USV is far away from the obstacle and the threat is considered eliminated, so the intensity value is set to 0. Collision threat layer L c The encounter point in the layer can represent the threat level of collision between the obstacle and the USV. When the encounter point is close to the center of the layer, it means that if the USV continues to maintain the original course, a collision will occur; the larger the intensity value of the encounter point, the shorter the expected time of collision and the more urgent the threat. Therefore, when there is no encounter point with a high intensity value in the center of the layer, it can be considered that the collision threat of the USV is low.
[0100] Figures 14 to 17The collision threat layers corresponding to four moments in the obstacle avoidance scenario are shown. Since the USV keeps a stable heading throughout the entire process, the position of the encounter point in the layer remains unchanged, but the intensity gradually increases as the encounter time decreases. It can be observed that Fig.14 There is a single meeting point in Fig.15 and Fig.16 The strength value of the encounter point gradually increases, combined with Figures 10 to 13 It can be found that the obstacle is moving towards the starboard side of the USV and has already met it at t = 130. Fig.17 The point will disappear.
[0101] Step 5: Construct low-dimensional state features:
[0102] Although high-dimensional state features can represent obstacle situation information, the USV's own navigation state and reference mission path are not suitable for expression using two-dimensional images. Therefore, it is necessary to introduce low-dimensional state features as a supplement to high-dimensional state features.
[0103] The designed low-dimensional state contains the LOS guidance error and the action output a at the previous moment t-1 Yaw angular velocity The LOS guidance error includes the distance error information y e , and heading error information χ d ,
[0104] In summary, the low-dimensional state feature vector S l Designed for:
[0105] Step 6, reward setting:
[0106] At the same time, around the two goals of USV path tracking and real-time obstacle avoidance, a suitable reward function is designed to evaluate the current state of the USV in real time. The design of the reward function follows the following rules: the closer the USV position is to the reference path, the closer the heading is to the expected heading, and the more stable the heading is, the higher the reward value. During the navigation process of the USV, the designed reward function is composed of the position error reward r d , heading error reward r ψ and the heading stability reward r s composition:
[0107]
[0108] In formula (a), the reward is described in the form of a negative exponential function, and the range of each sub-reward is limited to (-1, 1], k1, k2, k3 are the adjustment coefficients of each sub-reward, and σ represents the standard deviation of the USV heading over a period of time. By introducing the heading stability reward r s, which can make the deep reinforcement learning controller tend to choose a continuous and stable heading control strategy during training, thereby accelerating convergence. The reward function expression during navigation is defined as:
[0109] r=w d r d +w ψ r ψ +w s r s (b)
[0110] In formula (b), w d 、w ψ and w s is the weight coefficient.
[0111] When the USV goes out of bounds or collides, the mission is terminated and a termination reward is generated. Depending on the situation, the reward is designed as follows: When the USV exceeds the mission boundary, r = r o , r o Represents the out-of-bounds penalty reward, which is negative; when the USV collides with an obstacle, r = r c , similarly, r c Represents the collision penalty reward, which is negative; when the USV reaches the obstacle avoidance end area, k4 is the adjustment coefficient, r e It is the task completion reward, which is positive. The closer the end zone is to the reference path, the higher the termination reward is.
[0112] Step 7, action space design:
[0113] At the beginning of each time step, we need to start from the action set Select an action to update the desired heading, set The actions included are the desired heading -10°, the desired heading +10°, and the desired heading unchanged. The deep reinforcement learning policy network generates the selection probabilities of the above three actions. During the network training phase, it randomly selects Select an action from the list to ensure that the action selection is random and exploratory; during the network deployment phase, Select the action with the highest probability.
[0114] Step 8: Build a deep network:
[0115] The invention designs a multi-feature scale fusion deep convolutional neural network, the structural diagram of which is shown in the figure Fig.18 As shown. Network input obstacle track layer L s With the collision threat layer L cAfter multiple convolutional-pooling layers, the size of the feature tensor is compressed from 117×117 to 3×3, and the number of channels is expanded to 64. It is then flattened into a one-dimensional feature vector. Finally, the low-dimensional state feature and the one-dimensional feature vector are concatenated to obtain a common feature vector. The detailed parameters of the network layer are shown in Table 1. The PPO network includes a policy network and a value network. The two networks share a common feature vector and are passed through two independent fully connected networks to obtain the corresponding output. Considering that the vector length (576) of the high-dimensional feature after multiple convolution-pooling dimensionality reduction operations is quite different from the length (6) of the low-dimensional state feature vector, the input of the two-layer fully connected network introduces low-dimensional state features to speed up the network convergence.
[0116] Table 1 PPO network parameters
[0117]
[0118] Step 9, training parameter setting:
[0119] The USV obstacle avoidance virtual environment is constructed based on the OpenAI gym framework. The training parameters are set as follows: the USV control period is 1s, the adjustment coefficients k1, k2, k3, and k4 are 1 / 100, 1 / 30, 5, and 1 / 50 respectively; the weight coefficient w d , w ψ , w s They are 0.4, 0.4, and 0.2 respectively; the reward value parameter r o , r c , r e They are -500, -1000, and 50 respectively; the number of obstacles is randomly selected between 1 and 5.
[0120] The PPO network training hyperparameter settings are shown in Table 2:
[0121] Table 2 Hyperparameter settings
[0122]
[0123] Step 10, network training:
[0124] Using TensorFlow as a deep learning training framework, during training, the high-dimensional state L of the network input s and L c Use normalization; low-dimensional state feature vector S l Using normalization, the training of the deep network is completed using the proximal policy optimization algorithm.
[0125] Step 11, deployment implementation:
[0126] The USV starts from a given starting point and travels along a reference path. When it senses a collision threat, it actively changes its course to avoid the obstacle threat. After the threat is eliminated, it resumes its original route. The proposed obstacle avoidance system deployment scheme based on deep reinforcement learning is shown in the figure below. Fig.19 shown.
[0127] The obstacle avoidance system provided by the embodiment of the present invention is composed of a perception module, a decision module and an execution module. The perception module is responsible for comprehensively summarizing and processing reference path information, obstacle information and USV status information, and converting them into high-dimensional image information and low-dimensional status information; the decision module is composed of a deep neural network, and outputs the desired heading according to the status information; the function of the execution module is to output the control amount of the USV throttle and steering command according to the current USV navigation parameters and the desired heading, and then update its status information according to the USV dynamics and kinematics model. The three modules are executed in a continuous loop to realize dynamic real-time obstacle avoidance of the USV.
[0128] 2. Application Examples: In order to prove the creativity and technical value of the technical solution of the present invention, this section provides application examples of the technical solution of the claims on specific products or related technologies.
[0129] The USV local obstacle avoidance algorithm based on deep reinforcement learning provided by the embodiment of the present invention is applied to the intelligent obstacle avoidance of unmanned boats. In actual engineering applications, the obstacle avoidance strategy network is trained on the server, and the weight of the strategy network is deployed on the NVIDIA Jetson AGX Xavier edge computing platform carried by the unmanned boat. In actual applications, the navigation radar echo is used as the information input of the obstacle, and the navigation radar range is set to 1000 meters. After testing on the lake, the obstacle avoidance solution can quickly and efficiently complete the obstacle avoidance task.
[0130] 3. Evidence of the effects of the embodiments. The embodiments of the present invention have achieved some positive effects during the development or use process, and indeed have great advantages over the prior art. The following content is described in conjunction with the data, charts, etc. of the test process.
[0131] The effectiveness of the USV local obstacle avoidance algorithm based on deep reinforcement learning provided by the embodiment of the present invention is verified through simulation experiments. By establishing a mathematical model of underactuated unmanned boats, a dynamic obstacle generation model and a reward mechanism, the USV obstacle avoidance strategy is improved in continuous interaction with the environment, so that it can make effective decisions in complex scenarios and complete obstacle avoidance tasks more quickly and efficiently. The present invention creatively designs a model-free end-to-end path planning algorithm, and proposes two high-dimensional state features: a layer of obstacle tracks that can characterize the past and present, and a layer of collision threats that characterizes predicted future events. The designed deep convolutional neural network with multi-feature scale fusion can directly output decisions, simplifying the rule-based decision-making process of traditional obstacle avoidance methods, and has engineering application potential.
[0132] To further verify the effectiveness of the DRL obstacle avoidance strategy, this method and the traditional method were used to simulate the encounter situation at sea. The present invention selected two representative obstacle avoidance scenarios, simulating the USV driving in a busy port and crossing a busy waterway, respectively, hereinafter referred to as Scenario 1 and Scenario 2. The traditional method uses the artificial potential field method to control the USV to avoid obstacles. It can be found that the DRL method proposed in the present invention can understand the motion state and trend of each obstacle in the environment and make decisions that meet long-term benefits. ( Fig. 20 , Fig. 22 ) This capability is due to the high-dimensional obstacle history track images and collision threat images input by the DRL network, which enables it to make decisions based on the comprehensive information of the past, present and future. However, traditional methods usually parameterize the obstacle situation when dealing with such problems. In the parameterization process, most of the original information is not effectively utilized ( Fig.21 , Fig.23 ). Therefore, the DRL method can have better performance in complex obstacle avoidance scenarios.
[0133] It should be noted that the embodiments of the present invention can be implemented by hardware, software, or a combination of software and hardware. The hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or dedicated design hardware. It can be understood by a person of ordinary skill in the art that the above-mentioned devices and methods can be implemented using computer executable instructions and / or contained in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the carrier medium. The device and its modules of the present invention can be implemented by hardware circuits such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, etc., or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc., can also be implemented by software executed by various types of processors, and can also be implemented by a combination of the above-mentioned hardware circuits and software, such as firmware.
[0134] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any modifications, equivalent substitutions and improvements made by any technician familiar with the technical field within the technical scope disclosed by the present invention and within the spirit and principle of the present invention should be covered by the protection scope of the present invention.
Claims
1. An intelligent obstacle avoidance method for unmanned boats based on end-to-end deep reinforcement learning of radar images, characterized in that: The unmanned boat intelligent obstacle avoidance method based on end-to-end deep reinforcement learning of radar images includes: Construct a mathematical model of an underactuated unmanned boat, and use the relative position relationship between the obstacle and the unmanned boat and the information of the nearest encounter point to construct an obstacle track layer, a collision threat layer or other high-dimensional state features; and combine the navigation state, path error or other low-dimensional states to construct and train a multi-feature scale fusion deep convolutional neural network; use the trained multi-feature scale fusion deep convolutional neural network to perform intelligent obstacle avoidance for the unmanned boat; The unmanned boat intelligent obstacle avoidance method based on end-to-end deep reinforcement learning of radar images comprises the following steps: Step 1: Acquire an unmanned boat radar echo image, and perform rasterization processing on the unmanned boat radar echo image and convert it into a grayscale image; Step 2: construct an obstacle track layer and a collision threat layer to represent the track information of the obstacle relative to the USV and the predicted position and threat level of the collision; Step 3: Construct low-dimensional state features; construct a reward function for real-time evaluation of the current state of the USV; construct and train a deep convolutional neural network for multi-feature scale fusion; Step 4: Use the trained multi-feature scale fusion deep convolutional neural network to perform intelligent obstacle avoidance for the unmanned boat; The step 2 of constructing the obstacle track layer and the collision threat layer includes: (1) Construct the obstacle track layer: Take the pixel point at the center of the layer as the origin, horizontally to the right as the x-axis, and vertically upward as the y-axis to establish the layer coordinate system {l}, determine the mapping matrix R(κ) = diag(κ,κ) of the hull coordinate system {b} to the layer coordinate system {l}, and the position layer Represents the position of the obstacle in the layer coordinate system {l}. According to the mapping relationship, the coordinates and radius of the obstacle position feature at time t in the layer coordinate system {l} are determined respectively. and κr i , let the location layer The intensity value of all obstacle coverage areas is 255, and the recursive expression of the obstacle track layer is: Among them, κ represents the mapping coefficient, which is used to characterize the number of pixels in the layer coordinate system {l} corresponding to the unit distance of the hull coordinate system {b}; Represents the obstacle track layer; Represents the location layer; γ s Represents the position attenuation coefficient; the clip function is used to limit the pixel intensity in the layer to the interval [0 ,255]in; represents the position of the ith obstacle in the ship coordinate system {b}; r i represents the threat radius of the ith obstacle in the ship coordinate system {b}, i = 1, 2, …, n, and n represents the number of obstacles located in the situation awareness area observed by the USV radar at time t; (2) By calculating the encounter position relationship and time margin between each obstacle and USV in the situation awareness area, a collision threat layer is constructed using the collision threat layer construction method based on DCPA and the closest encounter time TCPA: Determine the collision threat layer L c is the relative position relationship of all obstacles at the closest encounter time in the layer coordinate system {l}, and the relative position of obstacles and USV at the closest encounter time According to the mapping relationship, the coordinates and radius of the relative encounter position feature of the i-th obstacle in the layer coordinate system {l} are and κr i , collision threat layer L c The intensity value of the area covered by the i-th encounter point in is: Among them, d i represents the DCPA of the ith obstacle relative to the USV in the ship coordinate system {b}; t i represents the TCPA of the ith obstacle relative to the USV in the ship coordinate system {b}; b i represents the CPA side angle at the closest encounter time; γ c represents the threat attenuation coefficient, when t i ≥0, the intensity value is closer to the maximum value 255; when t i When <0, the intensity value is 0; the collision threat layer L c The encounter point in the image is used to characterize the threat level of collision between the obstacle and the USV. When the encounter point is close to the center of the layer, it means that if the USV continues to maintain the original course, a collision will occur. The larger the intensity value of the encounter point, the shorter the expected time of collision and the more urgent the threat. When there is no encounter point with a high intensity value in the center of the layer, it is judged that the collision threat of the USV is low.
2. The unmanned boat intelligent obstacle avoidance method based on end-to-end deep reinforcement learning of radar images as claimed in claim 1 is characterized in that: The low-dimensional state feature vector in step 3 is: Among them, S l represents the low-dimensional state feature vector; a t-1 Indicates the action output at the previous moment; represents the yaw angular velocity; the low-dimensional state feature vector includes the LOS guidance error, and the LOS guidance error includes the distance error information y e , and heading error information χ d , The reward function in step 3 is the position error reward r d , heading error reward r ψ and the heading stability reward r s composition: r=w d r d +w ψ r ψ +w s r s ; The reward is described in the form of a negative exponential function, and the range of each sub-reward is limited to (-1, 1], k1, k2, k3 represent the adjustment coefficients of each sub-reward, σ represents the standard deviation of the USV heading over a period of time; r s represents the heading stability reward; w d 、w ψ and w s represents the weight coefficient; When the USV goes out of bounds or collides, the mission is terminated and a termination reward is generated. Depending on the situation, the rewards obtained are as follows: When the USV exceeds the mission boundary, r = r o , r o Represents the out-of-bounds penalty reward, which is negative; when the USV collides with an obstacle, r = r c , r c Represents the collision penalty reward, which is negative; when the USV reaches the obstacle avoidance end area, k4 represents the adjustment coefficient, r e Represents the task completion reward, which is positive. The closer the arrival at the end zone is to the reference path, the higher the termination reward is.
3. The unmanned boat intelligent obstacle avoidance method based on end-to-end deep reinforcement learning of radar images as claimed in claim 1 is characterized in that: The deep convolutional neural network for multi-feature scale fusion in step 3 includes: Input layer, used to input obstacle track layer L s With the collision threat layer L c The composed dual channel image; The convolution layer-pooling layer is used to compress the size of the feature tensor, expand the number of channels, and flatten it into a one-dimensional feature vector through multiple convolution-pooling processes; The flattening layer is used to concatenate the low-dimensional state features and the one-dimensional feature vector to obtain a common feature vector; The first strategy network layer is used to obtain the corresponding output using two independent fully connected networks; The second strategy network layer is used to obtain the corresponding output using two independent fully connected networks; The first value network layer is used to obtain the corresponding output using two independent fully connected networks; The second value network layer is used to obtain the corresponding output using two independent fully connected networks; At the same time, the multi-feature scale fusion deep convolutional neural network is extracted from the action set Select the action with the highest probability from The actions included are desired heading -10°, desired heading +10°, and desired heading unchanged.
4. The unmanned boat intelligent obstacle avoidance method based on end-to-end deep reinforcement learning of radar images as claimed in claim 1 is characterized in that: The training of a deep convolutional neural network for multi-feature scale fusion includes: Use TensorFlow as a deep learning training framework and input the high-dimensional state L of the network s and L c Use normalization processing; use standardization to process the low-dimensional state feature vector S l , using a proximal strategy optimization algorithm to train the multi-feature scale fusion deep convolutional neural network; At the same time, during training, randomly select from the action set according to the probability distribution Select an action to update the desired heading.
5. An unmanned boat intelligent obstacle avoidance system based on end-to-end deep reinforcement learning of radar images that implements the unmanned boat intelligent obstacle avoidance method based on end-to-end deep reinforcement learning of radar images as described in any one of claims 1 to 4, characterized in that: The unmanned boat intelligent obstacle avoidance system based on end-to-end deep reinforcement learning of radar images includes: A perception module, which is used to comprehensively summarize and process reference path information, obstacle information and USV status information, and convert the reference path information, obstacle information and USV status information into high-dimensional image information and low-dimensional status information; A decision module, used to output a desired heading based on state information using a deep neural network; The execution module is used to output the control amount of the USV throttle and steering command according to the current USV navigation parameters and the expected heading, and update the state information according to the USV dynamics and kinematics model.
6. A computer device, characterized in that: The computer device includes a memory and a processor, the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the unmanned boat intelligent obstacle avoidance method based on end-to-end deep reinforcement learning of radar images as described in any one of claims 1-4.
7. A computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the steps of the unmanned boat intelligent obstacle avoidance method based on end-to-end deep reinforcement learning of radar images as described in any one of claims 1 to 4.
8. An information data processing terminal, characterized in that: The information data processing terminal is used to implement the unmanned boat intelligent obstacle avoidance system based on end-to-end deep reinforcement learning of radar images as described in any one of claim 5.