Autonomous navigation and obstacle avoidance system based on deep learning

Abstract

The application provides a kind of autonomous navigation and obstacle avoidance system based on deep learning, comprising the following steps: S1, the monocular image is carried out semantic segmentation, so as to segment the obstacle and determine the water boundary, and detect the distance between the ship and the obstacle;S2, the semantic segmentation result is transmitted to semantic point cloud, to generate driving recommendation map;S3, global path planning is carried out through global positioning system GPS, the absolute position of the ship is positioned, and local path planning is carried out;S4, the fastest path and the shortest path of the ship are designed through local path planning;S5, the deep learning model is trained through semantic point cloud, the ship motion planning is carried out through deep reinforcement learning, and then the turning and speed of the ship are controlled, to realize the autonomous movement navigation of the ship.In the present application, depth reinforcement learning is used to control the steering engine to turn the rudder to the correct direction.In addition, global and local path planning can find the best route and the shortest and safest path.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an autonomous navigation and obstacle avoidance system based on deep learning. Background Technology

[0002] Traditional ships require human intervention to navigate, which increases the risk of accidents caused by human error and negligence. Therefore, an autonomous ship navigation technology is needed to follow efficient routes, enabling ships to navigate autonomously without human intervention. When navigating autonomously, ships need to avoid collisions with obstacles in oceans or rivers; thus, how to enable ships to turn spontaneously and how to autonomously control their speed are challenging tasks. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides an autonomous navigation and obstacle avoidance system based on deep learning, which solves the problems of how to enable ships to turn spontaneously and how to control the autonomous adjustment of ship speed in existing technologies.

[0004] The above-mentioned technical objective of the present invention is achieved through the following technical solution:

[0005] A deep learning-based autonomous navigation and obstacle avoidance system includes the following steps:

[0006] S1, perform semantic segmentation on the monocular image to segment obstacles, determine water boundaries, and detect the distance between the ship and the obstacles;

[0007] S2, the semantic segmentation results are passed to the semantic point cloud to generate a driving recommendation map;

[0008] S3 uses the Global Positioning System (GPS) for global path planning, locating the ship's absolute position, and performing local path planning.

[0009] S4, which designs the fastest and shortest paths for ships through local path planning;

[0010] S5 uses semantic point clouds to train a deep learning model, and deep reinforcement learning to plan ship motion, thereby controlling the ship's steering and speed to achieve autonomous navigation.

[0011] The present invention is further configured such that: in step S1, the semantic segmentation method is based on traversable regions and inaccessible regions, wherein traversable regions are divided into rivers and oceans, and the segmented regions are defined by the boundaries of the traversable regions.

[0012] The present invention is further configured such that: in step S2, when generating the driving suggestion map, image trajectories of obstacle points are created and tracked using an IBVS controller, wherein the i-th point Pi The image trajectory of (w) is represented as w∈[0,1], the initial position is represented as w=0, the target position is represented as w=1, and the vector P i (w) satisfies the boundary conditions

[0013]

[0014] P i (w) and pi(w), i = 1, ..., n are defined as parameters for each camera frame, and the object point of each camera frame is related to P in all frames. i (w) matches, where Wò[0,1];

[0015] Definition relative to F o The camera pose is F * ,but

[0016] P(F * ,F o )={R,t},

[0017] Let Fd(w) be a camera frame along the reconstructed camera trajectory, relative to F o The pose of Fd(w) is defined as

[0018] P(F d (w),F o )={R(w),d(w)},

[0019] The trajectory of the i-th object point can be represented by R(w) and d(w) as follows:

[0020]

[0021] in It's a parameterized point depth, and the UI is the camera frame F. o The i-th physical point ρ i ,

[0022]

[0023] By eliminating point depth To calculate singular value decomposition.

[0024] The present invention is further configured as follows: global path planning involves determining the optimal path from the starting point to the destination while avoiding obstacles, and is expressed as an optimization problem: minimizing f(x) with g(x) <= 0, where x is a vector of decision variables, f(x) is the objective function to be minimized, and g(x) is a constraint condition.

[0025] The present invention is further configured to: determine the degree of difference between the ship's actual course and the expected course by using a camera on the ship.

[0026] The present invention has the following advantages:

[0027] 1. Using deep learning algorithms, obstacles can be detected and identified more accurately and efficiently than traditional algorithms.

[0028] 2. The system can be trained using large-scale datasets, enabling it to learn from various scenarios and quickly adapt to new environments.

[0029] 3. The system can make real-time decisions and adjust its trajectory accordingly, thereby achieving safe and efficient navigation. Overall, the system provides a more reliable and flexible solution for autonomous navigation and obstacle avoidance. Attached Figure Description

[0030] Figure 1 This is a schematic diagram of the process steps in an embodiment of the present invention;

[0031] Figure 2 This is the overall framework of the blockchain-based asynchronous and semi-synchronous multi-agent distributed learning framework proposed in this embodiment of the invention in a heterogeneous environment;

[0032] Figure 3 This invention is used to calculate the distances between other ships and obstacles in this embodiment.

[0033] Figure 4 This is a schematic diagram of global ship maps and traffic analysis information in an embodiment of the present invention;

[0034] Figure 5 This is a schematic diagram illustrating the arc approximation definition of a local path in an embodiment of the present invention;

[0035] Figure 6 This is a schematic diagram illustrating how the segmented region is divided into different parts in an embodiment of the present invention, including recommended, not recommended, and unclassified regions. Detailed Implementation

[0036] The technical solutions of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0037] An autonomous ship navigation system employs semantic segmentation and deep reinforcement learning. The primary purpose of semantic segmentation is to define the ship's surrounding boundaries and detect obstacles. Furthermore, autonomous navigation utilizes deep reinforcement learning to perform path and motion planning based on dynamic real-time conditions. Onboard cameras are used to determine the degree of difference between the ship's actual and expected course. Figure 1 As shown, the overall framework of an autonomous navigation system consists of the following steps:

[0038] S1 performs semantic segmentation on the monocular image to segment obstacles, determine water boundaries, and detect the distance between ships and obstacles; for example... Figure 6As shown, the semantic segmentation method is based on traversable regions (recommended regions), inaccessible regions (not recommended regions), and unknown regions. Traversable regions are divided into rivers and oceans, and the segmented regions are defined by the boundaries of traversable regions.

[0039] S2, the semantic segmentation results are passed to the semantic point cloud to generate a driving recommendation map; for example... Figure 2 As shown, when generating the driving suggestion map, image trajectories of obstacle points are created and tracked using the IBVS controller, where the i-th point P... i The image trajectory of (w) is represented as w∈[0,1], the initial position is represented as w=0, the target position is represented as w=1, and the vector P i (w) satisfies the boundary conditions

[0040]

[0041] P i (w) and pi(w), i = 1, ..., n are defined as parameters for each camera frame, and the object point of each camera frame is related to P in all frames. i (w) matches, where Wò[0,1];

[0042] Definition relative to F o The camera pose is F * ,but

[0043] P(F * ,F o )={R,t},

[0044] This equation allows us to find the camera pose, represented by variables R,t and t. We can only determine t up to a certain scaling factor, which represents the unknown distance between the origin of the starting and target frames, such as... Figure 3 As shown. In fact, the normalized camera pose R,tnorm can be calculated using the essential matrix algorithm, where tnorm = t / |t|.

[0045] Let Fd(w) be a camera frame along the reconstructed camera trajectory, relative to F o The pose of Fd(w) is defined as

[0046] P(F d (w),F o )={R(w),d(w)},

[0047] The trajectory of the i-th object point can be represented by R(w) and d(w) as follows:

[0048]

[0049] in It's a parameterized point depth, and the UI is the camera frame F. o The i-th physical point ρ i ,

[0050]

[0051] By eliminating point depth To calculate singular value decomposition.

[0052] S3 uses the Global Positioning System (GPS) to perform global path planning, locate the ship's absolute position, and perform local path planning. Global path planning involves determining the best path from the starting point to the destination while avoiding obstacles. It is expressed as an optimization problem: minimizing f(x) with g(x) <= 0, where x is a vector of decision variables, f(x) is the objective function to be minimized, and g(x) is the constraint condition.

[0053] S4 designs the fastest and shortest routes for ships through local path planning; the routes are planned by a driving recommendation map system, and the path planning is based on GPS coordinates and other data. Figure 4 The global map shown. Points P0 to P... i The derivative can be approximated as a circular arc. Figure 5 The arcs between points are shown, which helps in finding dynamic local paths. The motion model used by the ship can be represented as follows:

[0054]

[0055] Where θ represents the ship's orientation and the point of contact (x, y) with the horizontal plane during the ship's turn. (x, y) represents the ship's position, and v is the turning speed. The dynamic constraints are as follows:

[0056] Δv|≤a max,linear ·dt,

[0057] and|Δω target |≤a max,angluar ·dt

[0058] Where Δv represents linear acceleration, Δω target It represents angular acceleration.

[0059] With each step the boat takes, it utilizes the centrifugal force of the river water.

[0060] |v target ·ω target |≤a max,lateral

[0061] Here |v|≤v max Represents the maximum speed, used to limit the ship's turning, while ω|≤ω maxThis represents the maximum angular velocity.

[0062] S5 uses semantic point clouds to train a deep learning model, and deep reinforcement learning is used for ship motion planning to control the ship's steering and speed, thus achieving autonomous navigation. Motion planning consists of two parts: collision avoidance and speed control, as described below:

[0063] Collision Avoidance: The problem of collision avoidance among multiple ships can be described using a reinforcement learning framework, including ship states. t Ship movement u t And the status of another ship. t The ship's position consists of a visible portion and an invisible (hidden) portion, and st = [st o ,st h In the global coordinate system, the visible states are the ship's position, velocity, and radius, denoted as:

[0064] s o =[p x p y v x v y ,r]∈R 5 ,

[0065] The invisible state is the target position, desired velocity, and direction, denoted as s. h =[pgx,pgy,vpref,Ψ]∈R 4 The motion consists of velocity and heading angle, denoted as u. t =[v t Ψ t ]∈R 2 ,Strategy Aiming to minimize the expected time E[t] to reach the target g [At the same time, avoid collisions with other objects.]

[0066]

[0067]

[0068]

[0069]

[0070] In (7), the expectation is about the unobservable state (intention) and policy of other agents, (8) is the collision avoidance constraint, (9) is the goal constraint, and (10) is the kinematics of the agent.

[0071] Simultaneously, a deep reinforcement learning framework is used to solve the policy problem, and the ship configuration is represented as follows: To achieve the target position and avoid obstacles, the reward function is described as R. col (s jn Deep reinforcement learning techniques can approximate the optimal value. In the learning process, this value encodes the ship's estimated time and objective. Although It can be used to generate the current state and actions, but according to:

[0072]

[0073] For each possible action, st+1 jn It may be a random process, and this rule starts from the current state st. jn Predict the next state st+1 jn And select the state V that has the highest value. * (st+1 jn The action u.

[0074] Since collision avoidance with other obstacles is unknown, this means that the state transition dynamics are... Assume that the velocity of a ship's kinematics is defined as Duration is extracted from the velocity value as:

[0075]

[0076]

[0077] In addition, the status is divided into two categories: i) information about the ship itself. ii) Everything within rivers and oceans. The network estimates probabilities based on upcoming actions, and the distance between ships is defined as:

[0078] s = [d g ,v pref ,ψ,r]

[0079]

[0080] Where, d g =|p g -p|2 is the distance between the ship and the target, and It is the distance to other ships.

[0081] Ships and boats can move according to changes in speed and heading angle. Speed ​​is defined as v. pref The distance between the boats is ±π / 6. The control handle, speed, and value, as well as the heading selection for 0, are [-π / 6, 0, π / 6]. These parameters are necessary for the boat's actual steering.

[0082] The sparse reward function is defined as:

[0083]

[0084] Where, d min This represents the distance to distant ships and obstacles.

[0085] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An autonomous navigation and obstacle avoidance system based on deep learning, characterized in that: Comprise the following steps: S1, monocular image is segmented semantically, thereby the obstacles are segmented and the water area boundary is determined, and the distance between the ship and the obstacle is detected; S2, the semantic segmentation result is passed to the semantic point cloud to generate a driving recommendation map; S3, global path planning is performed through a global positioning system (GPS), the absolute position of the ship is located, and local path planning is performed; S4, the fastest path and the shortest path of the ship are designed through the local path planning; S5, a deep learning model is trained through the semantic point cloud, ship motion planning is performed through deep reinforcement learning, thereby the turning and speed of the ship are controlled, and autonomous mobile navigation of the ship is realized; In step S1, the semantic segmentation method is based on passable areas and impassable areas, wherein the passable areas are divided into rivers and oceans, and the segmented areas are defined by the boundaries of the passable areas; In step S2, when generating the driving advice map, an image trajectory of the obstacle points is made, and tracking is performed using an IBVS controller, where the image trajectory of the i-th point is expressed as , the initial position is expressed as , the target position is expressed as w = 1, and the vector satisfies the boundary condition ， and defined as a parameter of each camera frame, each object point of each camera frame is matched with all frames of wherein ; The camera pose is defined relative to is then ， Let For a camera frame along the reconstructed camera trajectory, the pose is defined as The The pose is defined as ， The trajectory of the i-th object point can be expressed as R(w) and d(w): wherein is a parameterized point depth, is a camera frame of the i-th physical point , By eliminating the point depth , to compute singular value decomposition.

2. The deep learning-based autonomous navigation and obstacle avoidance system of claim 1, wherein: Global path planning involves determining an optimal path from a start to a destination while avoiding obstacles, formulated as an optimization problem: minimize subject to where x is a vector of decision variables, is the objective function to be minimized, are the constraints.

3. The deep learning-based autonomous navigation and obstacle avoidance system of claim 1, wherein: The degree of difference between the actual heading of the ship and the expected heading is determined through a camera on the ship.

Images