Near-shore scene visual SLAM (Simultaneous Localization and Mapping) method and system for unmanned ship

Abstract

The invention discloses an unmanned ship-oriented coastal scene visual SLAM method and system. The method comprises the steps of obtaining n frames of dynamic detection images of an unmanned ship coastal scene simultaneously containing the sky, the land bank and the water surface; performing target detection on the image by using a YOLOv5s network, and marking a dynamic target to obtain a detection result; performing adaptive edge detection on the image by using an edge detection algorithm, and performing edge elimination on the target to obtain an edge contour line of each region; constructing a segmentation line of the image according to the edge contour line, optimizing the segmentation line of the land bank and the water surface by using an RANSAC algorithm, and removing a dynamic region to obtain a static detection image; inputting the image into an ORB-SLAM3 system, and using an LK pyramid optical flow method and an epipolar geometric constraint method to judge and reject dynamic feature points to obtain static feature points; and inputting the static feature points into an ORB-SLAM3 system for positioning and map construction. According to the method and the system, the dynamic area in the image can be eliminated, the image processing efficiency of the system is improved, and the positioning and map building precision is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection in deep learning, and in particular to a near-shore scene visual SLAM method and system for unmanned ships. Background Art

[0002] Due to the complex environment, dense traffic flow and high navigation difficulty of near-shore scenes, more accurate and rich environmental information is needed to ensure the safety of autonomous navigation of unmanned ships. Simultaneous Localization and Mapping (SLAM) technology can enable sensor-carrying carriers to achieve positioning and map construction in unknown environments. Among them, visual SLAM uses cameras as sensors and uses images collected during the movement of the carrier to build a global model. Due to the rich information it collects and the advantages of low cost, compact structure and strong scalability, it has a wide range of applications in the fields of robots, augmented reality, autonomous driving and drones. It is currently recognized as an effective solution for achieving autonomous navigation and environmental perception.

[0003] In image processing theory, feature points refer to points where the grayscale value of an image jumps or where the curvature is large on the edge of the image. They are usually used for target detection, image segmentation, and feature extraction. Since feature points are scale invariant, current visual SLAM systems mainly use feature point methods for image processing and subsequent positioning and map construction. Therefore, they perform well in static scenes. However, in dynamic scenes, dynamic feature points often lead to erroneous data associations, making it difficult to achieve good positioning accuracy.

[0004] Due to different water environments, the nearshore navigation scenes of unmanned ships are diverse. At the same time, the fluctuating water surface, the clouds in the sky and the sailing ships make them dynamic. However, the existing nearshore scene visual SLAM system mainly focuses on the interference of water surface factors and lacks the response to other dynamic factors in the scene. At the same time, other sensors need to be integrated to make up for the shortcomings of visual SLAM, and the overall coupling of the system is too strong. Summary of the invention

[0005] The present invention provides a near-shore scene visual SLAM method and system for unmanned ships, so as to overcome the technical problem that the existing near-shore scene visual SLAM system mainly focuses on the interference of water surface factors and lacks the response to other dynamic factors in the scene. Therefore, in dynamic scenes, the acquired dynamic feature points often lead to erroneous detection results, making it difficult to obtain good positioning accuracy.

[0006] In order to achieve the above object, the technical solution of the present invention is:

[0007] A nearshore scene visual SLAM method for unmanned ships, comprising:

[0008] S1: Obtain n frames of dynamic detection images of the unmanned ship near the shore scene including the sky, the land and the water surface; set the sky and the water surface as dynamic areas, and set the land as a static area;

[0009] S2: Use the YOLOv5s network to perform target detection on the t-th frame of the dynamic detection image, mark the dynamic targets in the dynamic detection image, and obtain the dynamic target detection result, where t∈(2,n);

[0010] S3: Use the edge detection algorithm to perform adaptive edge detection on the dynamic detection image of the tth frame, and remove the edges of the dynamic targets according to the dynamic target detection results to obtain the edge contour lines of each area;

[0011] S4: construct segmentation lines of two dynamic areas and a static area in the dynamic detection image of the t-th frame according to the edge contour lines, optimize the segmentation line between the land bank and the water surface using the RANSAC algorithm, remove the dynamic area according to the segmentation line, and obtain the static detection image of the t-th frame;

[0012] S5: Input the static detection image of the tth frame into the ORB-SLAM3 system, extract feature points from the static detection image of the tth frame, use the LK pyramid optical flow method and the epipolar geometry constraint method to judge and eliminate the dynamic feature points in the feature points, and obtain static feature points;

[0013] S6: Input the static feature points into the ORB-SLAM3 system again to locate and map the unmanned boat in the nearshore scene.

[0014] Further, S4 constructs the segmentation lines of the two dynamic regions and the static region in the dynamic detection image of the t-th frame according to the edge contour lines, including:

[0015] S41, establish an image coordinate system with the vertex of the left edge of the image as the origin, the direction parallel to the edge contour as the x-axis, and the direction perpendicular to the edge contour as the y-axis, and set the dynamic detection image as I(x, y), where x and y are arbitrary coordinates in the image plane, i.e., pixel points;

[0016] S42, let the starting point of a certain edge segment in the edge contour be (x1, y1) and the end point be (x2, y2), select a straight line x=i, i=0,1,2,..., in the image coordinate system, construct the coordinates of the intersection of the straight line x=i, i=0,1,2,..., and the edge segment (x,y i ), as shown in formula (1),

[0017]

[0018] Expanding formula (1), we get formula (2):

[0019]

[0020] Among them, y i The ordinate represents the edge intersection point;

[0021] At x = i, let the coordinates of the upper edge of the shore be is the set of ordinates of all edge line intersections closest to the upper edge of the image; the coordinates of the lower edge of the shore reflection are is the set of ordinates of all edge line intersections closest to the bottom edge of the image;

[0022] Combining x=i, i=0,1,2,..., the straight line traverses all edge contour lines, and the upper edge of the shore, that is, the dividing line between the sky and the shore, is shown in formula (3).

[0023]

[0024] Among them, f up Indicates the dividing line between the sky and the shore;

[0025] The lower edge of the shore reflection, i.e., the dividing line between the water surface reflection and the non-reflection, is shown in formula (4):

[0026]

[0027] Connect the dividing line of the upper edge of the shore with the dividing line of the lower edge of the shore reflection to obtain the midpoint data, and use the midpoint data to generate the dividing line between the shore and the water surface, as shown in formula (5):

[0028]

[0029] Among them, f mid Indicates the dividing line between land and water.

[0030] Furthermore, S4 uses the RANSAC algorithm to optimize the segmentation line between the land bank and the water surface, including:

[0031] S43, sliding the midpoint data along the positive direction of the x-axis using a sliding window method, setting the window width, sliding the window continuously from the left edge of the detection image, using a backward difference method to calculate the backward difference value of the average value of the ordinates of all midpoint data in each window and the previous window, and using the backward difference method again to calculate the absolute value of each difference result for the backward difference value, when the absolute value is 0, setting the window corresponding to the difference result as a segment, traversing all windows to obtain multiple segments;

[0032] S44. Assume that there are N midpoints in a segment, randomly select 2 midpoints to form a straight line, and assume that all midpoints in the segment that fall on the straight line are interior points. Calculate the number of iterations of the RANSAC algorithm, as shown in formula (6):

[0033] r = log(1-p r ) / log(1-h 2 ) (6)

[0034] Among them, r represents the number of iterations, p r represents the confidence, h represents the proportion of internal points in N;

[0035] S45, randomly select two midpoints in a segment again to form a new straight line, recalculate the proportion h of the inner points on the straight line in N, and if h is greater than the previous result, recalculate the number of iterations; stop calculating the proportion when the calculated number of iterations is equal to or less than the number of experiments, and the positions of the two midpoints selected at this time are the best straight line fitting result of the current segment;

[0036] S46. Sort the inner points on the straight line where the two best midpoints are located from left to right, select the first and last inner points among all the inner points, and use them as the starting point and end point of the straight line to complete the water shoreline fitting result of the current segment, traverse all segments, repeat S44-S45, obtain the water shoreline fitting results of all segments, connect the water shoreline fitting results of all segments, and the obtained water shoreline is the optimized dividing line between the land shore and the water surface.

[0037] Furthermore, S5 inputs the static detection image of the tth frame into the ORB-SLAM3 system, extracts feature points from the static detection image of the tth frame, uses the LK pyramid optical flow method and the epipolar geometry constraint method to judge and eliminate the dynamic feature points in the feature points, and obtains static feature points, including:

[0038] S51, inputting the t-th frame static detection image into the ORB feature extraction module in the ORB-SLAM3 system, extracting feature points of the t-th frame static detection image, and obtaining feature points of the image;

[0039] S52, using the LK pyramid optical flow method, performing optical flow consistency calculation on the t-th frame static detection image and the t-1-th frame dynamic image, obtaining matching points in the t-1-th frame dynamic image that have the same optical flow as the feature points of the t-th frame static detection image and forming point pairs, and treating feature points that do not have the same optical flow as the matching points as dynamic feature points and deleting them;

[0040] S53, calculating a basic matrix using point pairs having the same optical flow;

[0041] S54. Determine the epipolar line according to the basic matrix and the matching points of the dynamic image of the t-1th frame, calculate the distance between the feature point of the static detection image of the tth frame and its corresponding epipolar line through the epipolar geometry constraint method, compare the distance with the set threshold, if it is greater than the threshold, then the point is regarded as a dynamic feature point and deleted, if it is less than or equal to the threshold, then it is retained, and the feature point finally retained is the final static feature point.

[0042] Furthermore, edge detection algorithms include Canny operator, maximum inter-class variance and maximum entropy segmentation algorithms.

[0043] Furthermore, an edge detection algorithm is used to perform adaptive edge detection on the dynamic detection image of the t-th frame, and the edge of the dynamic target is removed according to the dynamic target detection result to obtain the edge contour lines of each area, including:

[0044] S31, using the Canny operator and the maximum inter-class variance to perform preliminary edge detection on the dynamic areas of the sky and the shore in the detection image to obtain preliminary edge contour lines;

[0045] S32, optimizing the preliminary edge contour line using a maximum entropy segmentation algorithm to obtain an edge contour line containing a dynamic target;

[0046] S33, removing the edges of the dynamic targets according to the dynamic target detection result, and obtaining edge contours of each area.

[0047] A nearshore scene visual SLAM system for unmanned ships, comprising:

[0048] The image preprocessing module includes an image acquisition unit, an image edge detection unit and an image segmentation unit. The image acquisition unit is used to acquire n frames of dynamic detection images of unmanned ship nearshore scenes that include sky, land and water surface; the image edge detection unit is used to perform target detection and adaptive edge detection on the t-th frame of dynamic detection image according to the YOLOv5s network and edge detection algorithm to obtain edge contour lines of various regions; the image segmentation unit is used to construct a segmentation line according to the edge contour line, optimize the segmentation line using the RANSAC algorithm and eliminate the dynamic area according to the segmentation line to obtain the t-th frame of static detection image;

[0049] The feature point extraction module is used to input the static detection image of the tth frame into the ORB-SLAM3 system, extract the feature points of the static detection image of the tth frame, and use the LK pyramid optical flow method and the epipolar geometry constraint method to judge and eliminate the dynamic feature points in the feature points to obtain static feature points;

[0050] The image processing module is an ORB-SLAM3 system, which is used to process static feature points and locate and map the unmanned ship in near-shore scenes.

[0051] Beneficial effect: The present invention provides a nearshore scene visual SLAM method for unmanned ships. By using the YOLOv5s target detection network to perform target detection on images, dynamic targets in the scene are identified. The adaptive edge detection algorithm and the region segmentation algorithm can segment and eliminate dynamic targets and dynamic regions in the image according to the results of dynamic target detection, thereby providing a static scene for the operation of the SLAM system and improving the efficiency of the nearshore scene visual SLAM system in processing images. Dynamic feature points are eliminated by combining the LK pyramid optical flow method with epipolar geometry constraints to obtain static feature points, thereby improving the accuracy of the nearshore scene visual SLAM system in positioning and map construction based on feature points. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0053] Figure 1 A method flow chart of a nearshore scene visual SLAM method for unmanned ships provided by the present invention;

[0054] Figure 2 A target detection result diagram of a YOLOv5s network in one embodiment;

[0055] Figure 3 An edge detection result diagram obtained by combining Canny and Otsu in one embodiment;

[0056] Figure 4 An edge detection result diagram obtained by combining the Canny, Otsu and maximum entropy segmentation algorithms in one embodiment;

[0057] Figure 5 This is an adaptive edge detection result diagram obtained by combining an edge detection algorithm of an embodiment of the present invention with a YOLOv5s network;

[0058] Figure 6 A diagram of a segmentation line extraction process according to an embodiment of the present invention;

[0059] Figure 7 This is a segmentation line extraction result diagram of an embodiment of the present invention;

[0060] Figure 8 This is a diagram of the segmentation process of a segmentation line in one embodiment of the present invention;

[0061] Fig. 9 This is a result diagram of optimizing the segmentation line using the RANSAC algorithm according to an embodiment of the present invention;

[0062] Fig.10 A dynamic feature point extraction result diagram according to an embodiment of the present invention;

[0063] Fig.11 This is a static feature point extraction result diagram of an embodiment of the present invention;

[0064] Fig.12 A system framework diagram of a nearshore scene visual SLAM system for unmanned ships provided by the present invention;

[0065] Fig.13 An image processing flow chart of a nearshore scene visual SLAM system for unmanned ships provided by the present invention;

[0066] Fig.14 A visualization feature point matching result diagram of two consecutive frames according to an embodiment of the present invention;

[0067] Fig.15 A comparison diagram of the algorithm operation process of an embodiment;

[0068] Fig.16 A map window comparison diagram of an embodiment of the present invention;

[0069] Fig.17 A camera motion trajectory diagram according to an embodiment of the present invention. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0071] This embodiment provides a near-shore scene visual SLAM method for unmanned ships, such as Figure 1 As shown, including:

[0072] S1: Obtain n frames of dynamic detection images of the unmanned ship near the shore scene including the sky, the land and the water surface; set the sky and the water surface as dynamic areas, and set the land as a static area;

[0073] S2: Use the YOLOv5s network to perform target detection on the t-th frame of the dynamic detection image, mark the dynamic targets in the dynamic detection image, and obtain the dynamic target detection result, where t∈(2,n);

[0074] S3: Use the edge detection algorithm to perform adaptive edge detection on the dynamic detection image of the tth frame, and remove the edges of the dynamic targets according to the dynamic target detection results to obtain the edge contour lines of each area;

[0075] S4: construct segmentation lines of two dynamic areas and a static area in the dynamic detection image of the t-th frame according to the edge contour lines, optimize the segmentation line between the land bank and the water surface using the RANSAC algorithm, remove the dynamic area according to the segmentation line, and obtain the static detection image of the t-th frame;

[0076] S5: Input the static detection image of the tth frame into the ORB-SLAM3 system, extract feature points from the static detection image of the tth frame, use the LK pyramid optical flow method and the epipolar geometry constraint method to judge and eliminate the dynamic feature points in the feature points, and obtain static feature points;

[0077] S6: Input the static feature points into the ORB-SLAM3 system again to locate and map the unmanned boat in the nearshore scene.

[0078] Specifically, firstly, n frames of dynamic detection images of unmanned boat nearshore scenes including sky, land and water are obtained, real images are collected, and the authenticity of image processing results is improved; the YOLOv5s network is used to perform target detection on the t-th frame of dynamic detection image, and the dynamic targets in the dynamic detection image are marked to obtain the dynamic target detection results; the YOLOv5s network is used to select the YOLOv5s network as the target detection algorithm, and the possible dynamic targets are marked to realize the dynamic target detection of the nearshore image, which can predetermine the dynamic targets in the image and prepare for the subsequent elimination of dynamic targets; the edge detection algorithm is used to perform adaptive edge detection on the t-th frame of dynamic detection image, and the edge of the dynamic target is eliminated according to the dynamic target detection result to obtain the edge contour lines of each area, mainly using the Canny operator, and the maximum inter-class variance method is used to optimize the threshold of the Cany operator to improve the accuracy of edge detection, and then the maximum entropy segmentation algorithm is used to optimize the edge contour lines to remove the redundant edge contour lines that appear in the image due to the unclear distinction; the t-th frame of dynamic detection image is constructed according to the edge contour lines The segmentation lines between the two dynamic areas and the static area in the image are obtained by optimizing the segmentation line between the land bank and the water surface using the RANSAC algorithm. The dynamic area is eliminated according to the segmentation line to obtain the t-th frame of static detection image. The segmentation lines between the sky and the land bank, and the land bank and the water surface are obtained according to the edge contour line. The segmentation line between the land bank and the water surface is optimized using the RANSAC algorithm to obtain a smoother segmentation line. The dynamic area can be eliminated to the greatest extent according to the segmentation line to improve the static nature of the image. The t-th frame of static detection image is input into the ORB-SLAM3 system, and feature points are extracted from the t-th frame of static detection image. The dynamic feature points in the feature points are judged and eliminated using the LK pyramid optical flow method and the epipolar geometry constraint method to obtain static feature points. By using the LK pyramid optical flow method and the epipolar geometry constraint method, the dynamic feature points in the feature points can be judged and eliminated to obtain the final static feature points, thereby reducing the impact of dynamic feature points on the system and improving positioning accuracy. Finally, the static feature points are input into the ORB-SLAM3 system again to realize the positioning and map construction of near-shore scenes for unmanned ships.

[0079] In a specific embodiment, a scheme for obtaining n frames of dynamic detection images of an unmanned boat near the shore scene including the sky, the land and the water surface, wherein the sky and the water surface are dynamic areas and the land is a static area, is:

[0080] In this scheme, the zed2 camera is used to collect real dynamic images, and all the collected dynamic images are divided into n frames of dynamic detection images. The images contain the sky, land and water surface at the same time. The sky and water surface are the dynamic areas in the image, and the land is the static area in the image.

[0081] In this embodiment, real images are collected to improve the authenticity of the final image processing results.

[0082] In a specific embodiment, the YOLOv5s network is used to perform target detection on the t-th frame dynamic detection image, and the dynamic target in the dynamic detection image is marked to obtain the dynamic target detection result, where the solution of t∈(2,n) is:

[0083] Add dynamic targets such as ships and people to the detection list, input the t-th frame of dynamic detection image into the YOLOv5s network for detection, and if a target appears in the detection list, record its bounding box and target type.

[0084] YOLO is a neural network-based target detection algorithm with a relatively high overall evaluation. YOLOv5 has a lightweight feature and performs better in terms of training and reasoning speed, making it more suitable for target detection tasks that require high real-time performance and response speed. The YOLOv5s network is the network with the smallest depth and the smallest feature map width in the YOLOv5 series. Compared with other versions, it runs faster and has lower requirements on hardware devices, making it more suitable for deployment on mobile terminals or embedded systems. Selecting the YOLOv5s network as the target detection algorithm can accurately annotate dynamic targets, realize dynamic target detection in nearshore images, and prepare for the subsequent elimination of dynamic targets. The detection results are as follows: Figure 2 shown.

[0085] In a specific embodiment, an edge detection algorithm is used to perform adaptive edge detection on the t-th frame dynamic detection image, and the edge of the dynamic target is removed according to the dynamic target detection result, so as to obtain the edge contour line of each area:

[0086] S31, using the Canny operator and the maximum inter-class variance to perform preliminary edge detection on the dynamic areas of the sky and the shore in the detection image to obtain preliminary edge contour lines;

[0087] S32, optimizing the preliminary edge contour line using a maximum entropy segmentation algorithm to obtain an edge contour line containing a dynamic target;

[0088] S33, removing the edges of the dynamic targets according to the dynamic target detection result, and obtaining edge contours of each area.

[0089] According to the characteristics of nearshore images, this scheme selects edge detection method to extract element information to extract edge lines. After edge lines are extracted, the segmentation line between dynamic area and static area can be determined according to the edge lines, and dynamic areas and dynamic targets in the image can be eliminated.

[0090] The edge is a collection of pixels in an image where the grayscale values ​​of the surrounding pixels change significantly. Edge detection is to extract these significantly changed points for image analysis and processing. The edge detection algorithms in this scheme include the Canny operator, the maximum inter-class variance, and the maximum entropy segmentation algorithm.

[0091] Among them, the Canny operator has the ability of dual threshold detection and can perform non-maximum suppression on the edge gradient direction. Therefore, the edge detection algorithm based on the Canny operator has better adaptability to the noise interference caused by water surface ripples. However, the high and low thresholds TH and TL of the Canny operator are usually set manually according to the application scenario. In order to better apply it to different nearshore scenes, this paper associates the selection of the threshold with the current nearshore image, and adopts the maximum inter-class variance method with strong image edge recognition ability, also known as the Otsu method, to achieve adaptive acquisition of the threshold. The principle of the Otsu method is to divide the image into two parts, foreground and background, by calculating the grayscale histogram of the image, and use the variance to measure the uniformity of the grayscale distribution. The larger the inter-class variance between the background and the foreground, the greater the difference between the two parts of the image. Therefore, by maximizing the variance between the two classes as the optimal threshold of the Canny operator, the adaptive edge detection results after combining Canny and Otsu are as follows: Figure 3 As shown in the figure, it can be seen that the edge detection algorithm has a better detection effect for images with large foreground and background distinction, such as Figure 3 (a), but for images with small distinction between foreground and background, the detection result is not ideal, such as Figure 3 As shown in (b), there is a situation where clouds in the sky area are identified as foreground;

[0092] Use the maximum entropy segmentation algorithm to process images with a small degree of distinction between the foreground and the background. The maximum entropy segmentation algorithm has a better segmentation effect on images with fuzzy boundaries and small degrees of distinction between the foreground and the background.

[0093] The results of combining the adaptive edge detection results with the maximum entropy segmentation algorithm and the target detection algorithm are as follows: Figure 4 As shown, Figure 4 (a) is the binary image of the maximum entropy segmentation result. Figure 4 (b) Figure 4 (a) and Figure 3 (b) The result after merging calculations. To avoid the influence of dynamic targets, Figure 4 (b) Combined with the dynamic target detection results, the calculation is continued. Figure 5 shown.

[0094] In a specific embodiment, segmentation lines of two dynamic areas and a static area in the t-th frame dynamic detection image are respectively constructed according to edge contour lines, the segmentation line of the land bank and the water surface is optimized using the RANSAC algorithm, and the dynamic area is eliminated according to the segmentation line. The solution for obtaining the t-th frame static detection image is:

[0095] S41, establish an image coordinate system with the vertex of the left edge of the image as the origin, the direction parallel to the edge contour as the x-axis, and the direction perpendicular to the edge contour as the y-axis, and assume that the dynamic detection image is I(x, y), where x and y are arbitrary coordinates in the image plane, i.e., pixel points; Figure 6 As shown in (a), in this embodiment, the image coordinate system is constructed with the upper vertex of the left edge of the image as the origin;

[0096] S42. Assume that the starting point of a certain edge segment in the edge contour is (x1, y1) and the end point is (x2, y2), such as Figure 6 As shown in (a), select a straight line x=i, i=0,1,2,..., in the image coordinate system, construct the coordinates (x,y) of the intersection of the straight line x=ii=0,1,2,... and the edge segment. i ), as shown in formula (7),

[0097]

[0098] Expanding formula (7), we get formula (8):

[0099]

[0100] Among them, y i The ordinate represents the edge intersection point;

[0101] At x = i, let the coordinates of the upper edge of the shore be is the set of ordinates of all edge line intersections closest to the upper edge of the image; the coordinates of the lower edge of the shore reflection are is the set of ordinates closest to the bottom edge of the image among all edge line intersections; in this embodiment like Figure 6 (b)

[0102] Let the straight line x=i,i=0,1,2,...,traverse all edge contour lines. The upper edge of the shore, i.e., the dividing line between the sky and the shore, is shown in formula (9).

[0103]

[0104] Among them, f up Indicates the dividing line between the sky and the shore;

[0105] The lower edge of the shore reflection, i.e., the dividing line between the water surface reflection and the non-reflection, is shown in formula (10):

[0106]

[0107] Connect the dividing line of the upper edge of the shore with the dividing line of the lower edge of the shore reflection to obtain the midpoint data, and use the midpoint data to generate the dividing line between the shore and the water surface, as shown in formula (11):

[0108]

[0109] Among them, f mid Indicates the dividing line between the land and the water surface, such as Figure 6 (c)

[0110] S43, slide the midpoint data along the positive direction of the x-axis using a sliding window method, set the window width, continuously slide the window from the left edge of the detection image, use the backward difference method to calculate the backward difference value of each window and the average value of the ordinates of all midpoint data in the previous window, use the backward difference method again to calculate the absolute value of each difference result for the backward difference value, when the absolute value is 0, set the window corresponding to the difference result to be a segment, traverse all windows, and obtain multiple segments:

[0111] In this embodiment, the width of the sliding window is set to 5, the length is equal to the length of the edge of the image in the y-axis direction, and the step size is also set to 5;

[0112] Slide the window from the left edge of the image, and use the backward difference method to calculate the backward difference value of the average value of the vertical coordinates of all midpoint data in each window and the previous window, that is, calculate the difference value between the average value of the vertical coordinates of all midpoint data in each window and the average value of the vertical coordinates of all midpoint data in the window before the window. The backward difference method is to subtract the data of the previous window from the data of the next window; traverse all windows to obtain the backward difference values ​​of all two adjacent windows;

[0113] The backward difference method is used again to calculate the difference result for all the backward difference values ​​obtained. When the absolute value of the difference result is 0, all the corresponding consecutive numbered windows in the difference result are set as a segment.

[0114] Traverse all windows, calculate the difference results, set the windows corresponding to the difference results with absolute values ​​of 0 as segments, and obtain multiple segments. The overall process is as follows Figure 8 As shown;

[0115] S44. Assume that there are N midpoints in a segment, randomly select 2 midpoints to form a straight line, and assume that all midpoints in the segment that fall on the straight line are interior points. Calculate the number of iterations of the RANSAC algorithm, as shown in formula (12):

[0116] r = log(1-p r ) / log(1-h 2 ) (12)

[0117] Among them, r represents the number of iterations, p r represents the confidence, h represents the proportion of internal points in N;

[0118] S45, randomly select two midpoints in a segment again to form a new straight line, recalculate the proportion h of the inner points on the straight line in N, and if h is greater than the previous result, recalculate the number of iterations; stop calculating the proportion when the calculated number of iterations is equal to or less than the number of experiments, and the positions of the two midpoints selected at this time are the best straight line fitting result of the current segment;

[0119] S46. Sort the inner points on the straight line where the two best midpoints are located from left to right, select the first and last inner points among all the inner points, and use them as the starting point and end point of the straight line to complete the water shoreline fitting result of the current segment, traverse all segments, repeat S44-S45, obtain the water shoreline fitting results of all segments, connect the water shoreline fitting results of all segments, and the obtained water shoreline is the optimized dividing line between the land shore and the water surface.

[0120] In this scheme, the image coordinate system is constructed, and the regional segmentation of the sky and the land bank is well completed according to the extracted edge information. For the regional segmentation of the land bank and the water surface, it is necessary to further use the upper edge of the land bank and the lower edge of its reflection on the water surface. Since the two are in a mirror relationship with the water bank as the center, the two are connected and the midpoint of the connection is obtained to generate the water bank line to complete the segmentation of the land bank and the water surface. The segmentation line result is as follows: Figure 7 As shown in the figure, the blue line is the dividing line between the sky and the land bank, and the green line is the dividing line between the water surface reflection and the non-water surface reflection; due to the existence of water surface ripples, the edge data under the reflection will fluctuate, and the obtained midpoint information has a certain degree of deviation from the water shoreline. Therefore, the random sampling consensus algorithm (RANdomSAmple Consensus, RANSAC) is used to generate the water shoreline. This algorithm can effectively exclude outliers and noise data in the midpoint information and obtain a more robust water shoreline generation result. The results are shown in Fig. 9 shown.

[0121] In a specific embodiment, the t-th frame static detection image is input into the ORB-SLAM3 system, feature points are extracted from the t-th frame static detection image, and the dynamic feature points in the feature points are judged and eliminated using the LK pyramid optical flow method and the epipolar geometry constraint method. The solution for obtaining the static feature points is:

[0122] S51, inputting the t-th frame static detection image into the ORB feature extraction module in the ORB-SLAM3 system, extracting feature points of the t-th frame static detection image, and obtaining feature points of the image;

[0123] S52, using the LK pyramid optical flow method, calculate the optical flow consistency of the t-th frame static detection image and the t-1th frame dynamic image, obtain the matching points in the t-1th frame dynamic image that have the same optical flow as the feature points of the t-th frame static detection image and form point pairs, regard the feature points that do not have the same optical flow as the matching points as dynamic feature points and delete them, and the matching points of the two frames are as follows: Fig.14 (b)

[0124] S53, calculating a basic matrix using point pairs having the same optical flow;

[0125] S54. Determine the epipolar line according to the basic matrix and the matching points of the dynamic image of the t-1th frame, calculate the distance between the feature point of the static detection image of the tth frame and its corresponding epipolar line through the epipolar geometry constraint method, compare the distance with the set threshold, if it is greater than the threshold, then the point is regarded as a dynamic feature point and deleted, if it is less than or equal to the threshold, then it is retained, and the feature point finally retained is the final static feature point.

[0126] In this embodiment, the LK pyramid optical flow method is used, which has good robustness for images with large displacements between adjacent frames. By reducing the image resolution, the originally large displacement becomes small enough, while also reducing the interference of noise, thereby improving the adaptability of the algorithm to near-shore scenes.

[0127] Epipolar geometry constraint is a method that achieves camera motion estimation only by matching feature points in two frames of images. This method has good geometric intuitiveness, computational efficiency and matching accuracy. In this scheme, the epipolar geometry constraint method is used to appropriately relax the matching conditions by setting a threshold, which can improve its ability to resist noise in the scene and make it adaptable to near-shore scenes.

[0128] The dynamic feature point extraction results are as follows: Fig.10 and Fig.11 As shown, Fig.10 The yellow squares are dynamic feature points. Fig.11 The green squares in the figure are static feature points. Fig.10 (a) and Fig.11 (a) is the result of using only dynamic feature point elimination strategy. Fig.10 (b) and Fig.11 (b) is the result of adding the dynamic target detection algorithm and the dynamic area elimination scheme and then superimposing the dynamic feature point elimination scheme. Fig.10 and Fig.11It can be seen that this scheme further optimizes the feature point extraction process. In particular, when the dynamic target detection algorithm misses or the dynamic area elimination strategy fails, the dynamic feature point elimination strategy can also eliminate most of the dynamic feature points.

[0129] In a specific embodiment, the static feature points are input into the ORB-SLAM3 system again, and the solution for positioning and mapping the unmanned ship in the near-shore scene is:

[0130] The final acquired image is input into the ORB-SLAM3 system for subsequent positioning and map construction.

[0131] ORB-SLAM3 is a visual SLAM system based on ORB feature points. The system supports vision, vision plus inertial navigation, hybrid maps and multi-sensor fusion. It has the advantages of high stability, fast operation speed, and easy implementation. It can achieve high-precision positioning and map construction in complex static environments. Therefore, the present invention uses this system as the basic framework. ORB-SLAM3 mainly has three threads, namely tracking thread, local mapping thread and loop detection thread. The tracking thread estimates the camera's posture by extracting and matching feature points; the local mapping thread constructs a local map of the key frames obtained by the tracking thread and optimizes it; the loop detection thread detects whether the map has a loop, and performs map fusion according to the detection results to improve the accuracy and effectiveness of map construction.

[0132] This embodiment also provides a near-shore scene visual SLAM system for unmanned ships, such as Fig.12 As shown, including:

[0133] The image preprocessing module includes an image acquisition unit, an image edge detection unit and an image segmentation unit. The image acquisition unit is used to acquire n frames of dynamic detection images of unmanned ship nearshore scenes that include sky, land and water surface; the image edge detection unit is used to perform target detection and adaptive edge detection on the t-th frame of dynamic detection image according to the YOLOv5s network and edge detection algorithm to obtain edge contour lines of various regions; the image segmentation unit is used to construct a segmentation line according to the edge contour line, optimize the segmentation line using the RANSAC algorithm and eliminate the dynamic area according to the segmentation line to obtain the t-th frame of static detection image;

[0134] The feature point extraction module is used to input the static detection image of the tth frame into the ORB-SLAM3 system, extract the feature points of the static detection image of the tth frame, and use the LK pyramid optical flow method and the epipolar geometry constraint method to judge and eliminate the dynamic feature points in the feature points to obtain static feature points;

[0135] The image processing module is an ORB-SLAM3 system, which is used to process static feature points and locate and map the unmanned ship in near-shore scenes.

[0136] Specifically, this system is based on the ORB-SLAM3 system. Fig.13 As shown in the figure, an image preprocessing module is added in front of the ORB-SLAM3 system. First, n frames of dynamic detection images are obtained, and each image is detected using the YOLOv5s network. If a target in the detection list appears, the edge detection algorithm is used to detect the edge of the image to obtain the edge contour line. The region is segmented according to the edge contour line to remove dynamic targets and dynamic regions to ensure the static nature of the running scene. The LK pyramid optical flow method and the epipolar geometry constraint method are added to the ORB feature point extraction module of the original ORB-SLAM3 system to judge and eliminate the dynamic feature points in the feature points to obtain static feature points. Finally, the static feature points are input into the ORB-SLAM3 system. The unmanned ship is positioned and mapped in the near-shore scene.

[0137] The verification process of this system is as follows:

[0138] The present invention firstly uses the dynamic data set in the TUM RGB-D data set published by the Technical University of Munich to conduct experiments to test the accuracy of the system of the present invention; then the performance of the system of the present invention is evaluated through the near-shore real scene image sequence collected by the unmanned ship to verify its operation effect and stability. All experiments are carried out on a computer equipped with an Intel Core i7-10700K CPU, 16G memory and NVIDIA 2070 graphics card:

[0139] 1. Analysis of accuracy results of public datasets

[0140] The accuracy of visual SLAM systems is generally judged by absolute trajectory error (ATE) and relative pose error (RPE) as evaluation criteria. ATE is used to estimate the global consistency of the trajectory, and is evaluated by comparing the absolute distance between the estimated trajectory and the true trajectory; RPE mainly describes the local accuracy within a fixed time interval, and is generally used to measure the drift of the trajectory. This paper uses four classic dynamic datasets in the TUM dataset for testing, namely freiburg3_walking_static, freiburg3_walking_halfsphere, freiburg3_walking_rpy and freiburg3_walking_xyz. For the sake of ease of description, freiburg3 is abbreviated as f3 and walking is abbreviated as w.

[0141] The ORB-SLAM3 system and the system of the present invention are taken as evaluation objects, and are evaluated in three aspects: APE, RPE translation and RPE rotation. The system improvement rate δ is calculated using three parameters (retain four decimal places): root mean square error (RMSE), mean error and standard deviation (SD), as shown in formula (13):

[0142]

[0143] Among them, α represents the value of the corresponding parameter of the ORB-SLAM3 system, and β represents the value of the corresponding parameter of the system of the present invention. The ATE evaluation results are shown in Table 1, the evaluation results of RPE translation are shown in Table 2, and Table 3 is the evaluation results of RPE rotation.

[0144] Table 1 ATE evaluation results of ORB-SLAM3 and the system of the present invention

[0145]

[0146] Table 2 Evaluation results of RPE translation of ORB-SLAM3 and NSN-SLAM

[0147]

[0148] Table 3 Evaluation results of RPE rotation of ORB-SLAM3 and NSN-SLAM

[0149]

[0150]

[0151] It can be seen from the results that the performance of the system of the present invention in dynamic environments is greatly improved compared with ORB-SLAM3, and the δ of ATE-related parameters in some dynamic data sets has reached more than 95%.

[0152] Taking the RMSE of ATE as the evaluation index, the system of the present invention is compared with four classic dynamic scene VSLAM systems: DS-SLAM, Dyna-SLAM, YOLO-SLAM and SEG-SLAM. The specific results are shown in Table 4.

[0153] Table 4 Comparison results of ATE RMSE with classic dynamic scene visual SLAM system

[0154]

[0155] Among them, the bold part in Table 4 represents the best result. It can be seen that the system of the present invention also performs well in dynamic scenes and can effectively reduce the impact of the dynamic environment on the system pose estimation.

[0156] 2. Real scene experiment

[0157] 1) Accuracy of static feature point extraction

[0158] Fig.13 The visual feature point matching results of two consecutive frames are drawn. Fig.14 (a) is the ORB-SLAM3 system, Fig.14 (b) is the system of the present invention. It can be seen that after adding the target detection algorithm, dynamic area elimination strategy and dynamic feature point elimination strategy, although the number of feature points extracted by the system of the present invention is reduced, the accuracy is significantly improved.

[0159] 2) Operation process

[0160] A dataset was created using a real coastal scene image sequence collected by an unmanned ship to test the operation of ORB-SLAM3 and the system of the present invention. The dataset contains two typical dynamic targets: ships and people. After the operation starts, the operation results are recorded at regular intervals, such as Fig.15 As shown, Fig.15 (a) is the operation process of ORB-SLAM3. Fig.15 (b) The operation process of the system of the present invention. It can be seen that after ORB-SLAM3 runs for a period of time, the feature point tracking fails, while the system of the present invention completes the feature point tracking of the whole process.

[0161] 3) Operation results

[0162] After the data set is run, the map window and camera running trajectory generated by ORB-SLAM3 and the system of the present invention are as follows: Fig.16 and 17 As shown. Fig.16 The map window shows the environment sparse map (black dots represent historical landmarks, red dots represent current landmarks) and the camera motion trajectory (green lines represent the camera's motion trajectory, blue boxes represent key frames, and green boxes represent the current frame). Fig.16 (a) is the result of ORB-SLAM3 system. Fig.16 (b) is the result of the system of the present invention; Fig.17 For a clearer camera motion trajectory result, Fig.17 (a) is the trajectory diagram of the ORB-SLAM3 system. Fig.17 (b) is a trajectory diagram of the system of the present invention;

[0163] It can be seen from the figure that the ORB-SLAM3 system fails due to the failure of feature point tracking, while the system of the present invention has a smooth running trajectory, strong feature point tracking capability, good robustness, and can better adapt to near-shore scenes.

[0164] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A nearshore scene visual SLAM method for unmanned ships, characterized in that: include: S1: Obtain n frames of dynamic detection images of the unmanned ship near the shore scene including the sky, the land and the water surface; set the sky and the water surface as dynamic areas, and set the land as a static area; S2: Use the YOLOv5s network to perform target detection on the t-th frame of the dynamic detection image, mark the dynamic targets in the dynamic detection image, and obtain the dynamic target detection result, where t∈(2,n); S3: Use the edge detection algorithm to perform adaptive edge detection on the dynamic detection image of the tth frame, and remove the edges of the dynamic targets according to the dynamic target detection results to obtain the edge contour lines of each area; S4: construct segmentation lines of two dynamic areas and a static area in the dynamic detection image of the t-th frame according to the edge contour lines, optimize the segmentation line between the land bank and the water surface using the RANSAC algorithm, remove the dynamic area according to the segmentation line, and obtain the static detection image of the t-th frame; S5: Input the static detection image of the tth frame into the ORB-SLAM3 system, extract feature points from the static detection image of the tth frame, use the LK pyramid optical flow method and the epipolar geometry constraint method to judge and eliminate the dynamic feature points in the feature points, and obtain static feature points; S6: Input the static feature points into the ORB-SLAM3 system again to locate and map the unmanned boat in the nearshore scene.

2. A nearshore scene visual SLAM method for unmanned ships according to claim 1, characterized in that: S4 constructs the segmentation lines of the two dynamic regions and the static region in the dynamic detection image of the t-th frame according to the edge contour lines, including: S41, establish an image coordinate system with the vertex of the left edge of the image as the origin, the direction parallel to the edge contour as the x-axis, and the direction perpendicular to the edge contour as the y-axis, and set the dynamic detection image as I(x, y), where x and y are arbitrary coordinates in the image plane, i.e., pixel points; S42, let the starting point of a certain edge segment in the edge contour be (x1, y1) and the end point be (x2, y2), select a straight line x=i, i=0,1,2,..., in the image coordinate system, construct the coordinates of the intersection of the straight line x=i, i=0,1,2,..., and the edge segment (x,y i ), as shown in formula (1), Expanding formula (1), we get formula (2): Among them, y i The ordinate represents the edge intersection point; At x = i, let the coordinates of the upper edge of the shore be is the set of ordinates of all edge line intersections closest to the upper edge of the image; the coordinates of the lower edge of the shore reflection are is the set of ordinates closest to the bottom edge of the image among all edge line intersections; Combining x=i, i=0,1,2,..., the straight line traverses all edge contour lines, and the upper edge of the shore, that is, the dividing line between the sky and the shore, is shown in formula (3). Among them, f up Indicates the dividing line between the sky and the shore; The lower edge of the shore reflection, i.e., the dividing line between the water surface reflection and the non-reflection, is shown in formula (4): Connect the dividing line of the upper edge of the shore with the dividing line of the lower edge of the shore reflection to obtain the midpoint data, and use the midpoint data to generate the dividing line between the shore and the water surface, as shown in formula (5): Among them, f mid represents a collection of midpoint data, The vertical coordinate representing the midpoint data.

3. A nearshore scene visual SLAM method for unmanned ships according to claim 3, characterized in that: S4 uses the RANSAC algorithm to optimize the segmentation line between land and water, including: S43, sliding the midpoint data along the positive direction of the x-axis using a sliding window method, setting the window width, sliding the window continuously from the left edge of the detection image, using a backward difference method to calculate the backward difference value of each window and the average value of the ordinates of all midpoint data in the previous window, and using the backward difference method again to calculate the absolute value of each difference result for the backward difference value, when the absolute value is 0, setting the window corresponding to the difference result as a segment, traversing all windows to obtain multiple segments; S44. Assume that there are N midpoints in a segment, randomly select 2 midpoints to form a straight line, and assume that all midpoints in the segment that fall on the straight line are interior points. Calculate the number of iterations of the RANSAC algorithm, as shown in formula (6): r=log(1-p r ) / log(1-h 2 ) (6) Among them, r represents the number of iterations, p r represents the confidence, h represents the proportion of internal points in N; S45, randomly select two midpoints in the segment again to form a new straight line, recalculate the proportion h of the inner points on the straight line in N, and if h is greater than the previous result, recalculate the number of iterations; stop calculating the proportion when the calculated number of iterations is equal to or less than the number of experiments, and the positions of the two midpoints selected at this time are the best straight line fitting result of the current segment; S46. Sort the inner points on the straight line where the two best midpoints are located from left to right, select the first and last inner points among all the inner points, and use them as the starting point and end point of the straight line to complete the water shoreline fitting result of the current segment, traverse all segments, repeat S44-S45, obtain the water shoreline fitting results of all segments, connect the water shoreline fitting results of all segments, and the obtained water shoreline is the optimized dividing line between the land shore and the water surface.

4. The nearshore scene visual SLAM method for unmanned ships according to claim 1 is characterized in that: S5 inputs the static detection image of the tth frame into the ORB-SLAM3 system, extracts feature points from the static detection image of the tth frame, uses the LK pyramid optical flow method and the epipolar geometry constraint method to judge and eliminate the dynamic feature points in the feature points, and obtains static feature points, including: S51, inputting the t-th frame static detection image into the ORB feature extraction module in the ORB-SLAM3 system, extracting feature points of the t-th frame static detection image, and obtaining feature points of the image; S52, using the LK pyramid optical flow method, performing optical flow consistency calculation on the t-th frame static detection image and the t-1-th frame dynamic image, obtaining matching points in the t-1-th frame dynamic image that have the same optical flow as the feature points of the t-th frame static detection image and forming point pairs, and treating feature points that do not have the same optical flow as the matching points as dynamic feature points and deleting them; S53, calculating a basic matrix using point pairs having the same optical flow; S54. Determine the epipolar line according to the basic matrix and the matching points of the dynamic image of the t-1th frame, calculate the distance between the feature point of the static detection image of the tth frame and its corresponding epipolar line through the epipolar geometry constraint method, compare the distance with the set threshold, if it is greater than the threshold, then the point is regarded as a dynamic feature point and deleted, if it is less than or equal to the threshold, then it is retained, and the feature point finally retained is the final static feature point.

5. The nearshore scene visual SLAM method for unmanned ships according to claim 1 is characterized in that: Edge detection algorithms include Canny operator, maximum inter-class variance and maximum entropy segmentation algorithms.

6. A nearshore scene visual SLAM method for unmanned ships according to claim 5, characterized in that: Use the edge detection algorithm to perform adaptive edge detection on the dynamic detection image of the tth frame, and remove the edges of the dynamic targets according to the dynamic target detection results to obtain the edge contours of each area, including: S31, using the Canny operator and the maximum inter-class variance to perform preliminary edge detection on the dynamic areas of the sky and the shore in the detection image to obtain preliminary edge contour lines; S32, optimizing the preliminary edge contour line using a maximum entropy segmentation algorithm to obtain an edge contour line containing a dynamic target; S33, removing the edges of the dynamic targets according to the dynamic target detection result, and obtaining edge contours of each area.

7. A system for implementing the nearshore scene visual SLAM method for unmanned ships as claimed in claim 1, characterized in that: include: The image preprocessing module includes an image acquisition unit, an image edge detection unit and an image segmentation unit. The image acquisition unit is used to acquire n frames of dynamic detection images of unmanned ship nearshore scenes that include sky, land and water surface; the image edge detection unit is used to perform target detection and adaptive edge detection on the t-th frame of dynamic detection image according to the YOLOv5s network and edge detection algorithm to obtain edge contour lines of various regions; the image segmentation unit is used to construct a segmentation line according to the edge contour line, optimize the segmentation line using the RANSAC algorithm and eliminate the dynamic area according to the segmentation line to obtain the t-th frame of static detection image; The feature point extraction module is used to input the static detection image of the tth frame into the ORB-SLAM3 system, extract the feature points of the static detection image of the tth frame, and use the LK pyramid optical flow method and the epipolar geometry constraint method to judge and eliminate the dynamic feature points in the feature points to obtain static feature points; The image processing module is an ORB-SLAM3 system, which is used to process static feature points and locate and map the unmanned ship in near-shore scenes.

Images