Ship offshore distance and angle detection method and system based on machine vision

Abstract

The invention belongs to the technical field of ship navigation, and particularly relates to a ship offshore distance and angle detection method and system based on machine vision, and the method comprises the steps: obtaining and splicing a ship aerial view: symmetrically installing a plurality of fisheye cameras at the two sides of a ship, taking the center point of the detected ship as an original point, building an aerial view angle coordinate system, and carrying out the splicing of the aerial view; after distortion correction is carried out on all the fisheye camera pictures, projecting the fisheye camera pictures to a bird's-eye view angle coordinate system plane to obtain a spliced bird's-eye view; and obtaining the area of a shore from the aerial view through a semantic segmentation model BiseNet, solving the edge of the shore according to the area of the shore, and calculating the ship offshore distance and angle according to the edge of the shore. The system comprises an aerial view splicing module, a semantic segmentation module and a shore distance angle calculation module which are connected in sequence. According to the method, visual input is adopted, and multi-input decoupling errors caused by introduction of laser radar data are avoided.

Description

Technical Field

[0001] The present invention belongs to the technical field of ship navigation, and particularly relates to a method and system for detecting the offshore distance and angle of a ship based on machine vision. Background Art

[0002] When a ship enters or leaves a port and berths, the crew, especially for large ships, needs to timely master the angle and distance between the ship and the shore. Generally, the information is obtained by the crew observing near the edge of the deck and then feeding it back to the crew in the cab, supplemented by a millimeter-wave radar installed on the ship, and the berthing and unberthing tasks can be completed. Currently, lidar or a complex perception system based on lidar and machine vision is usually used to obtain the information of the ship and the shore. However, the lidar method cannot present an intuitive effect, and the fusion algorithm of lidar and vision is complex. Decoupling of multiple input systems is difficult and prone to information packet loss and instability, which affects the perception effect and perception efficiency. Summary of the Invention

[0003] The present invention solves the problems that the existing method for detecting the offshore distance and angle of a ship based on lidar cannot present an intuitive effect, and the fusion algorithm of lidar and vision is prone to information packet loss and instability, affecting the perception effect and perception efficiency, and proposes a method and system for detecting the offshore distance and angle of a ship based on machine vision.

[0004] The technical solutions claimed by the present invention are as follows:

[0005] A method for detecting the offshore distance and angle of a ship based on machine vision, comprising the following steps:

[0006] S1: Obtain and splice an aerial view of the ship, including the following steps:

[0007] S11: A plurality of fisheye cameras are symmetrically installed on both sides of the ship, and the number of fisheye cameras installed on each side is not less than two; the fisheye cameras are calibrated by the checkerboard calibration method;

[0008] S12: Taking the center point of the ship to be detected as the origin, establish an aerial view coordinate system; after the ship to be detected berths, place a plurality of markers with obvious features at a specified position on the shore, and record the coordinates of the markers in the aerial view coordinate system; first correct the distortion of the images taken by the fisheye cameras, and then calculate the projection transformation matrix according to the coordinates of the markers in the aerial view coordinate system and the coordinates in the distorted corrected image coordinate system;

[0009] S13: After the images of all fisheye cameras are corrected for distortion, project them onto the plane of the aerial view coordinate system to obtain the spliced aerial view;

[0010] S2: Obtain the area of the shore from the aerial view, including the following steps:

[0011] S21: Collect several panoramic bird's-eye view images when the actual ship is berthing or unberthing. Use intelligent marking tool software to manually mark the area of the shore to obtain a marked training set.

[0012] S22: Use the marked training set obtained in S21 to train the semantic segmentation model BiseNet. After the iterative training is completed, segment and identify the area of the shore in the bird's-eye view image.

[0013] S23: Use BiseNet to obtain the result of the shore area from the bird's-eye view image.

[0014] S3: Calculation of the distance and angle between the ship and the shore, including the following steps:

[0015] S31: Use a common edge detection algorithm to extract the edge of the shore area.

[0016] S32: Obtain the shortest distance from any point on the ship's side to the shore by calculating the Euclidean distance between two points.

[0017] S33: Based on the edge of the shore extracted in S31, find the tangent line of this edge, and then calculate the angle value between the detected ship and the shore according to the tangent slope and the principle of similar triangles.

[0018] Preferably, in S11, the fisheye camera is installed on the widest and longest main deck of the ship and as close as possible to the hull edge. The effective pixels of the fisheye camera are 5 million pixels, and the resolution is above 2592×1944.

[0019] Preferably, the specific process of establishing the bird's-eye view coordinate system in S12 is as follows: Take the center point of the detected ship as the origin of the bird's-eye view coordinate system, the direction of the ship's central axis as the Y-axis, the direction perpendicular to the Y-axis as the X-axis, the ground where the shore is located as the XOY plane, and the direction perpendicular to the ground as the Z-axis to establish a real-world coordinate system.

[0020] Preferably, in S12, the projection transformation matrix is solved by the Gaussian elimination method or the singular value decomposition method of the optimal axis; the marker is a triangular cone barrel, and the placement principle of the marker is that there are at least four markers in each fisheye camera image.

[0021] Preferably, in the panoramic bird's-eye view image stitched in S13, the overlapping area of two camera images is fused by pixel weighted averaging, and the area that cannot be covered by the camera fields of view at the bow and stern is filled by the method of using similar pixels.

[0022] Preferably, the distortion correction formula for the coordinates in the image coordinate system in S13 is:

[0023] P = Hp

[0024] Where: P represents the point p(x, y) in the image coordinate system, and the point mapped to the bird's-eye view coordinate system is P(X, Y); p represents the point p(x, y) in the image coordinate system; H represents the projection transformation matrix.

[0025] Preferably, the intelligent labeling tool software in S21 includes: AnyLabeling.

[0026] Preferably, the edge detection algorithm in S31 includes the canny edge detection algorithm, and the edge of the shore area in S31 is the edge close to the center of the picture.

[0027] The present invention also provides a ship offshore distance and angle detection system based on machine vision, including a bird's-eye view stitching module, a semantic segmentation module, and an offshore distance and angle calculation module connected in sequence;

[0028] The bird's-eye view stitching module is used to symmetrically install multiple fisheye cameras on both sides of the ship, first correct the distortion of the pictures taken by the fisheye cameras, and project the corrected coordinates onto the bird's-eye view coordinate system plane to obtain the stitched bird's-eye view;

[0029] The semantic segmentation module collects several stitched bird's-eye views during the actual ship berthing and unberthing, and uses intelligent labeling tool software to manually mark the shore area to obtain a labeled training set; then uses the obtained labeled training set to train the semantic segmentation model BiseNet. After the iterative training ends, the shore area in the bird's-eye view is segmented and recognized; finally, the result of the shore area is obtained from the bird's-eye view using BiseNet;

[0030] The offshore distance and angle calculation module uses a common edge detection algorithm to extract the edge of the shore area; and obtains the shortest distance from any point on the ship side to the shore by calculating the Euclidean distance between two points; at the same time, based on the extracted shore edge, find the tangent of this edge, and then according to the tangent slope and the principle of similar triangles, find the angle value between the detected ship and the shore.

[0031] Beneficial effects

[0032] The present invention provides a method and system for detecting the offshore distance and angle of a ship based on machine vision. The method includes symmetrically installing a plurality of fish-eye cameras on both sides of the ship to ensure that the range of the subsequent obtained bird's-eye view meets the target. After the images of all fish-eye cameras are corrected for distortion, they are projected onto the plane of the bird's-eye view coordinate system to obtain a spliced bird's-eye view. Only visual input is used to avoid introducing lidar data and causing decoupling errors in multiple inputs, and to solve the problems that the existing methods for detecting the offshore distance and angle of a ship based on lidar cannot present an intuitive effect, and the fusion algorithm of lidar and vision is prone to information packet loss and instability, affecting the perception effect and perception efficiency. First, several spliced bird's-eye views of the actual ship during berthing and unberthing are collected, and the intelligent marking tool software is used to manually mark the area of the shore to obtain a marked training set. Then, the obtained marked training set is used to train the semantic segmentation model BiseNet. After the iterative training is completed, the area of the shore in the bird's-eye view is segmented and recognized. Finally, the result of the shore area is obtained from the bird's-eye view using BiseNet. By introducing the deep learning semantic segmentation method, the area of the shore can be quickly obtained according to the spliced bird's-eye view. Since the points used in the camera projection transformation are all in the real-world coordinate system, after using BiseNet to obtain the area of the shore and calculate the edge, the real distance from any point on the shore to the ship can be directly calculated by calculating the distance between image points, without the conversion of complex algorithms, and finally the distance and angle from the shore can be directly presented in the bird's-eye view.

[0033] The fish-eye cameras are installed on the main deck of the ship, which is the widest and longest, and as close as possible to the hull edge to obtain a smaller field of view blind area. The effective pixels of the fish-eye cameras are 5 million pixels, and the resolution is above 2592×1944 to achieve the purpose of clearly visible targets within 100 meters on the spliced top view. Description of the Drawings

[0034] Figure 1 It is a flowchart of the method for detecting the offshore distance and angle of a ship based on machine vision according to an embodiment of the present invention.

[0035] Figure 2 It is a flowchart of the method for obtaining and splicing the bird's-eye view of a ship according to an embodiment of the present invention.

[0036] Figure 3 It is a schematic diagram of the installation scheme, perspective, and parameter calibration object of the bird's-eye view camera according to an embodiment of the present invention.

[0037] Figure 4 It is a display of the example results of detecting the shore area by BiseNet according to an embodiment of the present invention.

[0038] Figure 5 It is a schematic diagram of detecting the distance and angle between a ship and the shore according to an embodiment of the present invention. Specific implementation method

[0040] To make the objectives, technical solutions, and advantages of the present invention clearer, the following further describes the technical solutions clearly and completely in conjunction with the accompanying drawings of the present invention.

[0041] The First Group of Embodiments: A Method for Detecting the Offshore Distance and Angle of a Ship Based on Machine Vision

[0042] The present invention provides a method for detecting the offshore distance and angle of a ship based on machine vision, including the following steps:

[0043] S1: Obtain and splice the bird's-eye view of the ship, as Figure 2 shown, including the following steps:

[0044] S11: A plurality of fisheye cameras are symmetrically installed on both sides of the ship, and the number of fisheye cameras installed on each side is not less than two; in a specific embodiment of the present invention, at least two fisheye cameras are installed on each side of the ship, and the horizontal and vertical diagonal field of view angles are both 180°, that is, the fisheye cameras on both sides are symmetrically installed, and the fisheye cameras on the same side are on the same line. The effective pixels of the fisheye cameras are 5 million pixels, and the resolution is above 2592×1944, so as to achieve the purpose of clearly seeing the targets within 100 meters on the spliced top view, and the combined horizontal field of view covers 180 degrees of the ship's side. The cameras are installed on the widest and longest main deck of the ship, as close as possible to the hull edge, so as to obtain a smaller field of view blind area. The camera installation points are such as Figure 3 the points on the edge of the midship.

[0045] The fisheye cameras are calibrated by the checkerboard calibration method. The fisheye cameras have been internally calibrated before being installed on the ship. Usually, the checkerboard calibration method is used to obtain the internal parameter matrix K and the distortion parameter D.

[0046] S12: Taking the center point of the detected ship as the origin, establish a bird's-eye view coordinate system; after the detected ship docks, place a plurality of markers with obvious features at a specified position on the shore, and record the coordinates of the plurality of markers in the bird's-eye view coordinate system; first correct the distortion of the images captured by the fisheye cameras, and then calculate the projection transformation matrix according to the coordinates of the markers in the bird's-eye view coordinate system and the coordinates in the distorted corrected image coordinate system.

[0047] In a specific embodiment of the present invention, taking the center point of the detected ship as the coordinate origin, the middle axis direction of the ship as the Y-axis, the direction perpendicular to the Y-axis as the X-axis, the ground where the shore is located as the XOY plane, and the direction perpendicular to the ground as the Z-axis, establish a real-world coordinate system, that is, the final bird's-eye view coordinate system; because the ship's body size is fixed and the center point of the ship is selected as the origin, the position of the ship in the bird's-eye coordinate system is fixed, and both sides of the ship are parallel to the Y-axis; after the cameras are installed on the ship, use markers with obvious features, such as triangular cones, etc. After the ship docks, place cones at a specified position on the shore (such asFigure 3 As shown in [Figure 0], ensure that there are at least four markers in each camera view, and record the coordinates of the cones in the bird's-eye view coordinate system.

[0048] First, correct the distortion of the images captured by the fish-eye cameras. Then, based on the coordinates of the markers in the bird's-eye view coordinate system and their coordinates in the corrected image coordinate system, calculate the projection transformation matrix H (using Gaussian elimination method for the optimal axis or singular value decomposition method). The projection transformation from the image coordinate system to the top view can be obtained as follows. Let the point p(x, y) in the image coordinate system be mapped to the point P(X, Y) in the bird's-eye view coordinate system:

[0049] P = Hp (1)

[0050] The other side of the ship is also calibrated according to the above steps.

[0051] S13: After correcting the distortion of all fish-eye camera views, project them onto the plane of the bird's-eye view coordinate system to obtain the stitched bird's-eye view; after correcting the distortion of all camera views, project them onto the bird's-eye view plane according to formula (1), and fuse the overlapping areas of two camera views by pixel weighted averaging. To improve the effect of the bird's-eye view, the areas not covered by the bow and stern camera views in the figure can be filled with similar pixels.

[0052] S2: Obtain the shore area from the bird's-eye view, including the following steps:

[0053] S21: Collect several (about 200) stitched bird's-eye views of the actual ship during berthing and unberthing, and use intelligent labeling tool software (such as AnyLabeling) to manually label the shore area to obtain a labeled training set;

[0054] S22: Use the labeled training set obtained in S21 to train the semantic segmentation model BiseNet. After the iterative training is completed, segment and identify the shore area in the bird's-eye view;

[0055] S23: Use BiseNet to obtain the result of the shore area from the bird's-eye view ( Figure 4 );

[0056] S3: Calculate the distance and angle between the ship and the shore, including the following steps:

[0057] S31: Use common edge detection algorithms to extract the edges of the shore area; in the specific embodiment of the present invention, after obtaining the shore area, common edge detection algorithms, such as the canny edge detection algorithm, can be used to extract the edges of the shore area. Generally, the edge close to the center of the picture is considered as the required edge of the shore.

[0058] S32: Obtain the shortest distance from any point on the ship's side to the shore by calculating the Euclidean distance between two points; in a specific embodiment of the present invention, during the calibration process, the coordinates in the bird's-eye view are the same as those in the real scene, and the pixel coordinates of the ship itself are fixed. Therefore, the shortest distance from any point on the ship's side to the shore can be directly obtained by calculating the Euclidean distance between two points, such as Figure 5 the d1 and d2 shown

[0059] S33: Based on the shore edge extracted in S31, find the tangent line of this edge, and then, according to the tangent slope and the principle of similar triangles, find the angle value between the detected ship and the shore; in a specific embodiment of the present invention, after obtaining the shore edge, find the tangent line of this edge. Since the ship's side is perpendicular to the Y-axis, the angle value θ between the detected ship and the shore can be found according to the tangent slope and the principle of similar triangles

[0060] In a specific embodiment of the present invention, the specific process of the actual ship using the method of the present invention is as follows Figure 1 shown. At the same time, extract the frames of multiple fisheye cameras (i.e., the original video frames of the fisheye cameras), perform distortion correction on the pictures in the extracted frames of the fisheye cameras, and perform projection transformation on each fisheye camera picture. The pictures projected onto the bird's-eye view coordinate system plane are spliced and fused to obtain the spliced bird's-eye view picture; obtain the shore area through the semantic segmentation model BiseNet, find the shore edge using a common edge detection algorithm, obtain the shortest distance from any point on the ship's side to the shore by calculating the Euclidean distance between two points, obtain the angle between the actual ship and the shore according to the tangent slope of the shore edge, and obtain the ship's offshore distance and angle in real time through multiple loops

[0061] The second group of embodiments: A ship offshore distance and angle detection system based on machine vision

[0062] This group of embodiments provides a ship offshore distance and angle detection system based on machine vision, including a bird's-eye view splicing module, a semantic segmentation module, and an offshore distance and angle calculation module connected in sequence

[0063] The bird's-eye view splicing module is used to symmetrically install multiple fisheye cameras on both sides of the ship, first perform distortion correction on the pictures taken by the fisheye cameras, and project the corrected coordinates onto the bird's-eye view coordinate system plane to obtain the spliced bird's-eye view

[0064] The semantic segmentation module collects several spliced bird's-eye view pictures of the actual ship during berthing and unberthing, and uses intelligent labeling tool software to manually mark the shore area to obtain a labeled training set; then train the semantic segmentation model BiseNet with the obtained labeled training set. After the iterative training ends, segment and identify the shore area in the bird's-eye view; finally, obtain the result of the shore area from the bird's-eye view using BiseNet

[0065] The shore distance and angle calculation module uses common edge detection algorithms to extract the edges of the shore area; calculates the Euclidean distance between two points to obtain the shortest distance from any point on the ship's side to the shore; and based on the extracted shore edges, finds the tangent of this edge, and then calculates the angle between the detected ship and the shore according to the tangent slope and the principle of similar triangles.

[0066] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them. Although the present invention has been described in detail through the above preferred embodiments, those skilled in the art should understand that various changes can be made in terms of form and details without departing from the scope defined by the claims of the present invention.

Claims

1. A method for detecting the distance and angle of a ship from the shore based on machine vision, characterized in that: The steps include: S1: Acquire and stitch the aerial view of the ship, including the following steps: S11: A plurality of fisheye cameras are symmetrically installed on both sides of the ship, and the number of fisheye cameras installed on each side is not less than two; the fisheye cameras are calibrated by using a chessboard calibration method; S12: Establishing a bird's-eye view coordinate system with the center point of the detected ship as the origin; placing multiple markers with obvious features at designated positions on the shore after the detected ship docks, and recording the coordinates of the multiple markers in the bird's-eye view coordinate system; firstly correcting the distortion of the image taken by the fisheye camera, and then calculating the projection transformation matrix according to the coordinates of the markers in the bird's-eye view coordinate system and the coordinates of the image coordinate system after the distortion correction; S13: After distortion correction, all fisheye camera images are projected onto a bird's-eye view coordinate system plane to obtain a spliced ​​bird's-eye view; S2: Obtaining the shore area from the bird's-eye view, including the following steps: S21: Collect several spliced ​​bird's-eye views of actual ships when they dock or leave the berth, use intelligent marking tool software to manually mark the outbound area, and obtain a marked training set; S22: Use the labeled training set obtained in S21 to train the semantic segmentation model BiseNet. After the iterative training, segment and identify the shore area in the bird's-eye view. S23: Results of obtaining the shore area from the bird’s-eye view using BiseNet; S3: Calculation of the distance and angle between the ship and the shore, including the following steps: S31: using a commonly used edge detection algorithm to extract the edge of the shore area; S32: Obtain the shortest distance from any point on the ship's side to the shore by calculating the Euclidean distance between the two points; S33: Based on the edge of the shore extracted in S31, the tangent line of the edge is calculated, and then the angle between the detected ship and the shore is calculated according to the slope of the tangent line and the principle of similar triangles.

2. The method for detecting the distance and angle of a ship from the shore based on machine vision according to claim 1, characterized in that: The fisheye camera in S11 is installed on the widest and longest main deck of the ship and as close to the edge of the hull as possible. The effective pixel of the fisheye camera is 5 million pixels and the resolution is above 2592×1944.

3. The method for detecting the distance and angle of a ship from the shore based on machine vision according to claim 1 or 2, characterized in that: The specific process of establishing the bird's-eye view coordinate system described in S12 is: take the center point of the detected ship as the origin of the bird's-eye view coordinate system, the central axis direction of the ship as the Y axis, the direction perpendicular to the Y axis as the X axis, the ground where the shore is located as the XOY plane, and the direction perpendicular to the ground as the Z axis to establish a real-world coordinate system.

4. The method for detecting the distance and angle of a ship from the shore based on machine vision according to claim 3 is characterized in that: The projection transformation matrix in S12 is solved by Gaussian elimination method or singular value decomposition method of the optimal axis; the marker is a triangular pyramid barrel, and the placement principle of the marker is: there are at least four markers in each fisheye camera image.

5. The method for detecting the distance and angle of a ship from the shore based on machine vision according to claim 1 or 2, characterized in that: In the bird's-eye view stitched by S13, the overlapping areas of the two camera images are fused by pixel weighted averaging, and the areas that cannot be covered by the bow and stern cameras are supplemented by filling with similar pixels.

6. The method for detecting the distance and angle of a ship from the shore based on machine vision according to claim 5, characterized in that: The deformity correction formula of the coordinates in the image coordinate system in S13 is: P=HP Where: P represents the point p(x,y) in the image coordinate system, which is mapped to the point P(X,Y) in the bird's-eye view coordinate system; p represents the point p(x,y) in the image coordinate system; H represents the projection transformation matrix.

7. The method for detecting the distance and angle of a ship from the shore based on machine vision according to claim 1, characterized in that: The intelligent marking tool software described in S21 includes: AnyLabeling.

8. The method for detecting the distance and angle of a ship from the shore based on machine vision according to claim 1, characterized in that: The edge detection algorithm in S31 includes a canny edge detection algorithm.

9. The method for detecting the distance and angle of a ship from the shore based on machine vision according to claim 1, characterized in that: The edge of the area in the S31 middle bank is the edge close to the center of the picture.

10. A ship's distance from shore and angle detection system based on machine vision, characterized in that: It includes a bird's-eye view splicing module, a semantic segmentation module and a shore distance angle calculation module connected in sequence; The bird's-eye view splicing module is used to symmetrically install multiple fisheye cameras on both sides of the ship, and first correct the distortion of the pictures taken by the fisheye cameras, and then project the corrected coordinates onto the plane of the bird's-eye view coordinate system to obtain a spliced ​​bird's-eye view; The semantic segmentation module collects several spliced ​​bird's-eye views of actual ships when they are berthing or leaving the berth, and uses intelligent marking tool software to manually mark the area beyond the shore to obtain a marked training set; then the semantic segmentation model BiseNet is trained with the obtained marked training set, and after the iterative training is completed, the shore area in the bird's-eye view is segmented and identified; Finally, BiseNet was used to obtain the results of the shore area from the bird's-eye view; The shore distance angle calculation module uses a commonly used edge detection algorithm to extract the edge of the shore area; and obtains the shortest distance from any point on the ship side to the shore by calculating the Euclidean distance between two points; at the same time, based on the extracted edge of the shore, the tangent of this edge is calculated, and then the angle value between the detected ship and the shore is calculated based on the tangent slope and the principle of similar triangles.

Images