A ship automatic observation system and method based on AIS, deep learning and monocular ranging

Abstract

The present invention proposes a ship automatic lookout system and method based on AIS, deep learning and monocular ranging, the system comprising: a sea surface target detection unit: identifying the target and tracking the target through deep learning technology; an AIS data processing unit: processing, screening and calculating the distance and angle between the ship and the other ship after screening the received ship AIS data; a monocular ranging unit: the monocular ranging unit calculates the distance and angle between the other ship and the ship through a monocular ranging method; a video hull and AIS data matching unit: matching the distance and angle between the ship and the other ship calculated by the AIS data processing unit with the calculation result of the monocular ranging unit, and outputting the matching result to a visual video interface; a visual video interface. The present invention deeply combines AIS, deep learning and monocular ranging technology to achieve uninterrupted automatic lookout of ships, especially with a certain observation range in bad weather.

Description

Technical Field

[0001] The present invention relates to the field of intelligent ships, and in particular to an automatic ship observation system and method based on AIS, deep learning and monocular ranging. Background Art

[0002] In the field of smart ships, ship driving still mainly relies on manual lookout. With the maturity of deep learning algorithms for image processing, deep learning technology is applied to the field of smart ship driving to achieve the fusion of AIS information and video images, making the surrounding ship information intuitive and visual, making ship driving more intelligent, convenient and efficient, and greatly enhancing the safety of ships during driving.

[0003] Traditional manual lookout mainly relies on the driver to observe the situation outside the cockpit at regular intervals. However, manual lookout mainly relies on the driver's initiative, and once there is a slight omission, it may cause a ship safety accident. The automatic lookout technology can provide 24-hour uninterrupted all-round lookout to assist the driver in observing the sea surface conditions, and it also has a certain observable range at night. At the same time, it can still work normally in bad weather to assist the ship's driver in observing the sea surface conditions. The algorithm can identify various ships on the sea surface to alert the ship's driver.

[0004] The emergence of deep learning enables computers to identify targets such as ships as accurately as humans, completely eliminating interference from light, ship shaking, etc. The video ranging algorithm can accurately calculate the distance and angle between the ship in the video and the host ship.

[0005] However, there is currently no technology that combines AIS, deep learning and video ranging, i.e. monocular ranging, to form a matching algorithm to assist the crew in observing the situation around the ship during the ship's navigation, thereby increasing the crew's vigilance and greatly improving the safety and reliability of the ship's navigation. Summary of the invention

[0006] In order to make up for the intermittent and non-real-time nature of manual observation, as well as the unclear observation at night and in bad weather, and the narrow visual range that makes it impossible for ship drivers to intuitively and visually understand the information of ships on the sea surface, the present invention provides a ship automatic observation system and method based on AIS, deep learning and monocular ranging. The AIS data is deeply integrated with the distance and angle from the other ship to the ship calculated by the target recognition and monocular ranging method to achieve 24-hour automatic observation. It also has a certain observable range; it reduces the workload of the driver and enhances the safety and reliability of the ship during navigation.

[0007] An automatic ship observation system based on AIS, deep learning and monocular ranging, including:

[0008] Sea surface target detection unit: The sea surface target detection unit uses a camera to shoot the nearby sea surface in real time, captures and shoots sea surface video images, and detects whether there is a ship target in the video. If there is a ship target, it is regarded as an opponent ship, and the target is identified and tracked through deep learning technology;

[0009] AIS data processing unit: The AIS data processing unit includes an AIS data receiving module for receiving ship AIS data, and an AIS data processing module for processing, screening and calculating the received ship AIS data; the received ship AIS data includes the AIS data of the own ship and the other ship; the calculation is to calculate the distance and angle between the two ships based on the AIS data of the own ship and the other ship after screening;

[0010] Monocular ranging unit: the monocular ranging unit obtains the video image containing the information of the other ship from the sea surface target detection unit, and calculates the distance and angle between the other ship and the own ship by a monocular ranging method;

[0011] Video hull and AIS data matching unit: matches the distance and angle between the own ship and the other ship calculated by the AIS data processing unit with the distance and angle between the other ship and the own ship calculated by the monocular ranging unit, and outputs the matching result to the visual video interface;

[0012] Visual video interface: receiving the distance and angle matching results between the other ship and the own ship from the video hull and AIS data matching unit and the video picture of the sea surface target detection unit, and displaying them in real time on the visual video interface.

[0013] Furthermore, the sea surface target detection unit identifies the ship target in the video through the target recognition training module and the target tracking module; the target recognition training module uses the deep learning Yolov5 algorithm to mark and train the moving target; the target tracking module uses the trained Yolov5 algorithm to identify the ship target in the video and track the target, and gives the tracked target frame coordinates and tracking ID.

[0014] Furthermore, the Yolov5 algorithm uses NVIDIA's Deepstream framework for video analysis, using Deepstream's data source reading, video decoding, image preprocessing, model reasoning, prediction result visualization, and target tracking units after the results.

[0015] Furthermore, the AIS data receiving module uses radio equipment to receive real-time ship AIS data within a specified range on the sea surface in a TDMA communication mode; the AIS data content includes static information such as ship identification code, maritime mobile service identification code, call sign, ship name, ship width and ship type, position of the locator antenna on the ship, ship draft, and dynamic information such as ship latitude and longitude, UTC time and true heading.

[0016] Furthermore, the method in which the AIS data processing module processes, screens and calculates the received ship AIS data is as follows:

[0017] S1: Update the AIS information of the ship in real time, record the longitude and latitude, true heading, draft depth and other information of the ship;

[0018] S2: Record the AIS information of all ships except the own ship, filter the AIS data according to the filter conditions, and depict their tracks according to the longitude and latitude data; calculate the distance Da of the other ship relative to the own ship and the angle between the line connecting the two ships and the true north direction of the earth according to the longitude and latitude of the own ship and the longitude and latitude of the other ship; calculate the angle Aa of the other ship relative to the own ship according to the true heading Ts of the own ship;

[0019] S3: For the lost AIS data, the real-time longitude and latitude of the ship is predicted based on the longitude and latitude information and speed information of the ship, and the predicted driving trajectory is drawn.

[0020] Furthermore, the screening conditions include:

[0021] d. Remove data with non-compliant longitude and latitude, including longitude less than 0 or greater than 180 degrees;

[0022] e. Remove data with large jumps in longitude and latitude, such as when the distance between two AIS numbers is greater than N nautical miles;

[0023] f. The AIS track has abnormal turning angles. The ship is traveling in a smooth straight line or curve. If there is a sudden jump, the angle of the track line is calculated. If the angle is less than a certain threshold, the track is abnormal. If an abnormality occurs, the confidence of the track is reduced. When the confidence is reduced to a certain threshold Ta, the track is invalid.

[0024] Furthermore, the monocular ranging method comprises the following steps:

[0025] S1: Calculate the camera installation angle:

[0026] S2: Calculate the distance C from the camera to the bottom edge of the video in the world coordinate system;

[0027] S3: Calculate the longitudinal distance Dy from the other ship's position pt1 to the camera in the world coordinate system;

[0028] S4: Calculate the lateral distance Dx from the other ship's position pt1 to the camera in the world coordinate system;

[0029] S5: Calculate the distance Do and angle Ao from point pt1 to the camera in the world coordinate system using the longitudinal distance Dy and lateral distance Dx calculated by S3-S4.

[0030] Furthermore, the method for calculating the camera installation angle is:

[0031] S1: According to the imaging principle of the camera: According to the camera's built-in parameters, the video resolution WI*HI is obtained, the camera's vertical field of view angle is 2γ, and the camera's focal length is F;

[0032] S2: Take a horizontal line on the longitudinal axis of the video screen as the reference target. The coordinates of the left and right points of this horizontal line are pta and ptb respectively. The coordinates of the center point of this target are ptc. The width of the target in the world coordinate system is W1.

[0033] S3: Calculate the camera's CMOS width Ws by the camera's vertical field of view: Ws = 2Ftanγ;

[0034] S4: Calculate the target imaging width W0 on CMOS according to the camera's field of view, etc.:

[0035] S5: Calculate the actual distance D1 from the lens to the ptc based on S1-S4:

[0036]

[0037] S6: Calculate the camera installation angle based on the camera height Hc and D1:

[0038] Furthermore, the camera height Hc is calculated by the difference between the camera installation absolute height and the draft depth of the ship.

[0039] Furthermore, the method for calculating the distance C from the camera to the bottom edge of the video in the world coordinate system is:

[0040]

[0041] Hc is the camera height, α is the camera installation angle and 2γ is the camera vertical field of view angle.

[0042] Furthermore, the method for calculating the longitudinal distance Dy from the other party's ship position pt1 to the camera in the world coordinate system is:

[0043] S1: In the world coordinate system, the relationship between the camera height Hc and the distance C from the camera to the bottom edge of the video pt0 is:

[0044]

[0045] S2: In the world coordinate system, the relationship between the camera height Hc and the longitudinal distance Dy is:

[0046]

[0047] Dy=Y+C

[0048] S3: In the world coordinate system, the horizontal projection point of the other ship's position pt1 on the longitudinal center axis of the screen is pt3. Find the angle β between the line connecting pt3 and the camera lens and the line connecting the position pt0 of the lower edge of the video and the camera lens:

[0049]

[0050] S4: Calculate the longitudinal distance from the other ship's position pt1 to the camera in the world coordinate system, that is, the distance Dy from pt3 to the camera;

[0051] Where: α is the camera installation angle, 2γ is the camera vertical field of view angle, Py is the distance between pt3 and pt0 on CMOS, F is the camera focal length, Y is the distance from pt3 to pt0, and C is the vertical distance from the camera to the bottom edge of the video.

[0052] Furthermore, the method for calculating the lateral distance Dx from the other ship position pt1 to the camera in the world coordinate system is:

[0053] S1: In the world coordinate system, the horizontal projection point of the other ship's position pt1 on the horizontal center axis of the screen is pt4. Find the actual distance D4 from pt4 to the camera lens:

[0054]

[0055] Hc is the camera height, α is the camera installation angle, and β is the angle between the line connecting pt3 and the camera lens and the line connecting pt0 at the lower edge of the video and the camera lens in the world coordinate system;

[0056] S2: Find the horizontal distance px between pt1 and pt0 on CMOS:

[0057]

[0058] Ws is the CMOS width of the camera, WI*HI is the video resolution of the camera, and W1 is the width of the target in the world coordinate system;

[0059] S3: Based on the camera imaging principle, the following formula is obtained, and then Dx is calculated according to the values ​​of D4 and Px;

[0060]

[0061] F is the focal length of the camera.

[0062] Furthermore, the method for calculating the world coordinate system distance Do and angle Ao of the point pt1 to the camera by using the lateral distance Dx from the other ship position pt1 to the camera and the longitudinal distance Dy from the other ship position pt1 to the camera in the world coordinate system is:

[0063]

[0064]

[0065] An automatic ship surveillance method based on AIS, deep learning and monocular ranging, the steps are as follows:

[0066] S1: Calculate the distance and angle between the own ship and the other ship using AIS data: By obtaining AIS data, filtering AIS data, calculating the ship's AIS track, and calculating the distance and angle between the own ship and the other ship based on the ship's AIS track;

[0067] S2: Calculate the camera angle and update the camera height in real time;

[0068] S3: Calculate the distance and angle between the own ship and the other ship using target recognition and monocular ranging method: First, identify the ship target and obtain the ship target trajectory, and use the camera angle and height in S2 and the monocular ranging module to calculate the distance and angle between the own ship and the other ship;

[0069] S4: Match the distance and angle between the own ship and the other ship calculated by AIS data with the distance and angle between the own ship and the other ship calculated by target recognition and monocular ranging method, and display the matching result on the visualization interface of the system described in claims 1-13.

[0070] The present invention proposes a ship automatic observation system and method based on AIS, deep learning and monocular ranging, which deeply combines AIS, deep learning and monocular ranging technology to realize uninterrupted automatic observation of ships. The present invention uses the deep learning technology of the sea surface target detection unit to perform target recognition, uses the AIS data processing unit to process the AIS data, and calculates the distance and angle between the ship and the other ship after screening, and at the same time, the monocular ranging unit and the sea surface target detection unit use the video recognition target and monocular ranging technology to calculate the distance and angle between the ship and the other ship and perform real-time matching, realizing the deep combination of AIS, deep learning and monocular ranging technology. The visual video interface can continuously display the distance and angle information between the matched ship and the other ship, so that the system can be able to perform 24-hour uninterrupted observation, and can also obtain matching data at night and in bad weather, so that the system has a certain observable range under bad conditions. At the same time, matching AIS information to the video screen can display the surrounding ship information more clearly and intuitively on the visual interface, so that the lookout system can intuitively and accurately obtain information about the ship's surrounding environment; at the same time, the self-service lookout system also reduces the workload of the driver and is easy to install. It only requires a camera with a large field of view to be installed near the cab, and an industrial computer for image processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 A structural diagram of an automatic ship surveillance system based on AIS, deep learning and monocular ranging.

[0072] Figure 2 A flow chart of an automatic ship surveillance algorithm based on AIS, deep learning and monocular ranging.

[0073] Figure 3 Schematic diagram of the monocular ranging method. DETAILED DESCRIPTION

[0074] The present invention is further described below in conjunction with the accompanying drawings and embodiments. The purpose of the present invention is to monitor ships on the sea in real time and display ship-related information, help drivers observe the sea conditions, and achieve 24-hour automatic lookout. The original intention of the present invention is to reduce the workload of manual lookout of ships, make up for the shortcomings of manual lookout and night lookout, and provide an intuitive and visual automatic lookout tool. The present invention provides an automatic lookout algorithm for ships based on AIS, deep learning and monocular ranging, and is applied to the system.

[0075] like Figure 1 As shown, an automatic ship observation system based on AIS, deep learning and monocular ranging includes:

[0076] Sea surface target detection unit: The sea surface target detection unit uses a camera to shoot the nearby sea surface, captures the sea surface video image, and detects whether there is a ship target in the video. If there is a ship target, it is regarded as an opponent ship, and the target is identified and tracked;

[0077] AIS data processing unit: The AIS data processing unit includes an AIS data receiving module for receiving ship AIS data, and an AIS data processing module for processing, screening and calculating the received ship AIS data; the received ship AIS data includes the AIS data of the own ship and the other ship; the calculation is to calculate the distance and angle between the two ships based on the AIS data of the own ship and the other ship after screening;

[0078] Monocular ranging unit: the monocular ranging unit obtains the video image containing the information of the other ship from the sea surface target detection unit, and calculates the distance and angle between the other ship and the own ship by a monocular ranging method;

[0079] Video hull and AIS data matching unit: matches the distance and angle between the own ship and the other ship calculated by the AIS data processing unit with the distance and angle between the other ship and the own ship calculated by the monocular ranging unit, and outputs the matching result to the visual video interface;

[0080] Visual video interface: receiving the distance and angle matching results between the other ship and the own ship from the video hull and AIS data matching unit and the video picture of the sea surface target detection unit, and displaying them in real time on the visual video interface.

[0081] The sea surface target detection unit identifies the ship target in the video through the target recognition training module and the target tracking module; the target recognition training module uses the deep learning Yolov5 algorithm to mark and train the mobile target; the target tracking module uses the trained Yolov5 algorithm to identify and track the ship target in the video, and gives the tracked target frame coordinates and tracking ID.

[0082] Because the current target detection algorithms based on deep learning theory are mainly divided into one-stage series algorithms represented by Yolo, which are characterized by relatively low accuracy but fast detection speed. Two-stage series algorithms represented by faster-rcnn are characterized by high accuracy but slow detection speed. In order to improve the detection efficiency of the algorithm, we adopted the latest Yolov5 deep learning algorithm.

[0083] The Yolov5 algorithm uses NVIDIA's Deepstream framework for video analysis, using Deepstream's data source reading, video decoding, image preprocessing, model inference, prediction result visualization, and target tracking units after the results.

[0084] The AIS data receiving module uses radio equipment to receive real-time ship AIS data within a specified range on the sea surface in a TDMA communication mode; the AIS data content includes static information such as ship identification code, maritime mobile service identification code, call sign, ship name, ship width and ship type, position of the locator antenna on the ship, ship draft, etc., and dynamic information such as ship latitude and longitude, UTC time and true heading.

[0085] AIS information is not 100% accurate and a certain amount of error information may occur. The method by which the AIS data processing module processes, screens and calculates the received ship AIS data is as follows:

[0086] S1: Update the AIS information of the ship in real time, record the longitude and latitude, true heading, draft depth and other information of the ship;

[0087] S2: Record the AIS information of all ships except the own ship, filter the AIS data according to the filter conditions, and depict their tracks according to the longitude and latitude data; calculate the distance Da of the other ship relative to the own ship and the angle between the line connecting the two ships and the true north direction of the earth according to the longitude and latitude of the own ship and the longitude and latitude of the other ship; calculate the angle Aa of the other ship relative to the own ship according to the true heading Ts of the own ship;

[0088] S3: For the lost AIS data, the real-time longitude and latitude of the ship is predicted based on the longitude and latitude information and speed information of the ship, and the predicted driving trajectory is drawn.

[0089] The screening conditions include:

[0090] 1) Remove data with non-compliant longitude and latitude, including longitude less than 0 or greater than 180 degrees;

[0091] 2) Remove data with large jumps in longitude and latitude, such as when the distance between longitude and latitude between two AIS numbers is greater than N nautical miles;

[0092] 3) The AIS track has abnormal angles. The ship is traveling in a smooth straight line or curve. If there is a sudden jump, the angle of the track is calculated. If the angle is less than a certain threshold, the track is abnormal. If an abnormality occurs, the confidence of the track is reduced. When the confidence is reduced to a certain threshold Ta, the track is invalid.

[0093] The premise of monocular ranging is to preset the camera installation height and angle in advance. However, due to the inconvenience of installation and debugging on board, the algorithm must reduce manual intervention as much as possible and perform as much automatic calculation as possible.

[0094] like Figure 3 As shown, the monocular ranging method comprises the following steps:

[0095] S1: Calculate the camera installation angle:

[0096] S2: Calculate the distance C from the camera to the bottom edge of the video in the world coordinate system;

[0097] S3: Calculate the longitudinal distance Dy from the other ship's position pt1 to the camera in the world coordinate system;

[0098] S4: Calculate the lateral distance Dx from the other ship's position pt1 to the camera in the world coordinate system;

[0099] S5: Calculate the distance Do and angle Ao from point pt1 to the camera in the world coordinate system using the longitudinal distance Dy and lateral distance Dx calculated by S3-S4.

[0100] The method to calculate the camera installation angle is:

[0101] S1: According to the imaging principle of the camera: According to the camera's built-in parameters, the video resolution WI*HI is obtained, the camera's vertical field of view angle is 2γ, and the camera's focal length is F;

[0102] S2: Take a horizontal line on the longitudinal axis of the video screen as the reference target. The coordinates of the left and right points of this horizontal line are pta and ptb respectively. The coordinates of the center point of this target are ptc. The width of the target in the world coordinate system is W1.

[0103] S3: Calculate the camera's CMOS width Ws by the camera's vertical field of view: Ws = 2Ftanγ;

[0104] S4: Calculate the target imaging width W0 on CMOS according to the camera's field of view, etc.:

[0105] S5: Calculate the actual distance D1 from the lens to the ptc based on S1-S4:

[0106]

[0107] S6: Calculate the camera installation angle based on the camera height Hc and D1:

[0108] The camera height Hc is calculated by the difference between the camera installation absolute height and the ship's draft depth.

[0109] The method for calculating the distance C from the camera to the bottom edge of the video in the world coordinate system is:

[0110]

[0111] Hc is the camera height, α is the camera installation angle and 2γ is the camera vertical field of view angle.

[0112] The method for calculating the longitudinal distance Dy from the other ship position pt1 to the camera in the world coordinate system is:

[0113] S1: In the world coordinate system, the relationship between the camera height Hc and the distance C from the camera to the bottom edge of the video pt0 is:

[0114]

[0115] S2: In the world coordinate system, the relationship between the camera height Hc and the longitudinal distance Dy is:

[0116]

[0117] Dy=Y+C

[0118] S3: In the world coordinate system, the horizontal projection point of the other ship's position pt1 on the longitudinal center axis of the screen is pt3. Find the angle β between the line connecting pt3 and the camera lens and the line connecting the position pt0 of the lower edge of the video and the camera lens:

[0119]

[0120] S4: Calculate the longitudinal distance from the other ship's position pt1 to the camera in the world coordinate system, that is, the distance Dy from pt3 to the camera;

[0121] Where: α is the camera installation angle, 2γ is the camera vertical field of view angle, Py is the distance between pt3 and pt0 on CMOS, F is the camera focal length, Y is the distance from pt3 to pt0, and C is the vertical distance from the camera to the bottom edge of the video.

[0122] The method for calculating the lateral distance Dx from the other ship's position pt1 to the camera in the world coordinate system is:

[0123] S1: In the world coordinate system, the horizontal projection point of the other ship's position pt1 on the horizontal center axis of the screen is pt4. Find the actual distance D4 from pt4 to the camera lens:

[0124]

[0125] Hc is the camera height, α is the camera installation angle, and β is the angle between the line connecting pt3 and the camera lens and the line connecting pt0 at the lower edge of the video and the camera lens in the world coordinate system;

[0126] S2: Find the horizontal distance px between pt1 and pt0 on CMOS:

[0127]

[0128] Ws is the CMOS width of the camera, WI*HI is the video resolution of the camera, and W1 is the width of the target in the world coordinate system;

[0129] S3: Based on the camera imaging principle, the following formula is obtained, and then Dx is calculated according to the values ​​of D4 and Px;

[0130]

[0131] F is the focal length of the camera.

[0132] The method for calculating the world coordinate system distance Do and angle Ao from point pt1 to the camera using the lateral distance Dx from the other ship position pt1 to the camera and the longitudinal distance Dy from the other ship position pt1 to the camera in the world coordinate system is:

[0133]

[0134]

[0135] The matching is mainly based on the targets detected by deep learning, and the corresponding AIS information is matched for them. The main basis is: the ship distance Do and angle Ao calculated by monocular ranging on the video, and the ship distance Da and Aa calculated by AIS data. The two correspond to each other and give the matching result.

[0136] 1. The matching angle and distance thresholds are adjusted according to the size of Da.

[0137] 2. In order to reduce mismatching, it is necessary to determine whether the heading trend of the new matching result is consistent with the heading trend of the AIS information.

[0138] 3. For ships with the same identification code, once matched, no re-matching will be performed. However, the distance and angle difference will be calculated in real time. If the distance difference or angle difference becomes larger and larger, the matching will be cancelled.

[0139] The matching results are output in real time on the video screen, and the driver can watch the video screen in real time to understand the corresponding information of the corresponding ship.

[0140] The above is the entire process of the ship automatic observation algorithm based on AIS and deep learning technology. Compared with the traditional manual observation method, the present invention has the following technical advantages:

[0141] 1. It can provide 24-hour uninterrupted surveillance.

[0142] 2. It also has a certain observable range at night and in bad weather.

[0143] 3. The information of surrounding ships is more intuitive and visual.

[0144] Reduces the workload of drivers.

[0145] like Figure 2 As shown in FIG. 1 , a method for automatic ship surveillance based on AIS, deep learning and monocular ranging is shown in FIG. 1 , and the steps are as follows:

[0146] S1: Calculate the distance and angle between the own ship and the other ship using AIS data: By obtaining AIS data, filtering AIS data, calculating the ship's AIS track, and calculating the distance and angle between the own ship and the other ship based on the ship's AIS track;

[0147] S2: Calculate the camera angle and update the camera height in real time;

[0148] S3: Calculate the distance and angle between the own ship and the other ship using target recognition and monocular ranging method: First, identify the ship target and obtain the ship target trajectory, and use the camera angle and height in S2 and the monocular ranging module to calculate the distance and angle between the own ship and the other ship;

[0149] S4: Matching the distance and angle between the own ship and the other ship calculated by the AIS data with the distance and angle between the own ship and the other ship calculated by the target recognition and monocular ranging method, and displaying the matching result on the visual interface of the system.

[0150] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. An automatic ship observation system based on AIS, deep learning and monocular ranging, characterized in that: include: Sea surface target detection unit: The sea surface target detection unit uses a camera to shoot the nearby sea surface in real time, captures and shoots sea surface video images, and detects whether there is a ship target in the video. If there is a ship target, it is regarded as an opponent ship, and the target is identified and tracked through deep learning technology; AIS data processing unit: The AIS data processing unit includes an AIS data receiving module for receiving ship AIS data, and an AIS data processing module for processing, screening and calculating the received ship AIS data; the received ship AIS data includes the AIS data of the own ship and the other ship; the calculation is to calculate the distance and angle between the two ships based on the AIS data of the own ship and the other ship after screening; Monocular ranging unit: the monocular ranging unit obtains the video image containing the information of the other ship from the sea surface target detection unit, and calculates the distance and angle between the other ship and the own ship by a monocular ranging method; Video hull and AIS data matching unit: matches the distance and angle between the own ship and the other ship calculated by the AIS data processing unit with the distance and angle between the other ship and the own ship calculated by the monocular ranging unit, and outputs the matching result to the visual video interface; Visual video interface: receiving the distance and angle matching results between the other ship and the own ship from the video hull and AIS data matching unit and the video picture of the sea surface target detection unit, and displaying them in real time on the visual video interface.

2. The automatic ship observation system based on AIS, deep learning and monocular ranging according to claim 1, characterized in that: The sea surface target detection unit identifies the ship target in the video through the target recognition training module and the target tracking module; the target recognition training module uses the deep learning Yolov5 algorithm to mark and train the moving target; The target tracking module uses the trained Yolov5 algorithm to identify and track the ship target in the video, and gives the tracked target frame coordinates and tracking ID.

3. The automatic ship observation system based on AIS, deep learning and monocular ranging according to claim 2, characterized in that: The Yolov5 algorithm uses NVIDIA's Deepstream framework for video analysis, using Deepstream's data source reading, video decoding, image preprocessing, model inference, prediction result visualization, and target tracking units after the results.

4. The automatic ship observation system based on AIS, deep learning and monocular ranging according to claim 1 or 3, characterized in that: The AIS data receiving module uses radio equipment to receive real-time ship AIS data within a specified range on the sea surface in a TDMA communication mode; the AIS data content includes static information such as ship identification code, maritime mobile service identification code, call sign, ship name, ship width and ship type, position of the locator antenna on the ship, ship draft, etc., and dynamic information such as ship latitude and longitude, UTC time and true heading.

5. The automatic ship observation system based on AIS, deep learning and monocular ranging according to claim 1 or 4, characterized in that: The method by which the AIS data processing module processes, screens and calculates the received ship AIS data is as follows: S1: Update the AIS information of the ship in real time, record the longitude and latitude, true heading, draft depth and other information of the ship; S2: Record the AIS information of all ships except the own ship, filter the AIS data according to the filter conditions, and depict their tracks according to the longitude and latitude data; calculate the distance Da of the other ship relative to the own ship and the angle between the line connecting the two ships and the true north direction of the earth according to the longitude and latitude of the own ship and the longitude and latitude of the other ship; calculate the angle Aa of the other ship relative to the own ship according to the true heading Ts of the own ship; S3: For the lost AIS data, the real-time longitude and latitude of the ship is predicted based on the longitude and latitude information and speed information of the ship, and the predicted driving trajectory is drawn.

6. The automatic ship observation system based on AIS, deep learning and monocular ranging according to claim 5, characterized in that: The screening conditions include: a. Remove data with non-compliant longitude and latitude, including longitude less than 0 or greater than 180 degrees; b. Remove data with large changes in longitude and latitude, such as when the distance between two AIS numbers is greater than N nautical miles; c. The AIS track has abnormal turning angles. The ship is traveling in a smooth straight line or curve. If there is a sudden jump, the angle of the track line is calculated. If the angle is less than a certain threshold, the track is abnormal. If an abnormality occurs, the confidence of the track is reduced. When the confidence is reduced to a certain threshold Ta, the track is invalid.

7. The automatic ship observation system based on AIS, deep learning and monocular ranging according to claim 1 or 6, characterized in that: The monocular ranging method comprises the following steps: S1: Calculate the camera installation angle: S2: Calculate the distance C from the camera to the bottom edge of the video in the world coordinate system; S3: Calculate the longitudinal distance Dy from the other ship's position pt1 to the camera in the world coordinate system; S4: Calculate the lateral distance Dx from the other ship's position pt1 to the camera in the world coordinate system; S5: Calculate the distance Do and angle Ao from point pt1 to the camera in the world coordinate system using the longitudinal distance Dy and lateral distance Dx calculated by S3-S4.

8. The automatic ship observation system based on AIS, deep learning and monocular ranging according to claim 7, characterized in that: The method to calculate the camera installation angle is: S1: According to the imaging principle of the camera: According to the camera's built-in parameters, the video resolution WI*HI is obtained, the camera's vertical field of view angle is 2γ, and the camera's focal length is F; S2: Take a horizontal line on the longitudinal axis of the video screen as the reference target. The coordinates of the left and right points of this horizontal line are pta and ptb respectively. The coordinates of the center point of this target are ptc. The width of the target in the world coordinate system is W1. S3: Calculate the camera's CMOS width Ws by the camera's vertical field of view: Ws = 2Ftanγ; S4: Calculate the target imaging width W0 on CMOS according to the camera's field of view, etc.: S5: Calculate the actual distance D1 from the lens to the ptc based on S1-S4: S6: Calculate the camera installation angle based on the camera height Hc and D1:

9. The automatic ship observation system based on AIS, deep learning and monocular ranging according to claim 8, characterized in that: The camera height Hc is calculated by the difference between the camera installation absolute height and the ship's draft depth.

10. The automatic ship observation system based on AIS, deep learning and monocular ranging according to claim 9, characterized in that: The method for calculating the distance C from the camera to the bottom edge of the video in the world coordinate system is: Hc is the camera height, α is the camera installation angle and 2γ is the camera vertical field of view angle.

11. The automatic ship observation system based on AIS, deep learning and monocular ranging according to claim 10, characterized in that: The method for calculating the longitudinal distance Dy from the other ship position pt1 to the camera in the world coordinate system is: S1: In the world coordinate system, the relationship between the camera height Hc and the distance C from the camera to the bottom edge of the video pt0 is: S2: In the world coordinate system, the relationship between the camera height Hc and the longitudinal distance Dy is: Dy=Y+C S3: In the world coordinate system, the horizontal projection point of the other ship's position pt1 on the longitudinal center axis of the screen is pt3. Find the angle β between the line connecting pt3 and the camera lens and the line connecting the position pt0 of the lower edge of the video and the camera lens: S4: Calculate the longitudinal distance from the other ship's position pt1 to the camera in the world coordinate system, that is, the distance Dy from pt3 to the camera; Where: α is the camera installation angle, 2γ is the camera vertical field of view angle, Py is the distance between pt3 and pt0 on CMOS, F is the camera focal length, Y is the distance from pt3 to pt0, and C is the vertical distance from the camera to the bottom edge of the video.

12. The automatic ship observation system based on AIS, deep learning and monocular ranging according to claim 10, characterized in that: The method for calculating the lateral distance Dx from the other ship's position pt1 to the camera in the world coordinate system is: S1: In the world coordinate system, the horizontal projection point of the other ship's position pt1 on the horizontal center axis of the screen is pt4. Find the actual distance D4 from pt4 to the camera lens: Hc is the camera height, α is the camera installation angle, and β is the angle between the line connecting pt3 and the camera lens and the line connecting pt0 at the lower edge of the video and the camera lens in the world coordinate system; S2: Find the horizontal distance px between pt1 and pt0 on CMOS: Ws is the CMOS width of the camera, WI*HI is the video resolution of the camera, and W1 is the width of the target in the world coordinate system; S3: Based on the camera imaging principle, the following formula is obtained, and then Dx is calculated according to the values ​​of D4 and Px; F is the focal length of the camera.

13. The automatic ship observation system based on AIS, deep learning and monocular ranging according to claims 11 and 12, characterized in that: In the world coordinate system, the method for calculating the world coordinate system distance Do and angle Ao from point pt1 to the camera is: The pt1 is the position of the other ship, the Dx is the lateral distance from the other ship's position pt1 to the camera, and the Dy is the longitudinal distance from the other ship's position pt1 to the camera.

14. A method for automatic ship surveillance based on AIS, deep learning and monocular ranging, characterized in that: Here are the steps: S1: Calculate the distance and angle between the own ship and the other ship using AIS data: By obtaining AIS data, filtering AIS data, calculating the ship's AIS track, and calculating the distance and angle between the own ship and the other ship based on the ship's AIS track; S2: Calculate the camera angle and update the camera height in real time; S3: Calculate the distance and angle between the own ship and the other ship using target recognition and monocular ranging method: First, identify the ship target and obtain the ship target trajectory, and use the camera angle and height in S2 and the monocular ranging module to calculate the distance and angle between the own ship and the other ship; S4: Match the distance and angle between the own ship and the other ship calculated by the AIS data with the distance and angle between the own ship and the other ship calculated by target recognition and monocular ranging method, and display the matching result on the visual interface of the system described in one of claims 1-13.

Images