Ship positioning and monitoring system and method

Through the information collection and algorithm matching of ship positioning and monitoring system, the problem of insufficient real-time and efficient port ship identification is solved, automatic ship number labeling and continuous tracking are realized, and port traffic safety and monitoring efficiency are improved.

CN113780127BActive Publication Date: 2024-12-03WUHAN UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111008304.9
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-08-30
Publication Date
2024-12-03
Estimated Expiration
2041-08-30

AI Technical Summary

Technical Problem

In the prior art, port ship identification is not very real-time and efficient, resulting in frequent port traffic accidents, making it difficult to achieve fast and real-time dispatch and monitoring.

Method used

The ship positioning and monitoring system is adopted to obtain the ship's AIS information and binocular visual images through the information acquisition module, and the information matching and verification is used to match and verify information, real-time positioning and tracking are achieved.

Benefits of technology

It improves the real-time, accuracy and efficiency of ship identification, can automatically mark ship numbers and continuously track, and enhances the safety and monitoring efficiency of port traffic.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113780127B_ABST
    Figure CN113780127B_ABST
Patent Text Reader

Abstract

The present invention relates to a ship positioning and monitoring system and method, the method comprising: collecting ship AIS information and binocular visual images; matching the received ship AIS information and the binocular visual images, and real-time positioning and tracking of the ship. The present invention can perform video surveillance of ships in the monitoring area of ​​the port, and can automatically mark the ship number of each ship in the monitoring video and continuously track it, and can select the target ship to query static information and dynamic information.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of marine traffic technology, and in particular to a ship positioning and monitoring system and method. Background Art

[0002] With the development of the shipping industry, the number of ships entering and leaving ports has increased dramatically, and port transportation has become increasingly arduous and complex. The safety and efficiency of port transportation has become a global hot issue. In recent years, port ship traffic accidents have remained high, causing serious casualties and property losses. The main reasons are: some ships violate regulations, overload, and over-line accidents; accidents caused by poor crew driving skills, improper operation, or unfamiliarity with the waterway environment; poor port navigation conditions, which increase the difficulty of drivers' operations. At the same time, port congestion caused by improper port dispatching and command has also increased dramatically, greatly reducing the efficiency of port operations and causing serious economic losses.

[0003] In order to reduce the occurrence of port ship traffic accidents, ease the intensity of port ship management, and improve the work efficiency of ports, it is urgent to build a modern port dispatching command and intelligent process. Therefore, it is necessary to establish a port intelligent monitoring and dispatching command system to monitor and command ships entering and leaving the port in real time in the monitoring area of ​​the port, so that supervisors can observe the transmitted navigation information in real time and remind ships to correct their speed and course in time, so as to effectively avoid traffic accidents that may occur in ports and ensure smooth port traffic and effective operation of port operations. At the same time, the establishment of a port intelligent monitoring and dispatching command system is conducive to supervisors to query the static information such as the ship name, ship number, nationality, etc. of the ships in the monitoring area, and conduct video surveillance of suspicious targets; after a traffic accident occurs, they can intuitively judge the sea area where the accident occurred and provide safety guarantees for rescue in a timely manner; at the same time, save the traffic accident video to facilitate the identification and handling of accident responsibilities.

[0004] In the existing technology, the dispatch of ships in the port area still uses the traditional manual identification method, which cannot intelligently and efficiently determine the status of the ship, making it difficult to carry out fast and real-time dispatch. Therefore, how to improve the real-time and efficiency of port ship monitoring is an urgent problem to be solved. Summary of the invention

[0005] In view of this, it is necessary to provide a ship positioning and monitoring system to overcome the problems of low real-time and low efficiency of port ship identification in the prior art.

[0006] The present invention provides a ship positioning and monitoring system, comprising an information collection module and a monitoring center, wherein:

[0007] The information collection module is used to collect AIS information and binocular visual images of each ship at the port;

[0008] The monitoring center is used to match the received ship AIS information with the binocular visual image to locate and track the ship in real time.

[0009] Furthermore, the information collection module includes an AIS information collection device and a binocular collection device installed at the same position on the port shore, wherein:

[0010] The AIS information collection device is used to collect the ship AIS information of each ship, and includes a VHF antenna for receiving the transmitted AIS signal sent by each ship, an external GPS antenna for obtaining the longitude and latitude information of the information collection module, and an AIS transceiver for transmitting the transmitted AIS signal and the longitude and latitude information;

[0011] The binocular acquisition device is used to acquire the binocular visual image reflecting the monitoring image of each ship, and includes a rotating platform, a base for fixing the rotating platform, and a binocular camera supported by the rotating platform.

[0012] The present invention also provides a ship positioning and monitoring method, based on the ship positioning and monitoring system as described above, the method comprises:

[0013] Collect ship AIS information and binocular visual images;

[0014] The received ship AIS information and the binocular visual image are matched with each other to locate and track the ship in real time.

[0015] Furthermore, matching the received ship AIS information with the binocular visual image to locate and track the ship in real time includes:

[0016] According to the binocular visual image, the type of each ship is identified, and the type information corresponding to each ship is determined and marked;

[0017] According to the ship AIS information, determine the real distance information between each ship and the information collection module;

[0018] According to the marked type information and the real distance information, each ship is positioned and tracked in real time.

[0019] Furthermore, the identifying the type of each ship according to the binocular visual image, and determining and marking the type information corresponding to each ship includes:

[0020] Inputting the binocular vision image into a well-trained neural network based on the YOLOv5 algorithm;

[0021] Each ship in the detection image is labeled according to the predicted rectangular frame of the neural network to generate the label type information.

[0022] Furthermore, the network structure of the neural network includes an input layer, a backbone network Backbone layer, a Neck layer and a prediction layer, and the boundary loss function of the neural network is expressed by the following formula:

[0023]

[0024] in, represents the boundary loss function, represents the intersection-over-union ratio of the predicted rectangular frame and the actual rectangular frame, represents the minimum diagonal distance corresponding to the predicted rectangular box, represents the distance between the center point of the predicted rectangular box and the actual rectangular box, It represents a parameter for measuring the consistency of the aspect ratio of the predicted rectangular box, and is expressed by the following formula:

[0025]

[0026] in, and represents the width and height of the actual rectangular frame, and Indicates the width and height of the predicted rectangular box.

[0027] Further, the ship AIS information includes the transmitted AIS signal sent by each ship and the latitude and longitude information of each ship, and the determining the real distance information between each ship and the information collection module according to the ship AIS information includes:

[0028] Determine the east-west distance between each ship and the information collection module according to the longitude information in the transmitted AIS signal and the longitude information in the longitude and latitude information;

[0029] Determine the north-south distance between each ship and the information collection module according to the latitude information in the transmitted AIS signal and the latitude information in the longitude and latitude information;

[0030] Based on the Pythagorean theorem, the real distance information corresponding to each ship is calculated according to the east-west distance and the north-south distance.

[0031] Furthermore, the real-time positioning and tracking of each ship according to the annotation type information and the real distance information includes:

[0032] Based on the SURF feature point matching algorithm, according to the annotation type information of each ship, the predicted distance information between each ship and the information collection module is determined;

[0033] Comparing the predicted distance information with the actual distance information to generate a first comparison result;

[0034] Compare the ship type in the marked type information of each ship with the ship type in the corresponding ship AIS information to generate a second comparison result;

[0035] According to the first comparison result and the second comparison result, the ship AIS information and the predicted target frame in the binocular vision image are matched, and the ship number is marked to obtain a final detection frame.

[0036] Furthermore, the determining of the predicted distance information between each ship and the information acquisition module based on the SURF feature point matching algorithm and the annotation type information of each ship includes:

[0037] Based on the SURF feature point matching algorithm, feature point matching is performed on the left view and the right view of the binocular vision image;

[0038] Combined with the predicted rectangular frame of the annotation type information of each ship, the matched feature points are constrained and screened to determine the disparity information between the left view and the right view;

[0039] The predicted distance information is determined according to the disparity information.

[0040] Furthermore, the method further comprises:

[0041] Get the current frame image in multiple frames of monitoring video;

[0042] Based on the SORT multi-target tracking algorithm, the current frame image and the corresponding previous frame image are processed by a Kalman filter to determine the predicted state and predicted covariance of the final detection frame corresponding to the previous frame image;

[0043] Determine, according to the prediction state and the prediction covariance, an intersection-and-union ratio of each final detection frame of the current frame image and each final detection frame of the previous frame image, and match each final detection frame between the two frames according to the intersection-and-union ratio;

[0044] According to the final detection frame matched to the current frame image, the Kalman filter is updated, the updated state value is output, and the final tracking result is generated.

[0045] Compared with the prior art, the beneficial effects of the present invention include: first, effectively acquiring the AIS information and binocular visual images of each ship at the port, and carrying out ship identification based on various information; then, using the ship AIS information and binocular visual images to match each other, achieving the effect of mutual verification, avoiding inaccurate identification of one party's information, resulting in errors in ship identification, and using mutual supplementation and mutual verification between information to achieve the purpose of improving the real-time, accuracy and efficiency of ship identification. In summary, the present invention can perform video surveillance of ships in the monitoring area of ​​the port, and can automatically mark the ship number of each ship in the monitoring video and continuously track it, and can select the target ship to query static information and dynamic information. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of a flow chart of an embodiment of a ship positioning and monitoring system provided by the present invention;

[0047] Figure 2 A hardware schematic diagram of a specific embodiment of a ship positioning and monitoring system provided by the present invention;

[0048] Figure 3 A schematic diagram of a flow chart of an embodiment of a ship positioning and monitoring method provided by the present invention;

[0049] Figure 4 The present invention provides Figure 3 A schematic diagram of a flow chart of an embodiment of step S2 in FIG.

[0050] Figure 5 The present invention provides Figure 4 A schematic diagram of a flow chart of an embodiment of step S21;

[0051] Figure 6 The present invention provides Figure 4 A schematic diagram of a flow chart of an embodiment of step S22;

[0052] Figure 7 The present invention provides Figure 4 A schematic diagram of a flow chart of an embodiment of step S23;

[0053] Figure 8 The present invention provides Figure 7 A flow chart of an embodiment of step S231 in FIG.

[0054] Fig. 9 The present invention provides Figure 3 A flowchart of an embodiment of a subsequent step in FIG. DETAILED DESCRIPTION

[0055] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0056] In the description of the present invention, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0057] In the description of the present invention, reference to an "embodiment" means that a particular feature, structure, or characteristic described in conjunction with the embodiment may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive of other embodiments. It is explicitly and implicitly understood by those skilled in the art that the described embodiments may be combined with other embodiments.

[0058] The present invention provides a ship positioning and monitoring system, combined with Figure 1 Come and see, Figure 1 This is a flow chart of an embodiment of the ship positioning and monitoring system provided by the present invention. It combines the ship AIS information and binocular visual images to match and verify information, providing a new idea for further improving the real-time and high efficiency of ship identification and monitoring. The following are detailed descriptions:

[0059] The embodiment of the present invention provides a ship positioning and monitoring system, the ship positioning and monitoring system 000 includes an information collection module 001 and a monitoring center 002, wherein:

[0060] The information collection module 001 is used to collect AIS information and binocular visual images of each ship at the port;

[0061] The monitoring center 002 is used to match the received ship AIS information with the binocular visual image to locate and track the ship in real time.

[0062] In the embodiment of the present invention, firstly, the AIS information and binocular visual images of each ship at the port are effectively acquired, and the ship identification is carried out based on various information; then, the ship AIS information and binocular visual images are matched with each other to achieve the effect of mutual verification, so as to avoid the error of ship identification caused by inaccurate identification of one party's information, and to achieve the purpose of improving the real-time, accuracy and efficiency of ship identification by mutual supplementation and mutual verification of information.

[0063] As a preferred embodiment, the information collection module includes an AIS information collection device and a binocular collection device installed at the same position on the port shore, wherein:

[0064] The AIS information collection device is used to collect the ship AIS information of each ship, and includes a VHF antenna 10 for receiving the transmitted AIS signal sent by each ship, an external GPS antenna 11 for obtaining the longitude and latitude information of the information collection module, and an AIS transceiver 4 for transmitting the transmitted AIS signal and the longitude and latitude information;

[0065] The binocular acquisition device is used to acquire the binocular visual image reflecting the monitoring image of each ship, and includes a rotating platform, a base for fixing the rotating platform, and a binocular camera 1 supported by the rotating platform.

[0066] In the embodiment of the present invention, a VHF antenna and an external GPS antenna are provided in the AIS information collection device to respectively collect the transmitted AIS signals sent by each ship and the latitude and longitude information of the information collection module, and the above two kinds of information are sent through the AIS transceiver; a rotating platform, a base and a binocular camera are provided in the binocular collection device to monitor each ship in the port and obtain binocular visual images.

[0067] As a preferred embodiment, the system further includes an information transmission device for transmitting the AIS information and the binocular visual image to the monitoring center, and includes a network video server 2, a video transmission cable 5, a control line 6, a network cable 7, and an RS232 interface 9. In the embodiment of the present invention, the AIS information and the binocular visual image are effectively sent from the acquisition end to the processing end through the communication connection of the information transmission device.

[0068] As a preferred embodiment, the above system also includes power supply and lightning protection equipment for power supply and protection of the whole system. In the embodiment of the present invention, power supply and lightning protection equipment are provided to ensure the safe operation of the system.

[0069] As a specific example, Figure 2 Come and see, Figure 2The hardware schematic diagram of a specific embodiment of the ship positioning and monitoring system provided by the present invention includes a binocular camera 1, a network video server 2, a computer 3 in a monitoring center, an AIS receiver 4, a video transmission cable 5, a control line 6, a network cable 7, an external hard disk 8, an RS232 interface 9, a VHF antenna 10, and a GPS receiver 11. The binocular camera 1 is connected to the network video server 2 through the video transmission cable 5 and the control line 6; the VHF antenna 10 and the GPS receiver 11 are connected to the AIS receiver 4; the AIS receiver 4 is connected to the computer 3 in the monitoring center through the RS232 interface 9; the network video server 2 and the computer 3 in the monitoring center are connected through the network cable 7; the external hard disk 8 is inserted into the computer 3 in the monitoring center for storing data information.

[0070] In a specific embodiment of the present invention, the AIS information collection device of the port ship includes a VHF antenna 10, an AIS transceiver 4, and an external GPS antenna 11. The AIS transceiver 4 outputs the NMEA-0183 format telegram information from the data interface in a serial (RS-232 or RS-422) manner. Therefore, the computer 3 of the monitoring center does not contain a serial interface, and a corresponding converter is required to convert the serial port into a USB interface; the VHF antenna 10 of the AIS transceiver 4 is used to receive the AIS signal sent by the port ship, and the external GPS antenna 11 is used to obtain the longitude and latitude of the information collection device. After the two types of information are obtained, they will be output by the AIS transceiver 4 in a serial (RS-232 or RS-422) manner from the data interface to the computer of the monitoring center, and then decoded and stored in the database.

[0071] In a specific embodiment of the present invention, the binocular acquisition device of a port ship comprises a base, a rotating platform, and a binocular camera 1. The base is used to fix the rotating platform, and the rotating platform is used to support the binocular camera, and the shooting angle of the binocular camera can be adjusted; the binocular camera transmits the real-time video information collected by the left and right cameras to the embedded system terminal, and the video compression transmission unit of the terminal compresses the video information and sends it to the network video server 2 through the network transmission device, so that the computer 3 of the monitoring center can load and read it.

[0072] Among them, combined Figure 1 , Figure 2 The operating principle of the above system is as follows:

[0073] The AIS information collection device transmits the collected AIS signals and GPS information to the computer 3 of the monitoring center for decoding and entry into the database. At the same time, the binocular camera 1 collects real-time port monitoring video and uploads it to the network video server 2 for reading and storage by the computer 3 of the monitoring center. After obtaining the AIS information and GPS information of the port ship, as well as the real-time port monitoring video taken by the left and right cameras, the monitoring center 002 will complete the matching of the ship AIS signal in the database and the ship image in the monitoring video based on the comparison of the distance relationship between each ship in the binocular image and the PTZ camera and the comparison of the ship type, and mark the ship in the monitoring video with the ship number of the AIS information, and continue to track it with a rectangular frame in the subsequent frames of the video for the staff of the monitoring center to view, so as to facilitate further dispatching and commanding of the port ship. At the same time, the historical information is stored in the external hard disk 8 inserted in the computer 3 of the monitoring center for the processing of post-accident responsibility identification.

[0074] The embodiment of the present invention provides a method for positioning and monitoring a ship, Figure 3 Come and see, Figure 3 The flowchart of an embodiment of the ship positioning and monitoring method provided by the present invention includes steps S1 to S2, wherein:

[0075] In step S1, the ship AIS information and binocular visual images are collected;

[0076] In step S2, the received ship AIS information and the binocular visual image are matched with each other to locate and track the ship in real time.

[0077] In the embodiment of the present invention, firstly, the AIS information and binocular visual images of each ship at the port are effectively acquired, and the ship identification is carried out based on various information; then, the ship AIS information and binocular visual images are matched with each other to achieve the effect of mutual verification, so as to avoid the error of ship identification caused by inaccurate identification of one party's information, and to achieve the purpose of improving the real-time, accuracy and efficiency of ship identification by mutual supplementation and mutual verification of information.

[0078] It should be noted that the ship AIS information includes the AIS signal emitted by each ship and the latitude and longitude information of each ship, wherein the AIS signal includes but is not limited to the ship's positioning information and navigation status information, which is used to feedback the ship's geographic location, navigation parameters and ship information, while the ship information includes but is not limited to the ship type and ship identification number to identify the ship's identity.

[0079] It should be noted that, in combination with the above system, the ship positioning and monitoring method of the embodiment of the present invention, after collecting the AIS information sent by the port ship at a certain moment and the port binocular monitoring video, the computer 3 of the monitoring center uses the ship recognition algorithm based on YOLOv5 to identify and mark the type of the ship in the monitoring video, and compares the recognition result with the ship type of the AIS information in the database; at the same time, the binocular ranging algorithm based on SURF feature point matching is used to calculate the distance between each ship and the information collection device according to the images of the left and right cameras, and the real distance between each ship and the information collection device is calculated according to the longitude and latitude of the information collection device and the longitude and latitude of each ship in the AIS information, and the distance comparison is completed. By comparing the distance relationship with the ship category, the matching of the ship AIS information in the database and the ship image in the monitoring video is completed, and the ship number in the monitoring video is marked with the ship number in the form of a rectangular frame with the ship number of the AIS information, and then the multi-target tracking algorithm based on SORT is used to continuously track with a rectangular frame in the subsequent frames of the video for the staff of the monitoring center 002 to view, so as to facilitate further dispatching and commanding of port ships.

[0080] As a preferred embodiment, Figure 4 Come and see, Figure 4 The present invention provides Figure 3 The flowchart of step S2 in the embodiment of FIG. 1 is a flowchart of step S2, wherein step S2 specifically includes steps S21 to S23, wherein:

[0081] In step S21, the type of each ship is identified according to the binocular visual image, and the type information corresponding to each ship is determined and marked;

[0082] In step S22, the actual distance information between each ship and the information collection module is determined according to the ship AIS information;

[0083] In step S23, each ship is positioned and tracked in real time according to the annotation type information and the real distance information.

[0084] In an embodiment of the present invention, information about the port monitoring area is obtained through binocular visual images and ship AIS information, and the ship number of each ship is automatically marked in the monitoring video using a combination of target detection technology and binocular ranging technology to achieve positioning and tracking.

[0085] As a preferred embodiment, Figure 5 Come and see, Figure 5 The present invention provides Figure 4 The flowchart of step S21 in the embodiment of FIG. 1 is a flowchart of step S21, wherein step S21 specifically includes step S211 to step S212, wherein:

[0086] In step S211, the binocular vision image is input into a well-trained neural network based on the YOLOv5 algorithm;

[0087] In step S212, each ship in the detection image is labeled according to the predicted rectangular frame of the neural network to generate the label type information.

[0088] In the embodiment of the present invention, the ship recognition algorithm based on YOLOv5 has high accuracy and fast detection speed, can achieve real-time effect, and the model occupies little space.

[0089] As a preferred embodiment, the network structure of the neural network includes an input layer, a backbone network Backbone layer, a Neck layer and a prediction end Prediction layer, and the boundary loss function of the neural network is expressed by the following formula:

[0090]

[0091] in, represents the boundary loss function, represents the intersection-over-union ratio of the predicted rectangular frame and the actual rectangular frame, represents the minimum diagonal distance corresponding to the predicted rectangular box, represents the distance between the center point of the predicted rectangular box and the actual rectangular box, It represents a parameter for measuring the consistency of the aspect ratio of the predicted rectangular box, and is expressed by the following formula:

[0092]

[0093] in, and represents the width and height of the actual rectangular frame, and Indicates the width and height of the predicted rectangular box.

[0094] In the embodiment of the present invention, the ship recognition algorithm of YOLOv5 is used as a detector, and its boundary loss function is used to perform effective training to ensure the accuracy of the network.

[0095] In a specific embodiment of the present invention, the ship recognition algorithm based on YOLOv5 can be structurally divided into four modules, namely, the input end, the backbone network Backbone, Neck and the prediction end Prediction. Through the ship recognition algorithm based on YOLOv5, the computer can quickly and in real time detect the ship on each frame of the surveillance video, and mark the type of the ship on the image with a rectangular frame. The ship recognition model based on YOLOv5 is the result of training on a self-made data set. When making the data set, the official AIS ship type is used to mark the data set.

[0096] As a preferred embodiment, Figure 6 Come and see, Figure 6 The present invention provides Figure 4 The flowchart of step S22 in the embodiment of the present invention is as follows, step S22 specifically includes steps S221 to S223, wherein:

[0097] In step S221, the east-west distance between each ship and the information collection module is determined according to the longitude information in the transmitted AIS signal and the longitude information in the longitude and latitude information;

[0098] In step S222, the north-south distance between each ship and the information collection module is determined according to the latitude information in the transmitted AIS signal and the latitude information in the longitude and latitude information;

[0099] In step S223, based on the Pythagorean theorem, the real distance information corresponding to each ship is calculated according to the east-west distance and the north-south distance.

[0100] In the embodiment of the present invention, the transmitted AIS signal and the longitude and latitude information are utilized and combined to effectively determine the real distance information of each ship.

[0101] In a specific embodiment of the present invention, the real distance between each ship and the information collection device is calculated according to the longitude and latitude of the information collection device and the longitude and latitude of each ship in the AIS information, including: calculating the east-west distance between each ship and the information collection device according to the longitude of each ship and the longitude of the information collection device; calculating the north-south distance between each ship and the information collection device according to the latitude of each ship and the latitude of the information collection device; and calculating the real distance between each ship and the information collection device according to each east-west distance and the corresponding north-south distance in combination with the Pythagorean theorem.

[0102] As a preferred embodiment, Figure 7 Come and see, Figure 7 The present invention provides Figure 4The flowchart of step S23 in the embodiment of FIG. 1 is a flowchart of step S23, wherein step S23 specifically includes steps S231 to S234, wherein:

[0103] In step S231, based on the SURF feature point matching algorithm, the predicted distance information between each ship and the information acquisition module is determined according to the annotation type information of each ship;

[0104] In step S232, the predicted distance information is compared with the actual distance information to generate a first comparison result;

[0105] In step S233, the ship type in the marked type information of each ship is compared with the ship type in the corresponding ship AIS information to generate a second comparison result;

[0106] In step S234, according to the first comparison result and the second comparison result, the ship AIS information and the predicted target frame in the binocular vision image are matched, and the ship number is marked to obtain a final detection frame.

[0107] In the embodiment of the present invention, multiple information is combined to perform multi-faceted mutual verification and comparison, thereby ensuring the accuracy of the final detection frame.

[0108] Among them, when the error between the predicted distance information and the actual distance information is less than a preset threshold, the predicted distance information is judged to be consistent with the actual distance information, and a first comparison result is generated, which indicates that the predicted distance information in the binocular vision image is consistent with the actual distance information determined by the ship AIS signal, and the two information verify each other, thereby improving accuracy.

[0109] Among them, when the ship type in the marked type information is consistent with the ship type in the ship AIS information, a second comparison result is generated, indicating that the ship type identified by the binocular visual image is consistent with the ship type determined by the ship AIS signal, and the two information verify each other, further improving the accuracy.

[0110] Among them, multi-faceted matching and verification are carried out based on distance information and ship type to fully ensure the accuracy of ship number marking and the accuracy of the final detection frame.

[0111] In a specific embodiment of the present invention, when the binocular vision image is matched with the AIS information, the AIS information of the ship in the database and the image of the ship in the surveillance video are matched by comparing the distance relationship and the ship category respectively, and the ship number in the AIS information is used to mark the ship in the video. Figure 4 As shown, including:

[0112] The difference between the actual distance between the ship and the information collection front end and the ship distance measured by binocular ranging are compared with a preset threshold; the ship type in the received AIS signal is compared with the detection result of the ship recognition algorithm based on YOLOv5; if the difference between the actual distance between the ship and the front end and the ship distance measured by binocular ranging is below the preset threshold, and the detection result of the ship recognition algorithm based on YOLOv5 is the same as the ship type in the AIS signal, then the AIS information corresponds to the image of the ship in the video, and then the ship number of the ship in the monitoring video can be obtained from the AIS information and marked with a rectangular frame.

[0113] As a preferred embodiment, Figure 8 Come and see, Figure 8 The present invention provides Figure 7 The flowchart of step S231 in the embodiment of FIG. 1 is a flowchart of step S231, wherein step S231 specifically includes steps S2311 to S2313, wherein:

[0114] In step S2311, feature point matching is performed on the left view and the right view of the binocular vision image based on the SURF feature point matching algorithm;

[0115] In step S2312, the predicted rectangular frame of the annotation type information of each ship is combined to perform constraint screening on the matched feature points to determine the disparity information between the left view and the right view;

[0116] In step S2313, the predicted distance information is determined according to the disparity information.

[0117] In an embodiment of the present invention, a binocular ranging algorithm based on SURF feature point matching is used in ship ranging. When matching feature points of left and right views, this method has better matching effect under the condition of brightness changes, and has better performance in blur and operation speed, which greatly reduces the time spent on system operation.

[0118] In a specific embodiment of the present invention, a binocular distance measurement algorithm based on SURF feature point matching can calculate the distance of each ship from the information collection device based on the images taken by the left and right cameras. When matching the feature points of the left and right views, a SURF algorithm with better matching effect under brightness changes and better blur and computing speed is used. When the system is running, the SURF algorithm is first used to match the feature points of the left and right views, and then the ship frame detected by the YOLOv5 network is used to constrain the feature point pairs, so as to quickly and accurately obtain the effective disparity and perform distance calculation and positioning.

[0119] As a preferred embodiment, Fig. 9 Come and see, Fig. 9 The present invention provides Figure 3 The flowchart of the subsequent step 1 of the embodiment, after step S2, further includes steps S3 to S6, wherein:

[0120] In step S3, a current frame image in multiple frames of monitoring video is obtained;

[0121] In step S4, based on the SORT multi-target tracking algorithm, Kalman filter processing is performed on the current frame image and the corresponding previous frame image to determine the predicted state and predicted covariance of the final detection frame corresponding to the previous frame image;

[0122] In step S5, according to the prediction state and the prediction covariance, the intersection-and-union ratio of each final detection frame of the current frame image and each final detection frame of the previous frame image is determined, and according to the intersection-and-union ratio, each final detection frame between the two frames is matched;

[0123] In step S6, the Kalman filter is updated according to the final detection frame matched to the current frame image, and the updated state value is output to generate the final tracking result.

[0124] In an embodiment of the present invention, a multi-target tracking algorithm based on SORT is used for continuous tracking of ships. This algorithm has a simple structure and a fast computing speed and can meet the requirements of online tracking. At the same time, the tracking effect of this algorithm depends on the performance of the detector.

[0125] In a specific embodiment of the present invention, the SORT-based multi-target tracking algorithm can be combined with the detection results of the ship recognition algorithm based on YOLOv5 to mark the ship number and continuously track the ship in the monitoring video. The SORT-based multi-target tracking algorithm can be described in detail as follows: After collecting the monitoring video of the port, first detect the ship on the first frame and mark the ship number with a rectangular frame; when the subsequent frame comes in, first go to the Kalman filter to obtain the state prediction and covariance prediction generated by the detection frame of the previous frame; then, find the IOU of the target state prediction and the detection frame of this frame, and obtain the unique match with the largest IOU (data association part) through the Hungarian algorithm; then, use the target detection frame matched in this frame to update the Kalman tracker, calculate the Kalman gain, state update and covariance update, and output the state update value as the final detection frame result of this frame.

[0126] As a preferred embodiment, the above method can be applied to electronic nautical charts. In the embodiment of the present invention, supervisors can obtain electronic map displays of all ships within the port range in the monitoring center. Through the present invention, supervisors can select monitoring videos of designated areas for viewing, which is convenient for further dispatching and commanding of port ships.

[0127] As a preferred embodiment, the method further includes: querying information on a designated ship in the surveillance video. In the embodiment of the present invention, after completing the matching of the ship AIS information in the database and the ship image in the surveillance video, clicking on the designated ship in the surveillance video can view the static information and dynamic information of the ship, which all come from the AIS information sent by the ship.

[0128] As a preferred embodiment, the method further includes: saving the generated data in each identification process as historical information. In the embodiment of the present invention, after the real-time ship monitoring video is displayed in the computer of the monitoring center, the system will save the tracking results and ship information and upload them to the network video server, so as to facilitate the identification and handling of accident responsibilities after a traffic accident occurs in the port.

[0129] In a specific embodiment of the present invention, in combination with the above system, the specific process of the overall communication method and the data matching method is as follows:

[0130] The system operation steps are:

[0131] The first step is to obtain the AIS information sent by the port ship through the port ship AIS information collection front end, transmit it to the computer of the monitoring center for decoding, and enter it into the database;

[0132] The second step is to obtain the longitude and latitude of the information collection front end through the external GPS antenna in the port ship AIS information collection front end, transmit it to the computer in the monitoring center for decoding, and enter it into the database;

[0133] The third step is to obtain the port binocular monitoring video through the binocular monitoring image acquisition front end of the information acquisition front end, and upload it to the network video server for the monitoring center computer to read and store;

[0134] The fourth step is to complete the processing of AIS information, front-end GPS information and port binocular surveillance video through the computer in the monitoring center, and automatically mark the ship number and continuously track it in the surveillance video;

[0135] The specific steps of data matching completed in the monitoring center are as follows:

[0136] First, the system uses the ship recognition algorithm based on YOLOv5 to identify and mark the types of ships in the surveillance video, and compares the recognition results with the ship types in the AIS information in the database; at the same time, the binocular ranging algorithm based on SURF feature point matching is used to calculate the distance between each ship and the information collection front end according to the images of the left and right cameras, and the real distance between each ship and the information collection front end is calculated according to the longitude and latitude of the information collection front end and the longitude and latitude of each ship in the AIS information, and the distance comparison is completed. The specific comparison process is: compare the ship type in the received AIS signal with the detection result of the ship recognition algorithm based on YOLOv5; subtract the real distance between the ship and the information collection front end and the ship distance measured by binocular ranging, and compare it with the preset threshold; if the detection result of the ship recognition algorithm based on YOLOv5 is the same as the ship type in the AIS signal, and the difference between the real distance between the ship and the front end and the ship distance measured by binocular ranging is below the preset threshold, then the AIS information corresponds to the image of the ship in the video. After completing the matching of the ship AIS information in the database and the ship image in the surveillance video, the ship number in the surveillance video is marked in the form of a rectangular frame using the ship number in the AIS information. Then, a multi-target tracking algorithm based on SORT is used to continuously track the ship with a rectangular frame in subsequent frames of the video for the staff of the monitoring center to view, which facilitates further dispatch and command of port ships.

[0137] The following is a specific application example to better illustrate the technical solution of the present invention:

[0138] After the system is initialized, the electronic chart will display the various cameras in the port. At this time, the supervisor can select the camera to be monitored, and then control the pan / tilt to select the camera's shooting angle. At this time, the computer in the monitoring center will display the monitoring video with the ship number on the monitor and save this data to the hard disk. When observing the monitoring video, the supervisor can click on the ship to be monitored. The computer will display the ship's AIS information, including dynamic information such as the ship's position, heading, speed (vector line), turning speed and the nearest ship encounter distance, as well as static information such as the ship's name, call sign, ship type, length and width. After that, the supervisor can communicate with the ship based on this information for dispatch and command.

[0139] An embodiment of the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the ship positioning and monitoring method as described above is implemented.

[0140] Generally speaking, the computer instructions for implementing the method of the present invention can be carried by any combination of one or more computer-readable storage media. Non-transitory computer-readable storage media can include any computer-readable media except for the signal itself that is temporarily propagating.

[0141] Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or device.

[0142] Computer program code for performing the operation of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages, in particular, Python suitable for neural network computing and platform frameworks based on TensorFlow, PyTorch, etc. can be used. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0143] An embodiment of the present invention further provides a computing device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the ship positioning and monitoring method described above is implemented.

[0144] The computer-readable storage medium and computing device provided according to the above embodiments of the present invention can be implemented with reference to the specific description of the ship positioning and monitoring method as described above according to the present invention, and have similar beneficial effects as the ship positioning and monitoring method as described above, which will not be repeated here.

[0145] The invention discloses a ship positioning and monitoring system and method. Firstly, the AIS information and binocular visual images of each ship at the port are effectively acquired, and the ship identification is carried out based on multiple information. Then, the ship AIS information and the binocular visual images are matched with each other to achieve the effect of mutual verification, so as to avoid the error of ship identification caused by inaccurate identification of one party's information, and to achieve the purpose of improving the real-time, accuracy and efficiency of ship identification by mutual supplementation and mutual verification between information.

[0146] The technical solution of the present invention utilizes the combination of target detection technology and the AIS ship automatic identification system, so that more ship information can be queried in the monitoring video, and the ships can be monitored by video in the monitoring area of ​​the port. At the same time, the ship number of each ship can be automatically marked in the monitoring video and continuously tracked, and the target ship can be selected to query static information and dynamic information, which is convenient for supervisors to dispatch and command port ships, effectively avoid possible traffic accidents in the port, and ensure smooth port traffic and effective operation of port operations.

[0147] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A ship positioning and monitoring system, characterized in that: It includes information collection module and monitoring center, including: The information collection module is used to collect the AIS information and binocular visual images of each ship at the port; the information collection module includes an AIS information collection device and a binocular collection device installed at the same position on the port shore, wherein: the AIS information collection device is used to collect the ship AIS information of each ship, and includes a VHF antenna for receiving the transmitted AIS signal sent by each ship, an external GPS antenna for obtaining the longitude and latitude information of the information collection module, and an AIS transceiver for transmitting the transmitted AIS signal and the longitude and latitude information; the binocular collection device is used to collect the binocular visual image reflecting the monitoring image of each ship, and includes a rotating platform, a base for fixing the rotating platform, and a binocular camera supported by the rotating platform; The monitoring center is used to identify the type of each ship based on the received ship AIS information and the binocular visual image based on the binocular visual image and a preset ship recognition algorithm, determine and mark the corresponding annotation type information of each ship; determine the real distance information between each ship and the information acquisition module based on the ship AIS information; perform feature point matching on the left view and the right view of the binocular visual image based on a preset SURF feature point matching algorithm; perform constraint screening on the matched feature points in combination with the annotation type information of each ship, and determine the disparity information between the left view and the right view; determine the predicted distance information based on the disparity information; subtract the real distance between the ship and the information acquisition front end from the ship predicted distance information measured by binocular ranging, and generate a first comparison result when the difference is less than a preset threshold; compare the ship type in the received AIS signal with the ship type in the annotation type information, and generate a second comparison result when the above two ship types are consistent; match the ship AIS information and the predicted target frame in the binocular visual image according to the first comparison result and the second comparison result, and mark the ship number to obtain a final detection frame.

2. A ship positioning and monitoring method, characterized in that: Based on the ship positioning and monitoring system according to claim 1, the method comprises: Collect ship AIS information and binocular visual images; The received ship AIS information and the binocular visual image are matched with each other to locate and track the ship in real time.

3. The ship positioning and monitoring method according to claim 2, characterized in that: The step of matching the received ship AIS information with the binocular visual image to locate and track the ship in real time includes: According to the binocular visual image, the type of each ship is identified, and the type information corresponding to each ship is determined and marked; According to the ship AIS information, determine the real distance information between each ship and the information collection module; According to the marked type information and the real distance information, each ship is positioned and tracked in real time.

4. The ship positioning and monitoring method according to claim 3, characterized in that: The identifying the type of each ship according to the binocular visual image, and determining and marking the type information corresponding to each ship includes: Inputting the binocular vision image into a well-trained neural network based on the YOLOv5 algorithm; Each ship in the detection image is labeled according to the predicted rectangular frame of the neural network to generate the label type information.

5. The ship positioning and monitoring method according to claim 4, characterized in that: The network structure of the neural network includes an input layer, a backbone network Backbone layer, a Neck layer and a prediction layer, and the boundary loss function of the neural network is expressed by the following formula: in, represents the boundary loss function, represents the intersection-over-union ratio of the predicted rectangular frame and the actual rectangular frame, represents the minimum diagonal distance corresponding to the predicted rectangular box, represents the distance between the center point of the predicted rectangular box and the actual rectangular box, It represents a parameter for measuring the consistency of the aspect ratio of the predicted rectangular box, and is expressed by the following formula: in, and represents the width and height of the actual rectangular frame, and Indicates the width and height of the predicted rectangular box.

6. The ship positioning and monitoring method according to claim 3, characterized in that: The ship AIS information includes the transmitted AIS signal sent by each ship and the latitude and longitude information of each ship. The determining of the real distance information between each ship and the information collection module according to the ship AIS information includes: Determine the east-west distance between each ship and the information collection module according to the longitude information in the transmitted AIS signal and the longitude information in the longitude and latitude information; Determine the north-south distance between each ship and the information collection module according to the latitude information in the transmitted AIS signal and the latitude information in the longitude and latitude information; Based on the Pythagorean theorem, the real distance information corresponding to each ship is calculated according to the east-west distance and the north-south distance.

7. The ship positioning and monitoring method according to claim 2, characterized in that: The method further comprises: Get the current frame image in multiple frames of monitoring video; Based on the SORT multi-target tracking algorithm, the current frame image and the corresponding previous frame image are processed by a Kalman filter to determine the predicted state and predicted covariance of the final detection frame corresponding to the previous frame image; Determine, according to the prediction state and the prediction covariance, an intersection-and-union ratio of each final detection frame of the current frame image and each final detection frame of the previous frame image, and match each final detection frame between the two frames according to the intersection-and-union ratio; According to the final detection frame matched to the current frame image, the Kalman filter is updated, the updated state value is output, and the final tracking result is generated.