Method for automatically identifying and tracking ship by using camera

Abstract

The invention relates to a method for automatically identifying and tracking a ship by using a camera, which comprises the following steps of: training and evaluating a model, inputting a video frame picture as a parameter into the model to obtain a position of the ship detected by the model in an image, and calculating a pixel coordinate of a center point of the ship according to the length and the width of the detected position of the ship; according to the method, on the basis of the space coordinates of the camera, geodetic coordinates, Gaussian transformation and other methods, the effect of rapidly recognizing the ship in the video of the single camera can be achieved, and the method has the advantages of being high in accuracy and high in accuracy. And the ship identification efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence and geographic information processing, and is specifically applied to the automatic identification and tracking technology of ships on the Yangtze River waterway, belonging to the field of ship monitoring. Background Art

[0002] The Automatic Identification System (AIS) is a new navigation aid system used for maritime safety and communication between ships and shores, and between ships. It is usually composed of a communication controller connected to a VHF communicator, a GPS locator, a ship-borne display, and sensors, and can automatically exchange important information such as ship position, heading, and ship name. While the AIS installed on the ship sends this information outward, it also receives information from other ships within the VHF coverage range, thereby achieving automatic response.

[0003] Maritime ship detection and recognition is mainly divided into two steps: detection and recognition. Detection refers to finding the positions and bounding boxes of all ships in the marine image. Recognition mainly refers to determining the type and other characteristics of each ship (such as length, width, ship name, MMSI, etc.).

[0004] Object tracking algorithms are a type of computer vision algorithm used to track objects or targets in video sequences. Its main goal is to continuously determine and predict the position, motion, and other properties of the target at different time points by analyzing and processing pixel-level data in consecutive frames of the video. It is mainly divided into two categories: generative methods and discriminative methods:

[0005] 1. Generative algorithms model a given target area in the initial frame and search for the part most similar to the model in subsequent frames as the predicted target position. This type of method includes Kalman filtering, particle filtering, and mean-shift, but their tracking accuracy is relatively low.

[0006] 2. The discriminant algorithm regards the target tracking problem as a target detection task in each frame, uses the image features of the tracked target to train a classifier, uses the target area in the image as a positive sample and the background area as a negative sample, and uses the trained classifier to find the optimal solution in subsequent frames. The discriminant method continuously uses the tracking results in each frame to update the classifier during the tracking process.

[0007] The above existing technologies cannot effectively associate the ships detected in the video with the AIS data, which limits their value in practical applications. In the real world, there is an urgent need for a method to associate the ship objects detected in the video with the AIS information, so as to provide video content enhancement function. Summary of the invention

[0008] The purpose of the present invention is to develop a method for automatic ship identification and tracking that combines target detection and AIS system. By analyzing the pixel position of the center point of the ship in the camera image and converting it into the longitude and latitude information of the real world, and comparing it with the data in the AIS system, the mapping between the target in the image and the real ship is achieved.

[0009] In order to achieve the above object, the present invention proposes the following technical solution, the steps of which include:

[0010] Step 1: Screen and annotate the data to form a data set, and divide the data set into a training set, a validation set, and a test set;

[0011] Step 2: Use the training set and validation set for yolov7 model training to obtain the yolov7-based ship detection model and weight file;

[0012] Step 3: Input the test set into the model for detection and output the detection results;

[0013] Step 4: Model evaluation; if the evaluation result is unqualified, repeat steps 1-3 or proceed to the next stage if it is qualified;

[0014] Step 5: Input the video frame as a parameter into the model, and obtain the model to detect the position of the ship in the image, and calculate the pixel coordinates of the center point of the ship according to the length and width of the detected ship position;

[0015] Step 6: Convert the geodetic coordinates of the camera's geographical location into Gaussian plane coordinates through Gaussian forward transformation. The camera's geographical location here refers to the location of the camera's center.

[0016] Step 7: Determine the spatial coordinates of the pixel points on the image according to the focal length, the spatial coordinates of the camera center, and the angle, and then obtain the coordinates of the actual position in the Gaussian plane;

[0017] Step 8: Obtain the corresponding geodetic coordinates by performing an inverse Gaussian transformation on the Gaussian coordinates of the actual position obtained in step 7;

[0018] Step 9: Obtain the longitude and latitude information of all ships within 10 meters of the longitude and latitude in step 7 from the AIS system, find the ship information closest to the longitude and latitude calculated in step 8 by comparing them one by one, and draw the ship information into the video screen through code.

[0019] Step 10: Repeat steps 5, 7, 8 and 9 to realize the tracking function of the ship target in the frame sequence.

[0020] Furthermore, in step 1, the ship video frames are screened and annotated, and divided into training set, validation set, test set, and evaluation set according to a certain ratio; the data is divided according to 10:2:2:1 respectively. At the beginning of training, 1 / 3 of the training set is used for training. After the training, if the evaluation does not meet the expected goal, the training set is increased by 1 / 3 for training. After the training, if the evaluation does not meet the expected goal, the training set is trained in its entirety and the evaluation set data is mixed and trained together. Until the model meets the expected evaluation effect.

[0021] Further, step 2 obtains the coordinates of the horizontal axis x1 and the vertical axis y1 of the target from the upper left corner of the screen, as well as the horizontal axis span x2 and the vertical axis span y2 of the target by checking, so the horizontal axis coordinate of the center point of the target is x=x1+x2 / 2, and the vertical axis coordinate y=y1+y2 / 2;

[0022] Furthermore, step 5 is usually represented by a three-dimensional coordinate, which represents the position of the camera optical center in the three-dimensional space.

[0023] First, we need to obtain the camera's latitude and longitude coordinates, altitude, and pitch angle. These parameters are measured when the camera is installed and then saved in the database.

[0024] Secondly, through the Gauss projection formula, the geodetic coordinates (B, L) and the central meridian longitude L0 are known, and the Gauss plane coordinates (x, y) are calculated as follows:

[0025]

[0026] Where B is latitude, l = L-L0, the unit is radian, is the radius of curvature of the yoke, t = tan(B), η 2 =e 2 cos 2 (B), is the second eccentricity, a is the major semi-axis of the ellipsoid, b is the minor semi-axis, and X is the meridian arc length.

[0027] Finally, by substituting the camera parameters into the formula, we can get the Gaussian plane coordinates of the camera, and adding the altitude to form the three-dimensional space coordinates (X c ,Y c ,Z c ).

[0028] Further, step 7 calculates the Gaussian plane coordinates.

[0029] Assume that the focal length of the camera is f, and the spatial coordinates of the camera center point are (X c ,Y c ,Z c ).

[0030] For each pixel point (u, v) on the image, assuming that u and v in the camera coordinate system represent the horizontal and vertical directions of the pixel point respectively, the vector from the pixel point to the optical center can be expressed as (u, v, f).

[0031] Step 71, calculate the rotation matrix R

[0032] Given the camera's attitude (pitch, yaw and roll), the camera can be described as a rotation matrix R, which can be obtained by the following rotation matrix construction steps:

[0033] Calculate the rotation matrix Rx around the x-axis (roll), where roll represents the roll angle:

[0034]

[0035] Calculate the rotation matrix Ry around the y-axis (pitch), where pitch represents the pitch angle:

[0036]

[0037] Calculate the rotation matrix Rz around the z-axis (yaw), where yaw represents the yaw angle:

[0038]

[0039] Multiply the three rotation matrices to get the final rotation matrix R:

[0040] R=R z *R y *R x

[0041] Step 72: Calculate the spatial coordinates of the pixel points

[0042] By multiplying the vector (u, v, f) of the pixel point on the image by the rotation matrix R, and then adding the spatial coordinates of the camera center (X c ,Y c ,Z c ), the spatial coordinates corresponding to the pixel point can be calculated.

[0043] The specific steps are:

[0044] 1) For pixel point (u,v), construct vector (u,v,f)

[0045] 2) Multiply the vector (u, v, f) by the rotation matrix R to obtain the vector in the camera coordinate system.

[0046] 3) Add the vector in the camera coordinate system to the spatial coordinates of the camera center (X c ,Y c ,Zc ) to get the coordinates (X, Y, Z) of the pixel in the world coordinate system.

[0047] 4) The height information remains unchanged, and the Gaussian plane coordinates of the pixel point are (X, Y). Further, in step 8, the Gaussian coordinates of the actual position obtained in step 7 are transformed by inverse Gaussian transformation to obtain the corresponding geodetic coordinates. The calculation method is as follows:

[0048] Given the Gaussian plane coordinates (x, y) and the specified central meridian longitude L0, calculate the geodetic coordinates (B, L):

[0049]

[0050] in,

[0051] t f =tan(B f ),B f It is the latitude of the bottom point calculated by inverse calculation based on the meridian arc length X.

[0052] By substituting the Gaussian plane coordinates obtained in step 6 into the formula, the geodetic coordinates, that is, the longitude and latitude information, are obtained.

[0053] In the above implementation process, it should be noted that in order to ensure accuracy, all calculation results must retain at least 10 decimal places, otherwise the error will be too large to obtain the correct result.

[0054] The present invention provides a method for automatically identifying and tracking a ship using a camera, which has the following beneficial effects:

[0055] (1) The present invention classifies the data set and explains how to use the training set, further improving the model's ability to quickly adapt to the expected training method, improving the model's training efficiency, and shortening the required training time.

[0056] (2) The present invention adopts methods such as camera-based spatial coordinates, geodetic coordinates, and Gaussian transformation, which can quickly identify ships in a single camera video and improve the efficiency of ship identification.

[0057] (3) The present invention realizes the function of tracking the ship target in the frame sequence. Since the position of the ship is changing while sailing on the river, but the position of the camera is fixed, after calculating the relevant data of a single camera, the subsequent ships entering the camera area can be identified more quickly.

[0058] (4) The present invention establishes a model evaluation cycle program and a camera-based real-time recognition program for ships, which fully considers the complex influence of various factors and improves the information accuracy based on the training model. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 : Flow chart of a specific embodiment of the present invention DETAILED DESCRIPTION

[0060] The purpose of the present invention is to develop a method for automatic ship identification and tracking that combines target detection and AIS system. By analyzing the pixel position of the center point of the ship in the camera image and converting it into the longitude and latitude information of the real world, and comparing it with the data in the AIS system, the mapping between the target in the image and the real ship is achieved.

[0061] In order to achieve the above object, the present invention proposes the following technical solution, the steps of which include:

[0062] Step 1: Screen and annotate the data to form a data set, and divide the data set into a training set, a validation set, and a test set;

[0063] Step 2: Use the training set and validation set for yolov7 model training to obtain the yolov7-based ship detection model and weight file;

[0064] Step 3: Input the test set into the model for detection and output the detection results;

[0065] Step 4: Model evaluation; if the evaluation result is unqualified, repeat steps 1-3 or proceed to the next stage if it is qualified;

[0066] Step 5: Input the video frame as a parameter into the model, and obtain the model to detect the position of the ship in the image, and calculate the pixel coordinates of the center point of the ship according to the length and width of the detected ship position;

[0067] Step 6: Convert the geodetic coordinates of the camera's geographical location into Gaussian plane coordinates through Gaussian forward transformation. The camera's geographical location here refers to the location of the camera's center.

[0068] Step 7: Determine the spatial coordinates of the pixel points on the image according to the focal length, the spatial coordinates of the camera center, and the angle, and then obtain the coordinates of the actual position in the Gaussian plane;

[0069] Step 8: Obtain the corresponding geodetic coordinates by performing an inverse Gaussian transformation on the Gaussian coordinates of the actual position obtained in step 7;

[0070] Step 9: Obtain the longitude and latitude information of all ships within 10 meters of the longitude and latitude in step 7 from the AIS system, find the ship information closest to the longitude and latitude calculated in step 8 by comparing them one by one, and draw the ship information into the video screen through code.

[0071] Step 10: Repeat steps 5, 7, 8 and 9 to realize the tracking function of the ship target in the frame sequence.

[0072] Furthermore, in step 1, the ship video frames are screened and annotated, and divided into training set, validation set, test set, and evaluation set according to a certain ratio; the data is divided according to 10:2:2:1 respectively. At the beginning of training, 1 / 3 of the training set is used for training. After the training, if the evaluation does not meet the expected goal, the training set is increased by 1 / 3 for training. After the training, if the evaluation does not meet the expected goal, the training set is trained in its entirety and the evaluation set data is mixed and trained together. Until the model meets the expected evaluation effect.

[0073] Further, step 2 obtains the coordinates of the horizontal axis x1 and the vertical axis y1 of the target from the upper left corner of the screen, as well as the horizontal axis span x2 and the vertical axis span y2 of the target by checking, so the horizontal axis coordinate of the center point of the target is x=x1+x2 / 2, and the vertical axis coordinate y=y1+y2 / 2;

[0074] Furthermore, step 5 is usually represented by a three-dimensional coordinate, which represents the position of the camera optical center in the three-dimensional space.

[0075] First, we need to obtain the camera's latitude and longitude coordinates, altitude, and pitch angle. These parameters are measured when the camera is installed and then saved in the database.

[0076] Secondly, through the Gauss projection formula, the geodetic coordinates (B, L) and the central meridian longitude L0 are known, and the Gauss plane coordinates (x, y) are calculated as follows:

[0077]

[0078] Where B is latitude, l = L-L0, the unit is radian, is the radius of curvature of the yoke, t = tan(B), η 2 =e 2 cos 2 (B), is the second eccentricity, a is the major semi-axis of the ellipsoid, b is the minor semi-axis, and X is the meridian arc length.

[0079] Finally, by substituting the camera parameters into the formula, we can get the Gaussian plane coordinates of the camera, and adding the altitude to form the three-dimensional space coordinates (X c ,Y c ,Z c ).

[0080] Further, step 6 calculates the camera center position coordinates.

[0081] Assume that the focal length of the camera is f, and the spatial coordinates of the camera center point are (X c ,Y c ,Z c).

[0082] For each pixel point (u, v) on the image, assuming that u and v in the camera coordinate system represent the horizontal and vertical directions of the pixel point respectively, the vector from the pixel point to the optical center can be expressed as (u, v, f).

[0083] Step 61, calculate the rotation matrix R

[0084] Given the camera's attitude (pitch, yaw and roll), the camera can be described as a rotation matrix R, which can be obtained by the following rotation matrix construction steps:

[0085] Calculate the rotation matrix Rx around the x-axis (roll), where roll represents the roll angle:

[0086]

[0087] Calculate the rotation matrix Ry around the y-axis (pitch), where pitch represents the pitch angle:

[0088]

[0089] Calculate the rotation matrix Rz around the z-axis (yaw), where yaw represents the yaw angle:

[0090]

[0091] Multiply the three rotation matrices to get the final rotation matrix R:

[0092] R=R z *R y *R x

[0093] Step 62: Calculate the spatial coordinates of the pixel points

[0094] By multiplying the vector (u, v, f) of the pixel point on the image by the rotation matrix R, and then adding the spatial coordinates of the camera center (X c ,Y c ,Z c ), the spatial coordinates corresponding to the pixel point can be calculated.

[0095] The specific steps are:

[0096] 1) For pixel point (u,v), construct vector (u,v,f)

[0097] 2) Multiply the vector (u, v, f) by the rotation matrix R to obtain the vector in the camera coordinate system.

[0098] 3) Add the vector in the camera coordinate system to the spatial coordinates of the camera center (X c ,Yc ,Z c ) to get the coordinates (X, Y, Z) of the pixel in the world coordinate system.

[0099] 4) The height information remains unchanged, and the Gaussian plane coordinates of the pixel point are (X, Y). Further, in step 8, the Gaussian coordinates of the actual position obtained in step 7 are transformed by inverse Gaussian transformation to obtain the corresponding geodetic coordinates. The calculation method is as follows:

[0100] Given the Gaussian plane coordinates (x, y) and the specified central meridian longitude L0, calculate the geodetic coordinates (B, L):

[0101]

[0102] in,

[0103] t f =tan(B f ),B f It is the latitude of the bottom point calculated by inverse calculation based on the meridian arc length X.

[0104] By substituting the Gaussian plane coordinates obtained in step 6 into the formula, the geodetic coordinates, that is, the longitude and latitude information, are obtained.

[0105] In the above implementation process, it should be noted that in order to ensure accuracy, all calculation results must retain at least 10 decimal places, otherwise the error will be too large to obtain the correct result.

[0106] The present invention provides a method for automatically identifying and tracking a ship using a camera, which has the following beneficial effects:

[0107] (1) The present invention classifies the data set and explains how to use the training set, further improving the model's ability to quickly adapt to the expected training method, improving the model's training efficiency, and shortening the required training time.

[0108] (2) The present invention adopts methods such as camera-based spatial coordinates, geodetic coordinates, and Gaussian transformation, which can quickly identify ships in a single camera video and improve the efficiency of ship identification.

[0109] (3) The present invention realizes the function of tracking the ship target in the frame sequence. Since the position of the ship is changing while sailing on the river, but the position of the camera is fixed, after calculating the relevant data of a single camera, the subsequent ships entering the camera area can be identified more quickly.

[0110] (4) The present invention establishes a model evaluation cycle program and a camera-based real-time recognition program for ships, which fully considers the complex influence of various factors and improves the information accuracy based on the training model.

[0111] The above description of the present invention is obviously not limited to the above-mentioned methods. As long as various improvements are made using the method concept and technical solution of the present invention, or the concept and technical solution of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. A method for automatically identifying and tracking a ship using a camera, comprising the following steps: Step 1: Screen and annotate the data to form a data set, and divide the data set into a training set, a validation set, and a test set; Step 2: Use the training set and validation set for yolov7 model training to obtain the yolov7-based ship detection model and weight file; Step 3: Input the test set into the model for detection and output the detection results; Step 4: Model evaluation; if the evaluation result is unqualified, repeat steps 1-3 or proceed to the next stage if it is qualified; Step 5: Input the video frame as a parameter into the model, and obtain the model to detect the position of the ship in the image, and calculate the pixel coordinates of the center point of the ship according to the length and width of the detected ship position; Step 6: Convert the geodetic coordinates of the camera's geographical location into Gaussian plane coordinates through Gaussian forward transformation. The camera's geographical location here refers to the location of the camera center. Step 7: Determine the spatial coordinates of the pixel points on the image according to the focal length, the spatial coordinates of the camera center, and the angle, and then obtain the coordinates of the actual position in the Gaussian plane; Step 8: Obtain the corresponding geodetic coordinates by performing an inverse Gaussian transformation on the Gaussian coordinates of the actual position obtained in step 7; Step 9: Obtain the longitude and latitude information of all ships within 10 meters of the longitude and latitude in step 7 from the AIS system, find the ship information closest to the longitude and latitude calculated in step 8 by comparing them one by one, and draw the ship information into the video screen through code; Step 10: Repeat steps 5, 7, 8 and 9 to realize the tracking function of the ship target in the frame sequence.

2. According to claim 1, a method for automatically identifying and tracking a ship using a camera is characterized in that: The step 2 obtains the coordinates of the horizontal axis x1 and the vertical axis y1 of the target from the upper left corner of the screen, as well as the horizontal axis span x2 and the vertical axis span y2 of the target by checking. Therefore, the horizontal axis coordinate of the center point of the target is x=x1+x2 / 2, and the vertical axis coordinate is y=y1+y2 / 2.

3. According to the novel neural network echo extrapolation method for heavy rainfall process in a watershed as claimed in claim 1, it is characterized by: The calculation method of step 7 is as follows: Step 71, calculate the rotation matrix R; Given the camera's attitude (pitch, yaw and roll), the camera can be described as a rotation matrix R, which can be obtained by the following rotation matrix construction steps: Calculate the rotation matrix Rx around the x-axis (roll), where roll represents the roll angle: Calculate the rotation matrix Ry around the y-axis (pitch), where pitch represents the pitch angle: Calculate the rotation matrix Rz around the z-axis (yaw), where yaw represents the yaw angle: Multiply the three rotation matrices to get the final rotation matrix R: R=R z *R y *R x Step 72, calculating the spatial coordinates corresponding to the pixel point; By multiplying the vector (u, v, f) of the pixel point on the image by the rotation matrix R, and then adding the spatial coordinates of the camera center (X c ,Y c ,Z c ), the spatial coordinates corresponding to the pixel point can be calculated.

Images