A method for automatically identifying and tracking a ship using a camera

Abstract

The application relates to a method for automatically identifying and tracking a ship by using a camera, which comprises the following steps: model training evaluation, inputting a video frame picture as a parameter into a model, obtaining a position of a ship in an image detected by the model, and calculating a pixel coordinate of a center point of the ship according to a length and a width of the detected ship position; determining a space coordinate corresponding to a pixel point on the image according to a focal length, a space coordinate of a camera center and an angle, and then obtaining a coordinate in an actual position Gaussian plane; and the like. Based on the space coordinate of the camera, the geodetic coordinate and the Gaussian conversion method, the application can quickly identify the ship in a single camera video, and improves the ship identification efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence and geographic information processing, and in particular to the automatic identification and tracking technology of Yangtze River shipping vessels, belonging to the field of ship monitoring. BACKGROUND

[0002] Automatic Identification System (AIS) is a new type of navigation system for maritime safety and communication between ships and shores, ships and ships. It is usually composed of VHF communication machine, GPS positioning instrument, shipborne display and sensor connected to the communication controller, which can automatically exchange important information such as ship position, heading and ship name. The AIS installed on the ship not only sends out these information, but also receives the information of other ships within the VHF coverage range, so as to realize automatic response.

[0003] Maritime ship detection and identification mainly includes two steps: detection and identification. Detection refers to finding all the ship positions and bounding boxes in the image at sea. Identification mainly refers to determining the type and other characteristics (such as length, width, ship name, MMSI, etc.) of each ship.

[0004] Target tracking algorithm is a kind of computer vision algorithm for tracking objects or targets in video sequences. Its main goal is to continuously determine and predict the position, motion and other attributes of the target at different time points by analyzing and processing pixel-level data in consecutive frames of video. It mainly includes generative method and discriminative method:

[0005] 1. The generative algorithm models the given target region in the initial frame, and searches for the most similar part as the predicted target position in the subsequent frame. This method includes Kalman filter, particle filter and mean-shift, etc., but their tracking accuracy is relatively low.

[0006] 2. The discriminative algorithm regards the target tracking problem as a target detection task in each frame, uses the image features of the tracking target to train a classifier, takes the target region in the image as the positive sample and the background region as the negative sample, and uses the trained classifier to find the optimal solution in the subsequent frame. The discriminative method constantly updates the classifier using the tracking result in each frame during the tracking process.

[0007] The above prior art research cannot effectively associate the detected ships in the video with the AIS data, limiting its value in practical applications. In the real world, there is an urgent need for a method to associate the detected ship objects in the video with the AIS information, thereby providing video content enhancement functions. SUMMARY

[0008] The purpose of the present application is to develop a ship automatic identification and tracking method combined with target detection and AIS system. By analyzing the pixel position of the center point of the ship in the camera picture and converting it into real-world latitude and longitude information, and comparing it with the data in the AIS system, the mapping of the target in the image and the real ship is realized.

[0009] In order to achieve the above-mentioned purpose, the present application proposes the following technical scheme, the steps of which include:

[0010] Step 1, data screening and labeling are performed to form a data set, and the data set is divided into a training set, a validation set and a test set;

[0011] Step 2, the training set and the validation set are used for yolov7 model training to obtain a ship detection model based on yolov7 and a weight file;

[0012] Step 3, the test set is input into the model for detection, and the detection result is output;

[0013] Step 4, model evaluation; according to the evaluation result, repeat steps 1-3 if unqualified or enter the next link if qualified;

[0014] Step 5, input the video frame picture as a parameter into the model to obtain the position of the ship in the image detected by the model, and calculate the pixel coordinates of the center point of the ship according to the length and width of the detected ship position;

[0015] Step 6, convert the geodetic coordinates of the geographical position of the camera into Gaussian plane coordinates through Gaussian forward transformation, where the geographical position of the camera refers to the position of the center of the camera;

[0016] Step 7, determine the spatial coordinates corresponding to the pixel points on the image according to the focal length, the spatial coordinates of the camera center and the angle, and then obtain the coordinates in the actual position Gaussian plane;

[0017] Step 8, obtain the geodetic coordinates corresponding to the actual position Gaussian coordinates obtained in step 7 through Gaussian inverse transformation;

[0018] Step 9, obtain the latitude and longitude information of all ships within a 10-meter range of the latitude and longitude of step 7 from the AIS system, find the ship information closest to the latitude and longitude calculated in step 8 by one-by-one comparison, and draw the ship information to the video picture through code.

[0019] Step 10, repeat steps 5, 7, 8 and 9 to realize the tracking function of the ship target in the frame sequence.

[0020] Further, step 1, the ship video frame picture is screened and labeled, and is divided into training set, verification set, test set and evaluation set according to a certain proportion; the data is divided according to 10:2:2:1 respectively, and when starting training, 1 / 3 of the training set is trained, after training, if the evaluation does not meet the expected target, 1 / 3 of the training set is added for training, after training, if the evaluation does not meet the expected target, the training set is trained in its entirety and the data of the evaluation set is mixed together for training. Until the model meets the expected evaluation effect.

[0021] Further, step 2, the coordinates of the horizontal axis x1 and the vertical axis y1 of the target distance picture upper left corner are obtained by checking, and the horizontal axis span x2 and the vertical axis span y2 of the target are obtained, so the horizontal axis coordinate of the target center point is x=x1+x2 / 2, and the vertical axis coordinate y=y1+y2 / 2.

[0022] Further, step 5 is usually represented by a three-dimensional coordinate, representing the position of the camera optical center in three-dimensional space.

[0023] First, to obtain the latitude and longitude coordinates of the camera, the altitude, and the pitch angle, these parameters are measured when the camera is installed, and then saved to the database.

[0024] Second, by the Gauss projection direct formula, the geodetic coordinates (B, L) and the central meridian longitude L0 are known, and the Gauss plane coordinates (x, y) are calculated, and the formula is as follows:

[0025]

[0026] Where B is the latitude, l=L-L0, in radians, is the radius of curvature of the prime vertical circle, t=tan(B), η 2 =e 2 cos 2 (B), is the second eccentricity, a is the major axis of the rotating ellipsoid, b is the minor axis, and X is the meridian arc length.

[0027] Finally, the Gauss plane coordinates of the camera can be obtained by bringing the camera parameters into the formula, and the altitude can be added to form the three-dimensional space coordinates (X c ,Y c ,Z c ).

[0028] Further, step 7 calculates the camera Gauss plane coordinates.

[0029] Suppose the camera focal length is f, and the camera center point space coordinates are (X c ,Y c ,Z c ).

[0030] For each pixel point (u, v) on the image, assuming that u and v represent the horizontal and vertical directions of the pixel point in the camera coordinate system, the vector of the pixel point to the optical center can be represented as (u, v, f).

[0031] Step 71, calculate the rotation matrix R

[0032] Given the pose of the camera (pitch angle, yaw angle, and roll angle), the camera can be described as a rotation matrix R, which can be obtained through the following construction steps of the rotation matrix:

[0033] Calculate the rotation matrix Rx around the x-axis (roll), which represents the roll angle:

[0034]

[0035] Calculate the rotation matrix Ry around the y-axis (pitch), which represents the pitch angle:

[0036]

[0037] Calculate the rotation matrix Rz around the z-axis (yaw), which represents the yaw angle:

[0038]

[0039] Multiply the three rotation matrices to get the final rotation matrix R:

[0040] R = R z * R y * R x

[0041] Step 72, calculate the spatial coordinates corresponding to the pixel point

[0042] By multiplying the vector (u, v, f) of the pixel point on the image by the rotation matrix R, and then adding the spatial coordinates (X c , Y c , Z c ) of the camera center, the spatial coordinates corresponding to the pixel point can be calculated.

[0043] The specific steps are as follows:

[0044] 1) For pixel point (u, v), construct vector (u, v, f)

[0045] 2) Multiply the vector (u, v, f) by the rotation matrix R to get the vector in the camera coordinate system.

[0046] 3) Add the spatial coordinates (X c , Y c , Zc ), to obtain the coordinates (X, Y, Z) of the pixel point in the world coordinate system.

[0047] 4) The height information remains unchanged, and the Gaussian plane coordinates of the pixel point are (X, Y). Further, step 8 obtains the corresponding geodetic coordinates of the actual position by Gaussian inverse transformation, and the calculation method is as follows:

[0048] Given the Gaussian plane coordinates (x, y) and the specified central meridian L0, the geodetic coordinates (B, L) are calculated as follows:

[0049]

[0050] wherein,

[0051] t f = tan (B f ), B f is the foot point latitude calculated according to the meridian arc length X.

[0052] The geodetic coordinates, that is, the latitude and longitude information, are obtained by bringing the Gaussian plane coordinates obtained in step 6 into the formula.

[0053] In the above implementation process, it needs to be noted that, in order to ensure the accuracy, all calculation results need to be kept at least 10 decimal places, otherwise the error will be too large to obtain correct results.

[0054] The method for automatically identifying and tracking a ship using a camera has the following beneficial effects:

[0055] (1) The present application classifies data sets and describes how to use the training set, further improves the model to quickly adapt to the expected training method, improves the training efficiency of the model, shortens the required training time, and improves the training learning efficiency of the model.

[0056] (2) The present application adopts a method based on spatial coordinates, geodetic coordinates, Gaussian transformation, etc. of the camera, which can quickly identify the ship in a single camera video, and improve the ship identification efficiency.

[0057] (3) The present application realizes the tracking function of the ship target in the frame sequence. Since the position of the ship on the river changes, but the position of the camera is fixed, after calculating the relevant data of a single camera, the subsequent ship entering the camera area can be identified more quickly.

[0058] (4) The present application establishes a model evaluation cycle program and a real-time ship identification program based on the camera, fully considers the complex influence of various factors, and improves the information accuracy based on the training model. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 Flow chart of specific embodiments of the present application DETAILED DESCRIPTION

[0060] The purpose of the present application is to develop a ship automatic identification and tracking method combined with target detection and AIS system. By analyzing the pixel position of the center point of the ship in the camera picture and converting it into real-world latitude and longitude information, and comparing it with the data in the AIS system, the mapping of the target in the image and the real ship is realized.

[0061] In order to achieve the above-mentioned purpose, the present application proposes the following technical scheme, the steps of which include:

[0062] Step 1, data screening and labeling are performed to form a data set, and the data set is divided into a training set, a validation set and a test set;

[0063] Step 2, the training set and the validation set are used for yolov7 model training to obtain a ship detection model based on yolov7 and a weight file;

[0064] Step 3, the test set is input into the model for detection, and the detection result is output;

[0065] Step 4, model evaluation; according to the evaluation result, repeat steps 1-3 if unqualified or proceed to the next step if qualified;

[0066] Step 5, input the video frame picture as a parameter into the model to obtain the position of the ship in the image detected by the model, and calculate the pixel coordinates of the center point of the ship according to the length and width of the detected ship position;

[0067] Step 6, convert the geodetic coordinates of the geographic position of the camera into Gaussian plane coordinates through Gaussian forward transformation, where the geographic position of the camera refers to the position of the center of the camera;

[0068] Step 7, determine the spatial coordinates corresponding to the pixel point on the image according to the focal length, the spatial coordinates of the camera center and the angle, and then obtain the coordinates in the actual position Gaussian plane;

[0069] Step 8, obtain the corresponding geodetic coordinates of the actual position Gaussian coordinates obtained in step 7 through Gaussian inverse transformation;

[0070] Step 9, obtain the latitude and longitude information of all ships within a 10-meter range of the latitude and longitude of step 7 from the AIS system, find the ship information closest to the latitude and longitude calculated in step 8 by one-by-one comparison, and draw the ship information to the video picture through code.

[0071] Step 10, repeat steps 5, 7, 8 and 9 to realize the tracking function of the ship target in the frame sequence.

[0072] Further, step 1, the ship video frame picture is screened and labeled, and is divided into training set, verification set, test set and evaluation set according to a certain proportion; the data is divided according to 10:2:2:1 respectively, and when starting training, 1 / 3 of the training set is trained, after training, if the evaluation does not meet the expected target, 1 / 3 of the training set is added for training, after training, if the evaluation does not meet the expected target, the training set is trained in its entirety and the data of the evaluation set is mixed together for training. Until the model meets the expected evaluation effect.

[0073] Further, step 2, the coordinates of the horizontal axis x1 and the vertical axis y1 of the target distance picture upper left corner are obtained by checking, and the horizontal axis span x2 and the vertical axis span y2 of the target are obtained, so the horizontal axis coordinate of the target center point is x=x1+x2 / 2, and the vertical axis coordinate y=y1+y2 / 2.

[0074] Further, step 5 is usually represented by a three-dimensional coordinate, representing the position of the camera optical center in three-dimensional space.

[0075] Firstly, to obtain the latitude and longitude coordinates of the camera, the altitude, and the pitch angle, these parameters are measured when the camera is installed, and then saved to the database.

[0076] Secondly, by the Gauss projection direct formula, the Gauss plane coordinates (x, y) are calculated from the geodetic coordinates (B, L) and the central meridian longitude L0, and the formula is as follows:

[0077]

[0078] Where B is the latitude, l=L-L0, in radians, is the radius of curvature of the prime vertical circle, t=tan(B), η 2 =e 2 cos 2 (B), is the second eccentricity, a is the major axis of the rotating ellipsoid, b is the minor axis, and X is the meridian arc length.

[0079] Finally, the Gauss plane coordinates of the camera can be obtained by bringing the camera parameters into the formula, and the altitude can be added to form the three-dimensional space coordinates (X c , Y c , Z c ).

[0080] Further, step 6 calculates the camera center position coordinates.

[0081] Suppose the focal length of the camera is f, and the camera center point space coordinates are (X c , Y c , Z c).

[0082] For each pixel point (u, v) on the image, assuming that u and v represent the horizontal and vertical directions of the pixel point in the camera coordinate system, the vector of the pixel point to the optical center can be represented as (u, v, f).

[0083] Step 61, calculate the rotation matrix R

[0084] Given the pose of the camera (pitch angle, yaw angle and roll angle), the camera can be described as a rotation matrix R, which can be obtained through the following construction steps of the rotation matrix:

[0085] Calculate the rotation matrix Rx around the x-axis (roll), which represents the roll angle:

[0086]

[0087] Calculate the rotation matrix Ry around the y-axis (pitch), which represents the pitch angle:

[0088]

[0089] Calculate the rotation matrix Rz around the z-axis (yaw), which represents the yaw angle:

[0090]

[0091] Multiply the three rotation matrices to get the final rotation matrix R:

[0092] R = R z * R y * R x

[0093] Step 62, calculate the spatial coordinates corresponding to the pixel point

[0094] By multiplying the vector (u, v, f) of the pixel point on the image by the rotation matrix R, and then adding the spatial coordinates (X c , Y c , Z c ) of the camera center, the spatial coordinates corresponding to the pixel point can be calculated.

[0095] The specific steps are as follows:

[0096] 1) For pixel point (u, v), construct vector (u, v, f)

[0097] 2) Multiply the vector (u, v, f) by the rotation matrix R to get the vector in the camera coordinate system.

[0098] 3) Add the vector in the camera coordinate system to the spatial coordinates (X c , Yc ,Z c ), to obtain the coordinates (X, Y, Z) of the pixel point in the world coordinate system.

[0099] 4) The height information remains unchanged, and the Gaussian plane coordinates of the pixel point are (X, Y). Further, step 8 obtains the corresponding geodetic coordinates of the actual position by Gaussian inverse transformation, and the calculation method is as follows:

[0100] Given the Gaussian plane coordinates (x, y) and the specified central meridian L0, the geodetic coordinates (B, L) are calculated as follows:

[0101]

[0102] wherein,

[0103] t f =tan(B f ), B f is the foot point latitude calculated according to the meridian arc length X.

[0104] The geodetic coordinates, that is, the latitude and longitude information, are obtained by bringing the Gaussian plane coordinates obtained in step 6 into the formula.

[0105] In the above implementation process, it needs to be noted that, in order to ensure accuracy, all calculation results need to be kept at least 10 decimal places, otherwise the error will be too large to obtain correct results.

[0106] The method for automatically identifying and tracking a ship using a camera has the following beneficial effects:

[0107] (1) The present application classifies data sets and describes how to use the training set, further improves the model to quickly adapt to the expected training method, improves the training efficiency of the model, shortens the required training time, and improves the training learning efficiency of the model.

[0108] (2) The present application adopts a method based on camera space coordinates, geodetic coordinates, Gaussian transformation, etc., which can quickly identify ships in a single camera video, improving the ship identification efficiency.

[0109] (3) The present application realizes the tracking function of the ship target in the frame sequence. Since the ship position changes while sailing on the river, but the camera position is fixed, after calculating the relevant data of a single camera, the subsequent ships entering the camera area can be identified more quickly.

[0110] (4) The present application establishes a model evaluation cycle program and a real-time ship identification program based on a camera, fully considers the complex influence of various factors, and improves the information accuracy based on the training model.

[0111] The above description of the present application is only illustrative, and the implementation of the present application is not limited to the above-mentioned modes. Any improvement or direct application of the inventive concept and technical solution to other occasions without modification is within the protection scope of the present application.

Claims

1. A method for automatically identifying and tracking ships using a camera, comprising the following steps: Step 1: Filter and label the data to form a dataset, and divide the dataset into training set, validation set, and test set; Step 2: Use the training set and validation set to train the YOLOv7 model and obtain the ship detection model and weight file based on YOLOv7. Step 3: Input the test set into the model for detection and output the detection results; Step 4: Model evaluation; if the evaluation result is unsatisfactory, repeat steps 1-3; if the result is satisfactory, proceed to the next step. Step 5: Input the video frame as a parameter into the model to obtain the position of the ship in the image detected by the model, and calculate the pixel coordinates of the ship's center point based on the length and width of the detected ship position; Step 6: Convert the geodetic coordinates of the camera's location into Gaussian plane coordinates using a Gaussian forward transform. Here, the camera's geographical location refers to the location of the camera's center. Step 7: Determine the spatial coordinates of the pixels on the image based on the focal length, the spatial coordinates of the camera center, and the camera's attitude, including pitch, yaw, and roll angles. Then, obtain the coordinates of the actual position in the Gaussian plane. The calculation method for the seven steps is as follows. Step 71: Calculate the rotation matrix R; Given the camera's pitch, yaw, and roll angles, the camera can be described by a rotation matrix R, which can be obtained through the following steps for constructing a rotation matrix: Calculate the rotation matrix Rx about the x-axis (roll), where roll represents the roll angle: Calculate the rotation matrix Ry about the y-axis (pitch), where pitch represents the pitch angle: Calculate the rotation matrix Rz about the z-axis (yaw), where yaw represents the yaw angle: Multiplying the three rotation matrices together yields the final rotation matrix R: Step 72: Calculate the spatial coordinates of the pixels; this is done by multiplying the vector (u,v,f) of the pixels in the image by the rotation matrix R, and then adding the spatial coordinates of the camera center. (This allows us to calculate the spatial coordinates of a pixel.) Step 8: Obtain the corresponding geodetic coordinates from the Gaussian coordinates of the actual location obtained in Step 7 through inverse Gaussian transformation; Step 9: Obtain the latitude and longitude information of all ships within a 10-meter range of latitude and longitude from the AIS system in Step 7. By comparing them one by one, find the ship information that is closest to the latitude and longitude calculated in Step 8, and draw the ship information into the video screen using code. Step 10: Repeat steps 5, 7, 8, and 9 to achieve the function of tracking ship targets in the frame sequence.

2. The method for automatically identifying and tracking ships using a camera according to claim 1, characterized in that: Step 2 obtains the coordinates of the target's distance from the top left corner of the screen on the horizontal axis x1 and vertical axis y1, as well as the target's horizontal span x2 and vertical span y2. Therefore, the horizontal coordinate of the target's center point is x = x1 + x2 / 2, and the vertical coordinate is y = y1 + y2 / 2.

Images