Target identification and tracking method based on integration of shipborne navigation radar and dome camera

Abstract

The invention discloses a target identification and tracking method based on fusion of a shipborne navigation radar and a dome camera. The method comprises the following steps: constructing and training to obtain a ship type identification model and a ship license plate detection model; calculating latitude and longitude coordinates of the target; calculating an angle required by a dome camera to turn to a target acquisition direction, and calculating a focal length required by the target when a camera picture reaches a picture set proportion value; predicting a target latitude and longitude at the current time; the predicted target longitude and latitude coordinates are converted to the position in the current camera PTZ angle picture; fusing the target in the image and the target of the radar ID into the same target; calculating a local proportion size, a ship driving direction and a bow position, and calculating a corresponding camera PTZ value; and carrying out ship type identification and ship license plate detection, and identifying text characters of the ship license plate. According to the invention, modeling is carried out on the navigation trajectory of the target through the radar historical data, the current position of the target is predicted, and the sea target identification and tracking means are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of marine target recognition, and in particular to a target recognition and tracking method based on the fusion of a shipborne navigation radar and a ball camera. Background Art

[0002] At present, AI-related technologies such as image analysis and deep learning have made many important progress in land-based intelligent transportation systems such as license plate recognition and autonomous driving. However, related technologies are still lacking in automatic navigation on the sea, suspicious target identification, tracking and collision avoidance. For example, a public service ship finds a suspicious target on the radar and needs to further identify the target through a camera. However, the relative position of the target ship and the ship on the sea is dynamic and not fixed. Using a fixed panoramic camera can only obtain the target image in a certain direction, and it is impossible to automatically center the image on the target and adjust the focal length to zoom in on the target to obtain a clear large image of the target, and it is even impossible to obtain the target's license plate or hull number; in addition, the biggest difference between a ship's license plate and a car license plate is that the ship's license plate may be at any position on the hull, and the font size, color, shape, etc. are different. At the same time, the arrangement of the ship's license plate also has various situations such as horizontal, vertical and even arc. The use of traditional image processing algorithms can only have some effects on a specific one or two, and cannot be universalized, and the recognition effect and accuracy are low.

[0003] Many existing solutions are focused on solving the problem of ship license plate and hull number recognition at shore-based waterway checkpoints, mainly based on image or video for ship detection and character positioning classification and recognition. However, this method is not applicable to ship-borne ship license plate and hull number recognition. On the sea, the ship's course is dynamic and its relative position to the target is not fixed. In addition, considering that the target distance may be far, the position, font size, and arrangement of the license plates of different types of ships in different weather conditions are uncertain, and very few technologies can solve the problem of ship-borne target ship license plate and hull number recognition. For example, due to the relative distance of ships on the sea, the camera can only capture the rough outline of the ship at a fixed focal length, and the cropped ship image has insufficient resolution, making it difficult to effectively identify the ship license plate. Summary of the invention

[0004] In order to overcome the defects and shortcomings of the prior art, the present invention provides a target recognition and tracking method based on the fusion of shipborne navigation radar and ball camera. The present invention models the navigation trajectory of the target through radar historical data, realizes the prediction of the current position of the target, and solves the problem of failure of fusion of radar target and image recognition target caused by insufficient real-time performance and delay of the target position of the shipborne navigation radar. After spatial conversion using the predicted target position, the radar and image recognition targets are accurately fused to guide the ball camera image to center the target, and further uses image recognition technology to adjust the focal length of the camera, so that the camera can capture the image of the target enlarged to details, identify the target's ship type and license plate, and improve the means of maritime target recognition and tracking.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The present invention provides a target recognition and tracking method based on the fusion of a shipborne navigation radar and a ball camera, comprising the following steps:

[0007] Set up navigation radar, ball camera, GPS and compass on board, and adjust the horizontal angle of the ball camera to 0 degrees in line with the bow direction;

[0008] Get the ship images and ship license plate images collected by the ball camera, add corresponding labels, and build corresponding training sets and verification sets respectively;

[0009] Train the ship type recognition model and the ship license plate detection model;

[0010] The relative position between the target and the ship is obtained based on the navigation radar, and the longitude and latitude coordinates of the target are calculated based on the GPS and compass;

[0011] Get the current PTZ of the ball camera, calculate the angle required for the ball camera to turn to the target direction according to the bow direction and the target longitude and latitude, and calculate the focal length required for the target to reach the set screen ratio in the ball camera screen;

[0012] Predict the target's latitude and longitude at the current time based on the target's radar coordinate data, speed data, heading data and timestamp;

[0013] Convert the predicted target latitude and longitude coordinates to the position in the current PTZ angle image of the ball camera;

[0014] Merge the target in the image and the target in the radar ID into the same target;

[0015] According to the position of the target in the picture and the length and width of the identified target in the picture, calculate the local scale, the ship's direction of travel, the position of the bow, and calculate the corresponding PTZ value of the ball camera;

[0016] Perform ship type recognition and license plate detection to identify the text on the license plate.

[0017] As a preferred technical solution, the relative position between the suspicious target and the ship is obtained based on the navigation radar, and the longitude and latitude coordinates of the suspicious target are calculated based on the GPS and compass, which specifically includes:

[0018] Obtain the latitude and longitude position of the ship based on GPS;

[0019] Obtain the azimuth of the bow of the own ship relative to true north based on the compass;

[0020] Based on the navigation radar, the true azimuth, distance, relative speed, target ID, and target size of all targets are obtained, and the corresponding timestamps are stored;

[0021] The calculation formula of the longitude and latitude coordinates of the suspicious target includes:

[0022]

[0023] Where L2 represents the longitude of the suspicious target, L1 represents the longitude of the own ship, and R represents the radius of the earth;

[0024]

[0025] Among them, B2 represents the latitude of the suspicious target, B1 represents the latitude of the own ship, A represents the true azimuth of the target, and D represents the target distance.

[0026] As a preferred technical solution, calculating the angle required for the camera to turn to collect the target direction specifically includes:

[0027] Calculate the relative angle R from the bow direction to the target, expressed as:

[0028] R=AH

[0029] Among them, H represents the azimuth of the bow direction relative to the true north, and A represents the true azimuth of the target;

[0030] Normalize the relative angle R to the range of 0° to 360°;

[0031] Calculate the camera's steering angle S relative to the target direction a :

[0032] S a =RP

[0033] Where P represents the current angle of the camera. When P = 0, it means that the camera is in the same direction as the bow.

[0034] Set the steering angle S a Normalized to the range of 0° to 360°.

[0035] As a preferred technical solution, the focal length required for the target to reach the set screen ratio in the camera screen is calculated, which is specifically expressed as:

[0036]

[0037] Where Z is the focal length, H object represents the real size of the target, D represents the distance from the target to the camera, and H sensor It indicates the size of the camera CMOS sensor, and ration indicates the proportion of the target in the image.

[0038] As a preferred technical solution, the target latitude and longitude at the current time is predicted based on the target's radar coordinate data, speed data, heading data and timestamp, specifically including:

[0039] A polynomial regression model is constructed based on the historical data of the target latitude and longitude coordinates, which is expressed as:

[0040] y=a0+a1t+a2t 2 +…+a n t n

[0041] Among them, y represents longitude or latitude, t represents time, a0, a1, a2, ...a n represents the coefficient to be calculated, and n represents the number of polynomial regressions;

[0042] Based on the target ID, the historical data of the latitude and longitude coordinates of m targets are obtained, and the corresponding timestamps are represented as t1, t2, …, t m , and its corresponding longitude or latitude is represented by y1,y2,…,y m ;

[0043] The problem is transformed into a system of linear equations, expressed as:

[0044]

[0045] The polynomial regression equation is expressed as:

[0046] y=X·a

[0047] Where a=[a0,a1,…a n ] T is the coefficient vector to be found;

[0048] Solve the coefficients based on the least squares method and minimize the sum of square errors between the predicted target longitude and latitude and the true target longitude and latitude, expressed as:

[0049] min(a)‖yX·a‖ 2

[0050] For each time point t i Extract to the nth power in turn to form the design matrix X, and solve the coefficient vector as follows:

[0051] a=(X T X) -1 X T y

[0052] Substituting the predicted time into the resulting polynomial equation:

[0053]

[0054] Among them, the obtained coefficient vector a is the coefficient of the polynomial regression model [a0, a1, ... a n ].

[0055] As a preferred technical solution, the predicted target longitude and latitude coordinates are converted to the position in the current PTZ angle image of the ball camera, specifically including:

[0056] Convert the predicted target latitude and longitude coordinates into three-dimensional coordinates;

[0057] Transform the fixed coordinate system to the three-dimensional coordinate system under the current dome camera posture;

[0058] Map three-dimensional coordinates to two-dimensional camera array;

[0059] Convert the current ball camera coordinates to pixel coordinates to obtain the coordinates of the target's screen position in the ball camera's preset position.

[0060] As a preferred technical solution, the predicted target latitude and longitude coordinates are converted into three-dimensional coordinates, specifically including:

[0061]

[0062] y2=h1-h0

[0063]

[0064] Among them, the longitude and latitude coordinates of the ball camera are (x0, y0), the longitude and latitude coordinates of the predicted target point are (x1, y1), the three-dimensional coordinates of the target point are (x2, y2, z2), the height of the ball camera is h0, the height of the target point is h1, and the radius of the earth is R;

[0065] Transform the fixed coordinate system to the three-dimensional coordinate system under the current dome camera posture, including rotation around the Y axis and rotation around the X axis. The rotation around the Y axis is expressed as:

[0066]

[0067] Where, ρ represents the horizontal rotation angle;

[0068] Rotation around the X axis is expressed as:

[0069]

[0070] Where, σ represents the pitch rotation angle;

[0071] Map the three-dimensional coordinates to the two-dimensional camera array, expressed as:

[0072]

[0073] Where, f represents the focal length;

[0074] Convert the current ball camera coordinates to pixel coordinates to obtain the coordinates of the target's screen position in the preset position of the ball camera, expressed as:

[0075]

[0076] Among them, d x ,d y Indicates the pixel size on the camera's photosensitive element, where the number of pixels on the element is x cam ×y cam , X represents the horizontal coordinate of the target in the picture, and Y represents the vertical coordinate of the target in the picture.

[0077] As a preferred technical solution, the calculation of the corresponding ball camera PTZ value specifically includes:

[0078] The rotation and pitch of the camera are achieved by adjusting the P and T values, the focal length of the ball camera is adjusted, and the PTZ value required to enlarge the target part and center it is calculated through the picture.

[0079] As a preferred technical solution, identifying the text of the ship license plate specifically includes:

[0080] Get the rectangular area of ​​the ship plate in the picture and use opencv to extract the image of the area, and use the histogram equalization algorithm to enhance the image contrast;

[0081] Based on the extracted area image, the PaddleOCR framework and text recognition model are used to recognize the image text information and obtain the text of the ship license.

[0082] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0083] (1) The present invention uses a navigation radar to locate the target, then guides the ball camera to turn to the target direction, and adjusts the camera focal length according to the radar detection size and distance of the target, thereby achieving phased target magnification and 360-degree all-round target recognition and tracking. Compared with the existing solid-state radar guidance method, it has a larger recognition range and is more convenient to deploy.

[0084] (2) The present invention models the navigation trajectory of the target through historical radar data, and predicts the current position of the target, thereby solving the problem of failure in fusion of radar target and image recognition target caused by insufficient real-time performance and delay of the target position of the shipborne navigation radar.

[0085] (3) After the predicted target position is used for spatial conversion, the present invention accurately integrates radar and image recognition targets, guides the ball camera image to be centered on the target, and further uses image recognition technology to adjust the camera focal length, so that the camera can capture the target image magnified to details and identify the target's ship license plate.

[0086] (4) The present invention realizes tracking and photographing of moving targets through continuous radar guidance and fusion, effectively improving the efficiency and means of evidence collection for maritime targets, and achieving the technical effect of improving maritime target recognition and ship license plate recognition.

[0087] (5) The present invention can guide the camera dome camera to turn to the target direction by using the ship-borne navigation radar, and use the radar historical data to model the target navigation trajectory, predict the current position of the target, integrate radar and image target recognition, realize continuous tracking and shooting of moving targets, and magnify the tracking target through image recognition, so that it can recognize the details of the target ship moving on the sea surface and identify the ship type and license plate, thereby improving the means of identifying and tracking maritime targets. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 It is a flow chart of a target recognition and tracking method based on the fusion of a shipborne navigation radar and a ball camera of the present invention;

[0089] Figure 2 This is a schematic diagram of the camera coordinate rotation of the present invention;

[0090] Figure 3 This is a schematic diagram of the conversion between target coordinates and camera front coordinates of the present invention;

[0091] Figure 4 A schematic diagram of the camera array and image coordinate conversion of the present invention;

[0092] Figure 5 This is a schematic diagram of the target imaging position relationship of the present invention;

[0093] Figure 6This is a schematic diagram of the target imaging ratio relationship of the present invention;

[0094] Figure 7 This is a schematic diagram of the ship license plate detection position of the present invention;

[0095] Figure 8 Schematic diagram of radar target position of the present invention. DETAILED DESCRIPTION

[0096] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0097] Example

[0098] like Figure 1 As shown, this embodiment provides a target recognition and tracking method based on the fusion of a shipborne navigation radar and a ball camera, comprising the following steps:

[0099] S1: Deploy a navigation radar and a dome camera at a high position on the inspection ship, install GPS and compass on the ship, and adjust the horizontal angle of the dome camera to 0 degrees in line with the bow direction;

[0100] S2: Get various real ship pictures and ship license plate pictures taken by the ball camera, and use the ImgLabel tool to manually frame the location of the ship and the ship license plate on the picture, and add type labels, and divide the labeled pictures into two groups: ship type training set and ship type verification set, and ship license plate training set and ship license plate verification set, which are used for model training and accuracy testing respectively;

[0101] S3: Build a YOLOv4 network based on the Darknet framework, input the ship type training set after adding type labels into the YOLOv4 network and train it to obtain a trained ship type recognition model, input the ship license plate training set after adding type labels into the YOLOv4 network and train it to obtain a trained ship license plate detection model. Of course, in order to recognize the hull number, you can also build a hull number training set and a verification set, and train to obtain a hull number detection model;

[0102] S4: Detect suspicious targets based on navigation radar, obtain the relative position of the suspicious targets and the own ship (i.e., the detection ship), and calculate the longitude and latitude coordinates of the suspicious targets through GPS and compass;

[0103] The ship is sailing on the sea and starts the radar to scan nearby targets. When a suspicious target is found, the latitude and longitude coordinates, speed, heading and timestamp of the suspicious target are calculated and obtained through GPS, compass and radar data. Figure 8 As shown, the specific processing and calculation methods include:

[0104] (1) Obtain the latitude and longitude of the ship from GPS;

[0105] (2) Obtain the azimuth of the bow of the ship relative to true north from the compass;

[0106] (3) Obtain the true azimuth (azimuth relative to true north), distance, relative speed, target ID, and target size of all targets from the navigation radar, and save the data and timestamp of each target;

[0107] (4) Calculate the target latitude and longitude based on the ship's latitude and longitude and the target data obtained from the radar. The calculation method is as follows:

[0108] Target latitude B2:

[0109]

[0110] Where B1 is the latitude of the ship, A is the true azimuth of the target, which is the angle starting from the north in the clockwise direction, and D is the target distance in nautical miles or meters.

[0111] Target longitude L2:

[0112]

[0113] Among them, L1 represents the longitude of the ship, and R represents the radius of the earth, which is usually 6371 kilometers;

[0114] Finally, each timestamp and its corresponding acquired target latitude and longitude data are saved in the cache record;

[0115] S5: Get the current PTZ of the camera. According to the bow direction (the angle of the bow relative to the true north) and the longitude and latitude of the suspicious target, calculate the angle required for the camera to turn to see the general direction of the target. According to the target size measured by the radar, calculate the focal length Z value required to adjust the target to about 1 / 6 to 1 / 10 of the screen. Control the camera to turn to the calculated PTZ angle. Specifically include:

[0116] Calculate the relative angle R from the bow direction to the target, expressed as:

[0117] R=AH

[0118] Where H represents the azimuth of the bow relative to the true north (calculated clockwise from the true north);

[0119] Normalize the relative angle R to the range of 0° to 360°. If R < 0, add 360° to make it within the valid range.

[0120] Calculate the camera's steering angle S relative to the target direction a :

[0121] S a =RP

[0122] Where P represents the current angle of the camera. When P = 0, it means that the camera is in the same direction as the bow.

[0123] Set the steering angle S a Normalized to the range of 0° to 360°;

[0124] The calculated steering angle S a , we can know the angle that the camera needs to rotate clockwise relative to the current angle to align with the target direction. The focal length Z is calculated as follows:

[0125]

[0126] Among them, H object Indicates the real size (height or width) of the target, D indicates the distance from the target to the camera, H sensor represents the size of the camera CMOS sensor (in the direction consistent with the target direction), ration represents the proportion of the target in the image, and the calculated focal length Z is the required focal length to make the target occupy 1 / 10 of the size in the image;

[0127] S6: According to the radar coordinate data, speed data, heading data and timestamp of the suspicious target in step S4, the target latitude and longitude coordinate position at the current time is predicted, and the prediction calculation method includes:

[0128] S61: Use the target ID to obtain the target history data saved by the radar to establish a polynomial regression model. For an n-order polynomial regression, the model can be expressed as:

[0129] y=a0+a1t+a2t 2 +…+a n t n

[0130] Among them, y represents longitude or latitude, t represents time, a0, a1, a2, ...a n represents the coefficient to be determined;

[0131] S62: Use the target ID to obtain the historical data of the target in the radar for a period of time. Assume that there are m data points, and the time data are t1, t2, …, t m and the corresponding location data (such as longitude or latitude) y1,y2,…,y m .

[0132] S63: Convert the problem into a system of linear equations. Polynomial regression is actually a linear regression problem, except that the independent variables include different powers of time. The regression equation can be written in matrix form so that the coefficients can be solved using the least squares method.

[0133] make:

[0134]

[0135] Then the polynomial regression equation can be expressed as:

[0136] y=X·a

[0137] Where a=[a0,a1,…a n ] T is the coefficient vector to be found;

[0138] S64: Least square method to solve coefficients;

[0139] The goal of the least squares method is to minimize the sum of squared errors between the predicted value and the true value, that is:

[0140] min(a)‖yX·a‖ 2

[0141] Calculate the matrix X: transform each time point t i Extract to the nth power in sequence to form the design matrix X;

[0142] Solve for the coefficient vector a:

[0143] a=(X T X) -1 X T y

[0144] The obtained coefficient vector a is the coefficient of the polynomial regression model [a0, a1, …a n ];

[0145] S65: Use the model to make predictions;

[0146] Substituting the predicted time into the resulting polynomial equation:

[0147]

[0148] Here, y is used instead of longitude and latitude for convenience. The longitude and latitude are calculated separately using the above method.

[0149] Finally, the predicted target longitude and latitude position is obtained, and then the longitude and latitude coordinates are converted to the position in the current camera PTZ angle screen. The specific calculation method is:

[0150] 1. Conversion of latitude and longitude coordinates to three-dimensional coordinates;

[0151] When transforming from longitude and latitude coordinates to three-dimensional coordinates, the distance when the latitude changes by 1° is fixed, and the distance when the longitude changes by 1° is affected by the current latitude of the camera and the target point;

[0152] The three-dimensional coordinate system uses the camera coordinates as the origin, the east and north directions as the X-axis and Z-axis, and the height as the Y-axis;

[0153] Assume the latitude and longitude coordinates of the camera are (x0, y0), the predicted longitude and latitude coordinates of the target point are (x1, y1), the three-dimensional coordinates of the target point are (x2, y2, z2), the camera height is h0, the target point height is h1, and the radius of the earth is R;

[0154]

[0155] y2=h1-h0

[0156]

[0157] Among them, z0 is the latitude of the camera, and z1 is the latitude of the target;

[0158] 2. If Figure 2 As shown, since the ball camera needs to transform the fixed coordinate system into the three-dimensional coordinate system under the current camera posture during the cruising process, the operation includes horizontal rotation (around the Y axis) and pitch (around the X axis);

[0159] Rotation around the Y axis (horizontal rotation angle ρ):

[0160]

[0161] Rotation around the X axis (pitch rotation angle σ):

[0162]

[0163] 3. Map the three-dimensional coordinates to the two-dimensional camera array;

[0164] like Figure 3 As shown, the mapping of three-dimensional coordinates to the camera array can be obtained by triangle similarity, where the distance from the origin of the coordinate to the photosensitive element (focal length f);

[0165]

[0166] 4. Camera coordinates to pixel coordinates;

[0167] like Figure 4 As shown, the pixel size on the camera sensor is d x ,d y , the number of pixels on the element is x cam ×ycam (e.g. 1920*1080);

[0168]

[0169] Among them, X and Y are the coordinate positions of the target in the camera.

[0170] S7: Determine whether the position of the target radar in the camera is within the recognition frame of the ship target recognition. If the target is within the recognition frame, the target in the image is the corresponding target on the radar, that is, the target in the image and the target of the radar ID are merged into the same target, and the target ID of the radar and the target detected by the image are bound;

[0171] S8: Calculate the local scale, the ship's travel direction, the bow position and the corresponding camera PTZ value according to the position of the target in the picture and the length and width of the identified target in the picture;

[0172] Specifically, the coordinate position of the target in the picture and the length and height data of the detection frame are obtained, and the local position coordinates of the target are calculated with 1 / 3 of the length and height of the target detection frame as the rectangle. The PTZ values ​​required to control the center of the camera picture and zoom in to 1 / 6 to 1 / 2 of the picture are calculated respectively. The specific calculation method includes:

[0173] like Figure 5 As shown, the rotation and pitch of the camera are achieved by adjusting the P and T values. The purpose is to place the target in the center of the picture. If the ship needs to be displayed in the center of the picture, the camera needs to be rotated counterclockwise, which can be understood as rotating the dotted line by θ:

[0174]

[0175] Where f is the focal length, Δ is the distance from the center point of the target image on the CMOS plane, and Δ can be converted into the number of pixels from the target to the center of the image multiplied by the size of the imaging unit corresponding to one pixel;

[0176] like Figure 6 As shown in FIG. 1 , the purpose of adjusting the f value is to make the target occupy most of the screen. From CMOS1 to CMOS1′, the target occupies most of the screen by adjusting the focal length. Although CMOS2 and CMOS1 are of different sizes, that is, they are two different cameras, the focal length f can also be calculated through similar triangles.

[0177] Get the position of the screen where the camera finds the target, such as Figure 7 As shown, the PTZ value required to enlarge and center the target part is calculated through the picture, and the pan / tilt of the zoom camera is controlled to the PTZ value so that the local position on the target ship can be captured.

[0178] S9: Save the pictures of the bow, hull and stern respectively, and use the ship type recognition model and ship license plate detection model trained in step S3 to perform ship type recognition and ship license plate detection on the saved pictures;

[0179] S10: If a picture with a ship plate is detected, the rectangular area of ​​the ship plate in the picture is obtained and the image of the area is extracted using opencv, and the contrast of the picture is enhanced using a histogram equalization algorithm;

[0180] S11: Get the image extracted in step S10, use the PaddleOCR framework and the general text recognition model to recognize the text information in the image and obtain the text.

[0181] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A target recognition and tracking method based on the fusion of shipborne navigation radar and ball camera, characterized in that: The steps include: Set up navigation radar, ball camera, GPS and compass on board, and adjust the horizontal angle of the ball camera to 0 degrees in line with the bow direction; Get the ship images and ship license plate images collected by the ball camera, add corresponding labels, and build corresponding training sets and verification sets respectively; Train the ship type recognition model and the ship license plate detection model; The relative position between the target and the ship is obtained based on the navigation radar, and the longitude and latitude coordinates of the target are calculated based on the GPS and compass; Get the current PTZ of the ball camera, calculate the angle required for the ball camera to turn to the target direction according to the bow direction and the target longitude and latitude, and calculate the focal length required for the target to reach the set screen ratio in the ball camera screen; Predict the target's latitude and longitude at the current time based on the target's radar coordinate data, speed data, heading data and timestamp; Convert the predicted target latitude and longitude coordinates to the position in the current PTZ angle image of the ball camera; Merge the target in the image and the target in the radar ID into the same target; According to the position of the target in the picture and the length and width of the identified target in the picture, the local scale, the ship's direction of travel, and the position of the bow are calculated, and the corresponding PTZ value of the ball camera is calculated; Perform ship type recognition and license plate detection to identify the text on the license plate.

2. The target recognition and tracking method based on the fusion of shipborne navigation radar and ball camera according to claim 1 is characterized in that: The relative position between the suspicious target and the ship is obtained based on the navigation radar, and the longitude and latitude coordinates of the suspicious target are calculated based on the GPS and compass, including: Obtain the latitude and longitude position of the ship based on GPS; Obtain the azimuth of the bow of the own ship relative to true north based on the compass; Based on the navigation radar, the true azimuth, distance, relative speed, target ID, and target size of all targets are obtained, and the corresponding timestamps are stored; The calculation formula of the longitude and latitude coordinates of the suspicious target includes: Where L2 represents the longitude of the suspicious target, L1 represents the longitude of the own ship, and R represents the radius of the earth; Among them, B2 represents the latitude of the suspicious target, B1 represents the latitude of the own ship, A represents the true azimuth of the target, and D represents the target distance.

3. The target recognition and tracking method based on the fusion of shipborne navigation radar and ball camera according to claim 1 is characterized in that: Calculate the angle required for the camera to turn to the target direction, including: Calculate the relative angle R from the bow direction to the target, expressed as: R=AH Among them, H represents the azimuth of the bow direction relative to the true north, and A represents the true azimuth of the target; Normalize the relative angle R to the range of 0° to 360°; Calculate the camera's steering angle S relative to the target direction a : S a =R-P Where P represents the current angle of the camera. When P = 0, it means that the camera is in the same direction as the bow. Set the steering angle S a Normalized to the range of 0° to 360°.

4. The target recognition and tracking method based on the fusion of shipborne navigation radar and ball camera according to claim 1 is characterized in that: Calculate the focal length required for the target to reach the set screen ratio in the camera screen, which is specifically expressed as: Where Z is the focal length, H object represents the real size of the target, D represents the distance from the target to the camera, and H sensor It represents the size of the camera CMOS sensor, and ratoin represents the proportion of the target in the image.

5. The target recognition and tracking method based on the fusion of shipborne navigation radar and ball camera according to claim 1 is characterized in that: According to the target's radar coordinate data, speed data, heading data and timestamp, the target's latitude and longitude at the current time are predicted, including: A polynomial regression model is constructed based on the historical data of the target latitude and longitude coordinates, which is expressed as: y=a0+a1t+a2t 2 +…+a n t n Among them, y represents longitude or latitude, t represents time, a0, a1, a2, ...a n represents the coefficient to be calculated, and n represents the number of polynomial regressions; Based on the target ID, the historical data of the latitude and longitude coordinates of m targets are obtained, and the corresponding timestamps are represented as t1, t2, …, t m , and its corresponding longitude or latitude is represented by y1,y2,…,y m ; The problem is transformed into a system of linear equations, expressed as: The polynomial regression equation is expressed as: y=X·a Where a=[a0,a1,…a n ] T is the coefficient vector to be found; Solve the coefficients based on the least squares method and minimize the sum of square errors between the predicted target longitude and latitude and the true target longitude and latitude, expressed as: min(a)‖yX·a‖ 2 For each time point t i Extract to the nth power in turn to form the design matrix X, and solve the coefficient vector as follows: a=(X T X) -1 X T y Substituting the predicted time into the resulting polynomial equation: Among them, the obtained coefficient vector a is the coefficient of the polynomial regression model [a0, a1, ... a n ].

6. The target recognition and tracking method based on the fusion of shipborne navigation radar and ball camera according to claim 1 is characterized in that: Convert the predicted target latitude and longitude coordinates to the position in the current PTZ angle image of the ball camera, including: Convert the predicted target latitude and longitude coordinates into three-dimensional coordinates; Transform the fixed coordinate system to the three-dimensional coordinate system under the current dome camera posture; Map the three-dimensional coordinates to the two-dimensional camera array; Convert the current ball camera coordinates to pixel coordinates to obtain the coordinates of the target's screen position in the ball camera's preset position.

7. The target recognition and tracking method based on the fusion of shipborne navigation radar and ball camera according to claim 6 is characterized in that: Convert the predicted target latitude and longitude coordinates into three-dimensional coordinates, including: y2=h1-h0 Among them, the longitude and latitude coordinates of the ball camera are (x0, y0), the longitude and latitude coordinates of the predicted target point are (x1, y1), the three-dimensional coordinates of the target point are (x2, y2, z2), the height of the ball camera is h0, the height of the target point is h1, and the radius of the earth is R; Transform the fixed coordinate system to the three-dimensional coordinate system under the current dome camera posture, including rotation around the Y axis and rotation around the X axis. The rotation around the Y axis is expressed as: Where, ρ represents the horizontal rotation angle; Rotation around the X axis is expressed as: Where, σ represents the pitch rotation angle; Map the three-dimensional coordinates to the two-dimensional camera array, expressed as: Where, f represents the focal length; Convert the current ball camera coordinates to pixel coordinates to obtain the coordinates of the target's screen position in the preset position of the ball camera, expressed as: Among them, d x ,d y Indicates the pixel size on the camera's photosensitive element, where the number of pixels on the element is x cam ×y cam , X represents the horizontal coordinate of the target in the picture, and Y represents the vertical coordinate of the target in the picture.

8. The target recognition and tracking method based on the fusion of shipborne navigation radar and ball camera according to claim 1 is characterized in that: Calculate the corresponding PTZ value of the ball camera, including: The rotation and pitch of the camera are achieved by adjusting the P and T values, the focal length of the ball camera is adjusted, and the PTZ value required to enlarge the target part and center it is calculated through the picture.

9. The target recognition and tracking method based on the fusion of shipborne navigation radar and ball camera according to claim 1 is characterized in that: Identify the text of the ship's license plate, including: Get the rectangular area of ​​the ship plate in the picture and use opencv to extract the image of the area, and use the histogram equalization algorithm to enhance the image contrast; Based on the extracted area image, the PaddleOCR framework and text recognition model are used to recognize the image text information and obtain the text of the ship license.

Images