Target recognition and tracking method based on the fusion of shipborne navigation radar and ball camera

Abstract

The present invention discloses a target recognition and tracking method based on the fusion of a ship-borne navigation radar and a dome camera, comprising the following steps: constructing and training a ship type recognition model and a ship license plate detection model; calculating the latitude and longitude coordinates of the target; calculating the angle required for the dome camera to turn and capture the target direction, and calculating the focal length required for the target to reach a set image percentage in the camera image; predicting the target longitude and latitude at the current time; converting the predicted target longitude and latitude coordinates to the position in the current camera PTZ angle image; fusing the target in the image with the target identified by the radar ID into the same target; calculating the local scale size, the ship's direction of travel, and the bow position, and calculating the corresponding camera PTZ value; performing ship type recognition and ship license plate detection, and identifying the text on the ship license plate. The present invention models the target's navigation trajectory using historical radar data, predicts the target's current position, and improves maritime target recognition and tracking methods.

Description

Technical Field

[0001] The present invention relates to the technical field of marine target recognition, and in particular to a target recognition and tracking method based on the fusion of a shipborne navigation radar and a dome camera. Background Art

[0002] Currently, AI-related technologies such as image analysis and deep learning have made significant progress in land-based intelligent transportation systems, such as license plate recognition and autonomous driving. However, related technologies are still lacking for autonomous navigation at sea, suspicious target identification, tracking, and collision avoidance. For example, if a government vessel detects a suspicious target on radar, it needs to use a camera to further identify the target. However, the target ship's position relative to the host vessel on the sea is dynamic and unstable. Using a fixed panoramic camera can only capture an image of the target in a certain direction. It cannot automatically center the image on the target and adjust the focus to zoom in on the target for a clear and large image, nor can it capture the target's license plate or hull number. In addition, the biggest difference between ship license plates and vehicle license plates is that the license plates can be located anywhere on the hull, and the font size, color, and shape vary. Furthermore, the license plates can be arranged in various shapes, such as horizontal, vertical, and even curved. Traditional image processing algorithms can only be effective for specific ones or two types, and are not universally applicable, resulting in low recognition efficiency and accuracy.

[0003] Many existing solutions focus on solving the problem of ship license plate and hull number recognition at shore-based waterway checkpoints, primarily using images or videos for ship detection and character location, classification, and recognition. However, these methods are not suitable for shipborne license plate and hull number recognition. On the sea, the ship's course and position relative to the target are not fixed. Furthermore, given the potential distance to the target and uncertainties such as the position, font size, and arrangement of license plates for different types of ships in different weather conditions, few technologies can address the challenge of shipborne target ship license plate and hull number recognition. For example, due to the relative distance of ships on the sea, a camera with a fixed focal length can only capture the rough outline of the ship. The resulting cropped image of the ship has insufficient resolution, making it difficult to effectively identify the ship's license plate. Summary of the Invention

[0004] In order to overcome the defects and shortcomings of the existing technology, the present invention provides a target recognition and tracking method based on the fusion of shipborne navigation radar and dome camera. The present invention uses radar historical data to model the navigation trajectory of the target and predict the current position of the target. It solves the problem of failure of fusion of radar target and image recognition target caused by insufficient real-time performance and delay of the target position of the shipborne navigation radar. After using the predicted target position for spatial conversion, the radar and image recognition targets are accurately fused to guide the dome camera image to center the target, and further use image recognition technology to adjust the camera focal length, so that the camera can capture the target enlarged to the details of the picture, identify the target's ship type and license plate, and improve the means of maritime target recognition and tracking.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The present invention provides a target recognition and tracking method based on the fusion of a shipborne navigation radar and a dome camera, comprising the following steps:

[0007] Set up the navigation radar, dome camera, GPS and compass on board, and adjust the dome camera's horizontal angle of 0 degrees to align with the bow direction;

[0008] Obtain ship images and ship license plate images captured by the dome camera, add corresponding labels, and construct corresponding training and validation sets respectively;

[0009] Train the ship type recognition model and the ship license plate detection model;

[0010] Obtain the relative position of the target and the ship based on the navigation radar, and calculate the latitude and longitude coordinates of the target based on the GPS and compass;

[0011] Get the current PTZ of the dome camera. Based on the bow direction and the target's latitude and longitude, calculate the angle required for the dome camera to turn to capture the target direction. Calculate the focal length required for the target to reach the set screen ratio in the dome camera's image.

[0012] Predict the target's latitude and longitude at the current time based on the target's radar coordinate data, speed data, heading data, and timestamp;

[0013] Convert the predicted target latitude and longitude coordinates to the position in the current PTZ angle image of the dome camera;

[0014] Merge the target in the image and the target in the radar ID into the same target;

[0015] Calculate the local scale, ship's direction, and bow position based on the target's position in the image and the length and width of the identified target in the image, and calculate the corresponding PTZ value of the dome camera;

[0016] Perform ship type identification and license plate detection to recognize the text on the license plate.

[0017] As a preferred technical solution, the relative position of the suspicious target and the ship is obtained based on the navigation radar, and the longitude and latitude coordinates of the suspicious target are calculated based on the GPS and compass, which specifically includes:

[0018] Obtain the longitude and latitude of the ship based on GPS;

[0019] Obtain the azimuth of the own ship's bow relative to true north based on the compass;

[0020] Based on the navigation radar, the true azimuth, distance, relative speed, target ID, and target size of all targets are obtained and the corresponding timestamps are stored;

[0021] The calculation formula for the longitude and latitude coordinates of a suspicious target includes:

[0022]

[0023] Where L2 represents the longitude of the suspicious target, L1 represents the longitude of own ship, and R represents the radius of the earth.

[0024]

[0025] Among them, B2 is the latitude of the suspicious target, B1 is the latitude of the own ship, A is the true azimuth of the target, and D is the target distance.

[0026] As a preferred technical solution, calculating the angle required for the camera to turn to capture the target direction specifically includes:

[0027] Calculate the relative angle R from the bow direction to the target, expressed as:

[0028] R=AH

[0029] Where H represents the azimuth of the bow relative to true north, and A represents the true azimuth of the target;

[0030] Normalize the relative angle R to the range of 0° to 360°;

[0031] Calculate the camera's steering angle S relative to the target direction a :

[0032] S a =RP

[0033] Where P represents the current angle of the camera. When P = 0, it means that the camera is aligned with the bow direction.

[0034] Set the steering angle S a Normalized to the range of 0° to 360°.

[0035] As a preferred technical solution, the focal length required for the target to reach the set screen ratio in the camera image is calculated, which is specifically expressed as:

[0036]

[0037] Among them, Z represents the focal length, H object Indicates the actual size of the target, D indicates the distance from the target to the camera, H sensor Indicates the size of the camera CMOS sensor, and ration indicates the proportion of the target in the image.

[0038] As a preferred technical solution, the target latitude and longitude at the current time is predicted based on the target's radar coordinate data, speed data, heading data and timestamp, specifically including:

[0039] A polynomial regression model is constructed based on the historical data of the target latitude and longitude coordinates, which can be expressed as:

[0040] y=a0+a1t+a2t 2 +…+a n t n

[0041] Among them, y represents longitude or latitude, t represents time, a0, a1, a2, ...a n represents the coefficient to be determined, and n represents the number of polynomial regressions;

[0042] Based on the target ID, the historical data of the latitude and longitude coordinates of m targets are obtained, and the corresponding timestamps are represented as t1, t2, ..., t m , and its corresponding longitude or latitude is expressed as y1,y2,…,y m ;

[0043] The problem is transformed into a system of linear equations, expressed as:

[0044]

[0045] The polynomial regression equation is expressed as:

[0046] y=X·a

[0047] Where a=[a0,a1,…a n ] T is the coefficient vector to be found;

[0048] Solve the coefficients based on the least squares method and minimize the sum of squared errors between the predicted target latitude and longitude and the true target latitude and longitude, expressed as:

[0049] min(a)‖yX·a‖ 2

[0050] At each time point t i Extract to the nth power in sequence to form the design matrix X, and solve the coefficient vector as follows:

[0051] a=(X T X) -1 X T y

[0052] Substituting the predicted time into the resulting polynomial equation:

[0053]

[0054] Among them, the coefficient vector a is the coefficient of the polynomial regression model [a0, a1, ... a n ].

[0055] As a preferred technical solution, the predicted target longitude and latitude coordinates are converted to the position in the current PTZ angle image of the ball camera, specifically including:

[0056] Convert the predicted target latitude and longitude coordinates into three-dimensional coordinates;

[0057] Transform the fixed coordinate system into the three-dimensional coordinate system under the current dome camera posture;

[0058] Mapping three-dimensional coordinates to two-dimensional camera array;

[0059] Convert the current dome camera coordinates to pixel coordinates to obtain the coordinates of the target's position on the screen in the dome camera's preset position.

[0060] As a preferred technical solution, the predicted target latitude and longitude coordinates are converted into three-dimensional coordinates, specifically including:

[0061]

[0062] y2=h1-h0

[0063]

[0064] The latitude and longitude coordinates of the dome camera are (x0, y0), the predicted latitude and longitude coordinates of the target point are (x1, y1), the three-dimensional coordinates of the target point are (x2, y2, z2), the height of the dome camera is h0, the height of the target point is h1, and the radius of the earth is R.

[0065] The fixed coordinate system is transformed into the three-dimensional coordinate system under the current dome camera posture, including rotation around the Y axis and rotation around the X axis. The rotation around the Y axis is expressed as:

[0066]

[0067] Where ρ represents the horizontal rotation angle;

[0068] Rotation around the X axis is expressed as:

[0069]

[0070] Where, σ represents the pitch rotation angle;

[0071] Map the three-dimensional coordinates to the two-dimensional camera array, expressed as:

[0072]

[0073] Where f represents the focal length;

[0074] Convert the current camera coordinates to pixel coordinates to get the coordinates of the target position in the preset position of the camera, which is expressed as:

[0075]

[0076] Among them, d x d y Indicates the pixel size on the camera's photosensitive element, where the number of pixels on the element is x cam ×y cam , X represents the horizontal coordinate of the target in the picture, and Y represents the vertical coordinate of the target in the picture.

[0077] As a preferred technical solution, the calculation of the corresponding ball camera PTZ value specifically includes:

[0078] By adjusting the P and T values, the camera can be rotated and pitched, the focal length of the dome camera can be adjusted, and the PTZ value required to zoom in on the target part and center it can be calculated through the image.

[0079] As a preferred technical solution, identifying the text of the ship license plate specifically includes:

[0080] Get the rectangular area of ​​the ship plate in the picture and use OpenCV to extract the image of the area, and use the histogram equalization algorithm to enhance the image contrast;

[0081] Based on the extracted area image, the PaddleOCR framework and text recognition model are used to recognize the image text information and obtain the text of the ship license plate.

[0082] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0083] (1) The present invention uses a navigation radar to locate the target, guides the dome camera to turn to the target direction, and adjusts the camera focal length according to the radar detection size and distance of the target, thereby achieving phased target magnification and 360-degree all-round target recognition and tracking. Compared with the existing solid-state radar guidance method, it has a larger recognition range and is more convenient to deploy.

[0084] (2) The present invention models the navigation trajectory of the target through historical radar data and predicts the current position of the target, thereby solving the problem of failure in the fusion of radar target and image recognition target caused by insufficient real-time performance and delay of the target position of the shipborne navigation radar.

[0085] (3) After the predicted target position is used for spatial conversion, the present invention accurately integrates radar and image recognition targets, guides the dome camera image to center the target, and further uses image recognition technology to adjust the camera focal length, so that the camera can capture the target image magnified to details and identify the target's ship license plate.

[0086] (4) The present invention realizes tracking and shooting of moving targets through continuous radar guidance and fusion, effectively improving the efficiency and means of evidence collection of maritime targets, and achieving the technical effect of improving maritime target recognition and ship license plate recognition.

[0087] (5) The present invention can guide the camera dome camera to turn to the target direction by using the ship-borne navigation radar, and use the radar historical data to model the target navigation trajectory, predict the current position of the target, integrate radar and image target recognition, realize continuous tracking and shooting of moving targets, and amplify the tracking target through image recognition, so that it can identify the details of the target ship moving on the sea surface and identify the ship type and license plate, thereby improving the means of identifying and tracking targets at sea. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Figure 1 Schematic diagram of the flow of the target recognition and tracking method based on the fusion of shipborne navigation radar and dome camera of the present invention;

[0089] Figure 2 This is a schematic diagram of the camera coordinate rotation of the present invention;

[0090] Figure 3 Schematic diagram of the conversion between target coordinates and camera array coordinates of the present invention;

[0091] Figure 4 Schematic diagram of camera array and image coordinate conversion in the present invention;

[0092] Figure 5 Schematic diagram of the target imaging position relationship of the present invention;

[0093] Figure 6This is a schematic diagram of the target imaging ratio relationship of the present invention;

[0094] Figure 7 This is a schematic diagram of the ship license plate detection position of the present invention;

[0095] Figure 8 Schematic diagram of radar target position of the present invention. DETAILED DESCRIPTION

[0096] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0097] Example

[0098] like Figure 1 As shown, this embodiment provides a target recognition and tracking method based on the fusion of shipborne navigation radar and dome camera, including the following steps:

[0099] S1: Deploy a navigation radar and a dome camera high above the inspection vessel, install a GPS and a compass on board, and adjust the dome camera's horizontal angle of 0 degrees to align with the bow direction;

[0100] S2: Obtain various real-life ship images and ship license plate images captured by a dome camera, and use the ImgLabel tool to manually frame the locations of the ships and ship license plates on the images, and add type labels. The labeled images are then divided into two groups: a ship type training set and a ship type verification set, and a ship license plate training set and a ship license plate verification set, which are used for model training and accuracy testing, respectively.

[0101] S3: Build a YOLOv4 network based on the Darknet framework, input the ship type training set with type labels into the YOLOv4 network and train it to obtain a trained ship type recognition model. Input the ship license plate training set with type labels into the YOLOv4 network and train it to obtain a trained ship license plate detection model. Of course, in order to recognize the hull number, you can also build a hull number training set and a validation set, and train to obtain a hull number detection model.

[0102] S4: Detect suspicious targets based on navigation radar, obtain the relative position of the suspicious targets and the own ship (i.e., the detection ship), and calculate the longitude and latitude coordinates of the suspicious targets using GPS and compass;

[0103] The ship is sailing on the sea and starts the radar to scan nearby targets. When a suspicious target is found, the ship calculates and obtains the longitude and latitude coordinates, speed, heading and timestamp of the suspicious target through GPS, compass and radar data, such as Figure 8 As shown, the specific processing and calculation methods include:

[0104] (1) Obtain the longitude and latitude of the ship from GPS;

[0105] (2) Obtain the azimuth of the bow of the own ship relative to true north from the compass;

[0106] (3) Obtain the true azimuth (azimuth relative to true north), distance, relative speed, target ID, and target size of all targets from the navigation radar, and save the data and timestamp of each target;

[0107] (4) Calculate the target latitude and longitude based on the ship's latitude and longitude and the target data obtained from the radar. The calculation method is as follows:

[0108] Target latitude B2:

[0109]

[0110] Where B1 is the latitude of the own ship, A is the true azimuth of the target, which is the angle measured clockwise from due north, and D is the distance to the target in nautical miles or meters.

[0111] Target longitude L2:

[0112]

[0113] Where L1 represents the longitude of the ship, and R represents the radius of the earth, which is usually 6371 kilometers.

[0114] Finally, each timestamp and its corresponding acquired target latitude and longitude data are saved to the cache record;

[0115] S5: Obtain the current PTZ of the camera. Based on the bow direction (the angle of the ship's bow relative to true north) and the longitude and latitude of the suspicious target, calculate the angle required for the camera to turn to see the general direction of the target. Based on the target size measured by the radar, calculate the focal length Z value required to adjust the target's screen area to approximately 1 / 6 to 1 / 10. Control the camera to the calculated PTZ angle. Specifically, the following steps are performed:

[0116] Calculate the relative angle R from the bow direction to the target, expressed as:

[0117] R=AH

[0118] Where H is the azimuth of the bow relative to true north (calculated clockwise from true north);

[0119] Normalize the relative angle R to the range of 0° to 360°. If R < 0, add 360° to make it within the valid range.

[0120] Calculate the camera's steering angle S relative to the target direction a :

[0121] S a =RP

[0122] Where P represents the current angle of the camera. When P = 0, it means that the camera is aligned with the bow direction.

[0123] Set the steering angle S a Normalized to the range of 0° to 360°;

[0124] By calculating the steering angle S a , we can know the angle that the camera needs to rotate clockwise relative to the current angle to align with the target direction. The focal length Z is calculated as follows:

[0125]

[0126] Among them, H object Indicates the actual size (height or width) of the target, D indicates the distance from the target to the camera, H sensor represents the size of the camera's CMOS sensor (in the direction consistent with the target direction), ration represents the proportion of the target in the image, and the calculated focal length Z is the focal length required to make the target occupy 1 / 10 of its size in the image;

[0127] S6: Based on the radar coordinate data, speed data, heading data and timestamp of the suspicious target in step S4, the target latitude and longitude coordinate position at the current time is predicted. The prediction calculation method includes:

[0128] S61: Use the target ID to obtain the target history data saved by the radar to establish a polynomial regression model. For an n-order polynomial regression, the model can be expressed as:

[0129] y=a0+a1t+a2t 2 +…+a n t n

[0130] Among them, y represents longitude or latitude, t represents time, a0, a1, a2, ...a n represents the coefficient to be determined;

[0131] S62: Use the target ID to obtain the historical data of the target in the radar for a period of time. Assume there are m data points, time data t1, t2, ..., t m and the corresponding location data (such as longitude or latitude) y1,y2,…,y m .

[0132] S63: Convert the problem into a system of linear equations. Polynomial regression is actually a linear regression problem, except that the independent variables include different powers of time. The regression equation can be written in matrix form so that the coefficients can be solved using the least squares method.

[0133] make:

[0134]

[0135] Then the polynomial regression equation can be expressed as:

[0136] y=X·a

[0137] Where a=[a0,a1,…a n ] T is the coefficient vector to be found;

[0138] S64: Least squares method to solve coefficients;

[0139] The goal of the least squares method is to minimize the sum of squared errors between the predicted value and the true value, that is:

[0140] min(a)‖yX·a‖ 2

[0141] Calculate the matrix X: transform each time point t i Extract to the nth power in sequence to form the design matrix X;

[0142] Solve for the coefficient vector a:

[0143] a=(X T X) -1 X T y

[0144] The obtained coefficient vector a is the coefficient of the polynomial regression model [a0, a1, ... a n ];

[0145] S65: Using the model to make predictions;

[0146] Substituting the predicted time into the resulting polynomial equation:

[0147]

[0148] Here, y is used instead of longitude and latitude for convenience. Longitude and latitude are calculated separately using the above method.

[0149] Finally, the predicted target longitude and latitude positions are obtained, and then the longitude and latitude coordinates are converted to the position in the current camera PTZ angle image. The specific calculation method is:

[0150] 1. Conversion of latitude and longitude coordinates to three-dimensional coordinates;

[0151] When transforming from latitude and longitude coordinates to three-dimensional coordinates, the distance when the latitude changes by 1° is fixed, while the distance when the longitude changes by 1° is affected by the current latitude of the camera and the target point;

[0152] The three-dimensional coordinate system takes the camera coordinates as the coordinate origin, the east and north directions as the X axis and Z axis, and the height as the Y axis;

[0153] Assume the latitude and longitude coordinates of the camera are (x0, y0), the predicted latitude and longitude coordinates of the target point are (x1, y1), the three-dimensional coordinates of the target point are (x2, y2, z2), the camera height is h0, the target height is h1, and the radius of the earth is R;

[0154]

[0155] y2=h1-h0

[0156]

[0157] Among them, z0 is the latitude of the camera, and z1 is the latitude of the target;

[0158] 2. If Figure 2 As shown in the figure, since the dome camera needs to transform the fixed coordinate system into the three-dimensional coordinate system under the current camera posture during the cruising process, the operation includes horizontal rotation (around the Y axis) and pitch (around the X axis);

[0159] Rotation around the Y axis (horizontal rotation angle ρ):

[0160]

[0161] Rotation around the X axis (pitch rotation angle σ):

[0162]

[0163] 3. Mapping the three-dimensional coordinates to the two-dimensional camera array;

[0164] like Figure 3 As shown, the mapping of three-dimensional coordinates to the camera array can be obtained by triangle similarity, where the distance from the coordinate origin to the photosensitive element (focal length f);

[0165]

[0166] 4. Camera coordinates to pixel coordinates;

[0167] like Figure 4 As shown, the pixel size on the camera sensor is d x d y , the number of pixels on the element is x cam ×ycam (e.g. 1920*1080);

[0168]

[0169] Among them, X and Y are the coordinate positions of the target in the camera.

[0170] S7: Determine whether the target's radar position in the camera is within the recognition frame of the ship target recognition. If the target is within the recognition frame, the target in the image is the corresponding target on the radar. That is, the target in the image and the target in the radar ID are fused into the same target, and the target ID of the radar and the target detected in the image are bound.

[0171] S8: Calculate the local scale, the ship's direction, the bow position, and the corresponding camera PTZ value based on the target's position in the image and the length and width of the identified target in the image;

[0172] Specifically, obtain the coordinate position of the target in the image and the length and height data of the detection frame, calculate the local position coordinates of the target with 1 / 3 of the length and height of the target detection frame as the rectangle, and calculate the PTZ values ​​required to control the camera image to be centered and zoomed to 1 / 6 to 1 / 2 of the image. The specific calculation method includes:

[0173] like Figure 5 As shown, the rotation and pitch of the camera are achieved by adjusting the P and T values. The purpose is to place the target in the center of the picture. If the ship needs to be displayed in the center of the picture, the camera needs to be rotated counterclockwise, which can be understood as rotating the dotted line by θ:

[0174]

[0175] Where f is the focal length, Δ is the distance from the center point of the target image on the CMOS plane, and Δ can be converted into the number of pixels from the target to the center of the image multiplied by the size of the imaging unit corresponding to one pixel;

[0176] like Figure 6 As shown in the figure, the purpose of adjusting the f-number is to make the target occupy the majority of the frame. From CMOS1 to CMOS1′, the focus is adjusted to make the target occupy the majority of the frame. Although CMOS2 and CMOS1 are different sizes, that is, they are two different cameras, the focal length f can also be calculated using similar triangles.

[0177] Get the position of the camera detecting the target, such as Figure 7 As shown, the PTZ value required to zoom in and center the target part is calculated through the picture, and the pan / tilt of the zoom camera is controlled to the PTZ value so that the local position on the target ship can be captured.

[0178] S9: Save images of the bow, hull, and stern respectively, and use the ship type recognition model and ship license plate detection model trained in step S3 to perform ship type recognition and ship license plate detection on the saved images;

[0179] S10: If a picture with a ship license plate is detected, the rectangular area of ​​the ship license plate in the picture is obtained and the image of the area is extracted using OpenCV, and the histogram equalization algorithm is used to enhance the contrast of the picture;

[0180] S11: Get the image extracted in step S10, use the PaddleOCR framework and the general text recognition model to recognize the text information in the image and obtain the text.

[0181] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be considered as equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A target recognition and tracking method based on the fusion of shipborne navigation radar and ball camera, characterized in that: The steps include: Set up the navigation radar, dome camera, GPS and compass on board, and adjust the dome camera's horizontal angle of 0 degrees to align with the bow direction; Obtain ship images and ship license plate images captured by the dome camera, add corresponding labels, and construct corresponding training and validation sets respectively; Train the ship type recognition model and the ship license plate detection model; Obtain the relative position of the target and the ship based on the navigation radar, and calculate the latitude and longitude coordinates of the target based on the GPS and compass; Get the current PTZ of the dome camera. Based on the bow direction and the target's latitude and longitude, calculate the angle required for the dome camera to turn to capture the target direction. Calculate the focal length required for the target to reach the set screen ratio in the dome camera's image. Predict the target's latitude and longitude at the current time based on the target's radar coordinate data, speed data, heading data, and timestamp; Convert the predicted target latitude and longitude coordinates to the position in the current PTZ angle image of the dome camera; Merge the target in the image and the target in the radar ID into the same target; Calculate the local scale, ship's direction, and bow position based on the target's position in the image and the length and width of the identified target in the image, and calculate the corresponding PTZ value of the dome camera; Perform ship type identification and license plate detection to recognize the text on the license plate.

2. The target recognition and tracking method based on the fusion of shipborne navigation radar and ball camera according to claim 1 is characterized in that: The relative position of the suspicious target and the own ship is obtained based on the navigation radar, and the longitude and latitude coordinates of the suspicious target are calculated based on the GPS and compass. Specifically, the following steps are performed: Obtain the longitude and latitude of the ship based on GPS; Obtain the azimuth of the own ship's bow relative to true north based on the compass; Based on the navigation radar, the true azimuth, distance, relative speed, target ID, and target size of all targets are obtained and the corresponding timestamps are stored; The calculation formula for the longitude and latitude coordinates of a suspicious target includes: Where L2 represents the longitude of the suspicious target, L1 represents the longitude of own ship, and R represents the radius of the earth. Among them, B2 is the latitude of the suspicious target, B1 is the latitude of the own ship, A is the true azimuth of the target, and D is the target distance.

3. The target recognition and tracking method based on the fusion of shipborne navigation radar and ball camera according to claim 1 is characterized in that: Calculate the angle required for the camera to turn to capture the target direction, including: Calculate the relative angle R from the bow direction to the target, expressed as: R=AH Where H represents the azimuth of the bow relative to true north, and A represents the true azimuth of the target; Normalize the relative angle R to the range of 0° to 360°; Calculate the camera's steering angle S relative to the target direction a : S a =R-P Where P represents the current angle of the camera. When P = 0, it means that the camera is aligned with the bow direction. Set the steering angle S a Normalized to the range of 0° to 360°.

4. The target recognition and tracking method based on the fusion of shipborne navigation radar and ball camera according to claim 1 is characterized in that: Calculate the focal length required for the target to reach the set screen ratio in the camera image, specifically expressed as: Among them, Z represents the focal length, H object Indicates the actual size of the target, D indicates the distance from the target to the camera, H sensor Indicates the size of the camera CMOS sensor, and ration indicates the proportion of the target in the image.

5. The target recognition and tracking method based on the fusion of shipborne navigation radar and ball camera according to claim 1 is characterized in that: Based on the target's radar coordinate data, speed data, heading data, and timestamp, the target's latitude and longitude at the current time are predicted, specifically including: A polynomial regression model is constructed based on the historical data of the target latitude and longitude coordinates, which can be expressed as: y=a0+a1t+a2t 2 +…+a n t n Among them, y represents longitude or latitude, t represents time, a0, a1, a2, ...a n represents the coefficient to be determined, and n represents the number of polynomial regressions; Based on the target ID, the historical data of the latitude and longitude coordinates of m targets are obtained, and the corresponding timestamps are represented as t1, t2, ..., t m , and its corresponding longitude or latitude is expressed as y1,y2,…,y m ; The problem is transformed into a system of linear equations, expressed as: The polynomial regression equation is expressed as: y=X·a Where a=[a0,a1,…a n ] T is the coefficient vector to be found; Solve the coefficients based on the least squares method and minimize the sum of squared errors between the predicted target latitude and longitude and the true target latitude and longitude, expressed as: min(a)‖yX·a‖ 2 At each time point t i Extract to the nth power in sequence to form the design matrix X, and solve the coefficient vector as follows: a=(X T X) -1 X T y Substituting the predicted time into the resulting polynomial equation: Among them, the coefficient vector a is the coefficient of the polynomial regression model [a0, a1, ... a n ].

6. The target recognition and tracking method based on the fusion of shipborne navigation radar and ball camera according to claim 1 is characterized in that: Convert the predicted target latitude and longitude coordinates to the current position in the PTZ angle image of the dome camera, including: Convert the predicted target latitude and longitude coordinates into three-dimensional coordinates; Transform the fixed coordinate system into the three-dimensional coordinate system under the current dome camera posture; Mapping three-dimensional coordinates to two-dimensional camera array; Convert the current dome camera coordinates to pixel coordinates to obtain the coordinates of the target's position on the screen in the dome camera's preset position.

7. The target recognition and tracking method based on the fusion of shipborne navigation radar and ball camera according to claim 6 is characterized in that: Convert the predicted target latitude and longitude coordinates to three-dimensional coordinates, including: y2=h1-h0 The latitude and longitude coordinates of the dome camera are (x0, y0), the predicted latitude and longitude coordinates of the target point are (x1, y1), the three-dimensional coordinates of the target point are (x2, y2, z2), the height of the dome camera is h0, the height of the target point is h1, and the radius of the earth is R. The fixed coordinate system is transformed into the three-dimensional coordinate system of the current dome camera posture, including rotation around the Y axis and rotation around the X axis. The rotation around the Y axis is expressed as: Where ρ represents the horizontal rotation angle; Rotation around the X axis is expressed as: Where, σ represents the pitch rotation angle; Map the three-dimensional coordinates to the two-dimensional camera array, expressed as: Where f represents the focal length; Convert the current camera coordinates to pixel coordinates to get the coordinates of the target position in the preset position of the camera, which is expressed as: Among them, d x d y Indicates the pixel size on the camera's photosensitive element, where the number of pixels on the element is x cam ×y cam , X represents the horizontal coordinate of the target in the picture, and Y represents the vertical coordinate of the target in the picture.

8. The target recognition and tracking method based on the fusion of shipborne navigation radar and ball camera according to claim 1 is characterized in that: Calculate the corresponding PTZ value of the dome camera, including: By adjusting the P and T values, the camera can be rotated and pitched, the focal length of the dome camera can be adjusted, and the PTZ value required to zoom in on the target part and center it can be calculated through the image.

9. The target recognition and tracking method based on the fusion of shipborne navigation radar and ball camera according to claim 1 is characterized in that: Recognize the text of the ship's license plate, including: Get the rectangular area of ​​the ship plate in the picture and use OpenCV to extract the image of the area, and use the histogram equalization algorithm to enhance the image contrast; Based on the extracted area image, the PaddleOCR framework and text recognition model are used to recognize the image text information and obtain the text of the ship license plate.

Images