A multi-target tracking method for sea scenes

Abstract

A multi-target tracking method for sea scenes includes step S1, YOLOv8m detection model training; step S2, sending a picture into the YOLOv8m detection model to output target detection frame information; step S3, judging whether the current picture is the first frame picture, if so, initializing the multi-target tracker, entering step S4, if not, using the camera motion compensation algorithm to perform a registration operation on the picture, entering step S4; step S4, sending the picture and the target detection frame information into the multi-target tracker, matching the target detection frame and the trajectory prediction frame based on IoU, ReID features, and Mahalanobis distance, generating new trajectory information and updating the extended Kalman filter; step S5, judging whether the current picture is the last frame picture, if so, ending, outputting the tracking trajectory set, if not, going to step S2. The present invention improves the stable tracking capability of the unmanned boat for obstructing targets in sea scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of unmanned autonomous navigation of ships, and in particular to a multi-target tracking method for sea surface scenes. Background Art

[0002] Autonomous navigation of unmanned ships is one of the hot issues in the current research of ocean and ships, and multi-target tracking in sea scenes is the key link to realize autonomous navigation of ships, and plays a vital role in the environmental perception of unmanned boats. Complex sea environments and special application scenarios will affect the performance of target tracking methods, such as violent camera movement caused by ship navigation, partial or complete occlusion of targets due to waves, mutual occlusion of docked ships, changes in target posture caused by ship movement, water droplets attached to camera lenses, and even thick fog at sea and reflections on the water surface.

[0003] With the deepening of research, a large number of excellent target trackers have emerged in the field of target tracking, such as the DeepSORT algorithm based on detection and tracking, the ByteTrack algorithm, the FairMOT algorithm based on joint detection and tracking, the CenterTrack algorithm, the MOTR algorithm based on Transformer, the MeMOTR algorithm, etc. Considering from multiple angles such as industrial deployment, tracking accuracy, and speed, the DeepSORT algorithm is one of the best choices. The algorithm absorbs the advantages of the SORT algorithm and incorporates deep learning technologies such as ReID feature extraction, so that the entire algorithm achieves a good balance in tracking accuracy and speed, and the tracking effect is stable, so it is widely used in the industry. However, the target tracking method that relies solely on DeepSORT is difficult to achieve stable tracking for moving cameras and complex sea scenes, and is prone to ID switching, which is not conducive to the obstacle avoidance operation of unmanned boats during autonomous navigation. Summary of the invention

[0004] In view of the problem that the existing tracking methods are not stable enough in tracking sea surface targets when unmanned boats are autonomously navigating, the present invention proposes a multi-target tracking method for sea surface scenes based on DeepSORT, supplemented by a camera motion compensation algorithm and a Mahalanobis distance matching algorithm.

[0005] In order to solve the above technical problems, the present invention adopts the following technical methods: a multi-target tracking method for sea surface scenes, comprising:

[0006] Step S1, using the camera carried by the unmanned boat to collect sea scene images, and training the YOLOv8m detection model after preprocessing the images;

[0007] Step S2, sending the image to be detected into the trained YOLOv8m detection model to perform sea scene target detection and output target detection frame information;

[0008] Step S3, determine whether the current image is the first frame. If so, initialize the multi-target tracker and then enter step S4. If not, use the camera motion compensation algorithm to perform image registration between two adjacent frames of the image, calculate the radiation matrix, and use the radiation matrix to correct the prior estimated position predicted by the extended Kalman filter, and then enter step S4;

[0009] Step S4, sending the image and target detection frame information to the multi-target tracker, the extended Kalman filter generates a trajectory prediction frame, and the target detection frame and the trajectory prediction frame are matched based on IoU, ReID features, and Mahalanobis distance, so as to generate new trajectory information and update the extended Kalman filter;

[0010] Step S5, determine whether the current picture is the last frame of the picture, if so, end the multi-target tracker and output the final tracking trajectory set, if not, go to step S2.

[0011] Furthermore, in step S1, image preprocessing includes: firstly annotating the image sequence in MOT format by semi-automatic annotation software, then converting the image into a YOLO format file, and dividing it into a training set, a validation set and a test set according to a certain ratio.

[0012] Furthermore, in step S1, when the YOLOv8m detection model is trained on the image:

[0013] 1) Use data enhancement methods including image flipping, translation, cropping, HSV saturation enhancement, and HSV value enhancement;

[0014] 2) Select SGD as the optimizer, set the initial learning rate to 0.01, the decay rate to 0.0005, and the momentum to 0.937;

[0015] 3) Use the pre-trained weights provided by YOLOv8, use the warm-up learning strategy for the first three training rounds, and use the cosine annealing strategy for the subsequent rounds;

[0016] 4) Use binary cross entropy loss as the category loss function, judge "whether it is this category" for each category, and output the confidence; the calculation formula of the binary cross entropy loss is as follows:

[0017] (1)

[0018] in, is the binary cross entropy loss; is the number of prediction samples; is the number of categories; For the The samples belong to Probability of class; is the number of positive samples, Indicates Whether the class target exists in the trajectory prediction box In; use the Sigmoid function to calculate the probability , the calculation formula is as follows:

[0019] (2)

[0020] in, The output of the model is The samples belong to The raw score of the class;

[0021] 5) The target probability loss and bounding box loss use the CIoU+DFL loss function;

[0022] CIoU loss The calculation formula is as follows:

[0023] (3)

[0024] in, Represents the intersection-over-union ratio of the target detection box and the real box; Represent the center points of the target detection box and the real box respectively; Represents the Euclidean distance between two center points; Represents the diagonal distance of the minimum enclosed area between the target detection box and the real box; represents the penalty factor, which makes the aspect ratio of the real frame and the target detection frame tend to be consistent in the positive direction;

[0025] The expression of DFL loss is as follows:

[0026] (4)

[0027] in, Indicates DFL loss; , They are the predicted value and near-predicted value output by the YOLOv8m detection model network respectively; , , They are the actual value of the label, the integral value of the label, and the integral value of the adjacent labels.

[0028] Furthermore, in step S3, when the camera motion compensation algorithm is used to process the image:

[0029] 1) Calculate the feature points in the image background through corner detection, extract key points, use sparse optical flow for feature tracking, and delete local outliers;

[0030] 2) Calculate the radiation matrix using the random sampling consensus algorithm , thereby extending the trajectory prediction frame obtained by the extended Kalman filter from The coordinate system of the frame is transformed to In the coordinate system of the frame; the radiation matrix The calculation formula is as follows:

[0031] (5)

[0032] in, are the rotation and scale matrices in the radial matrix, represents a 2×2 matrix; is the translation matrix in the radiation matrix, represents a 2×1 column vector; is the corresponding coordinate transformation element, ;

[0033] 3) Consider the state vector in the multi-target tracking algorithm ,in Represents the center point position coordinates; Represents the aspect ratio, i.e. width / height; Represents height; Respectively represent The rates of change of the four variables are defined as , for:

[0034] (6)

[0035] The prediction covariance matrix of the extended Kalman filter after camera motion compensation is calculated according to the following formula (7) and in the The predicted state vector at the frame time completes the image registration between two adjacent frames;

[0036] (7)

[0037] in, , They are the front and back extended Kalman filters of camera motion compensation in the first The predicted state vector at the frame time; , They are the front and back extended Kalman filters of camera motion compensation in the first Prediction covariance matrix at frame time.

[0038] Furthermore, in step S4, when the target detection frame and the Kalman filter prediction frame are matched based on ReID features, IoU, and Mahalanobis distance:

[0039] 1) No.t -1 frame trajectory is formed into trajectory prediction box through the prediction stage of extended Kalman filter;

[0040] 2) The information of the confirmed trajectory prediction box and the target detection box is sent to the cascade matching, and the cost matrix is ​​constructed according to the motion information and ReID feature information, and the first matching is completed through the Hungarian algorithm;

[0041] 3) The predicted frame of the unconfirmed dynamic trajectory is matched based on IoU, and the cost matrix is ​​constructed based on IoU together with the trajectory and target that failed to match the first time, and the second matching is completed through the Hungarian algorithm;

[0042] 4) The second failed matching trajectory and the second failed matching target are matched based on the Mahalanobis distance, and the cost matrix is ​​constructed according to the Mahalanobis distance between the trajectory prediction box and the target detection box, thereby completing the third matching based on the Mahalanobis distance;

[0043] 5) The successful matching trajectories in the three matching processes are sent to the extended Kalman filter update stage. The unmatched trajectories in the third matching process, including the unconfirmed trajectories and the confirmed trajectories that exceed the maximum number of unmatched matches, are directly deleted. The new trajectories formed by the trajectories that do not exceed the maximum number of unmatched matches and the matching failed targets are sent to the trajectory manager, and a new ID is assigned to the new trajectory. Combined with the trajectory that has completed the extended Kalman filter update stage, they form the first t The trajectory set of the frame;

[0044] 6) When the next frame arrives, continue with step 1) and repeat this cycle until the tracking is completed.

[0045] Furthermore, in step S4, the matching process based on the Mahalanobis distance is as follows:

[0046] 1) After ReID feature matching and IoU-based matching, a third matching is performed for the remaining unmatched target detection boxes and tracking trajectories, i.e., matching based on Mahalanobis distance;

[0047] 2) Calculate the target detection box And trajectory prediction box Mahalanobis distance , the formula is as follows:

[0048] (8)

[0049] in, represents the covariance matrix;

[0050] 3) Define the cost matrix , the cost matrix element The calculation formula is as follows:

[0051] (9)

[0052] Among them, the parameters is the maximum Mahalanobis matching distance;

[0053] 4) Complete the cost matrix calculation based on Mahalanobis distance, use the Hungarian algorithm for matching, obtain the global optimal solution, and complete the third matching.

[0054] Preferably, during trajectory prediction and updating, when using an extended Kalman filter to fit the nonlinear motion problem of the target:

[0055] 1) Initialize state variables and error covariance matrix;

[0056] 2) Prediction phase, including calculation of the system state prediction equation as shown in equation (10) and the error covariance matrix as shown in equation (11): ;

[0057] (10)

[0058] (11)

[0059] in, is the observation noise, is the process noise, is the nonlinear state transfer function of the system; is the process noise covariance matrix, for time right The Jacobian matrix of for time right The Jacobian matrix of ; is the extended Kalman filter after camera motion compensation in the first The predicted state vector at the frame time; is the extended Kalman filter after camera motion compensation in the first The prediction covariance matrix at the frame time;

[0060] 3) Calculate Kalman gain :

[0061] (12)

[0062] in, is the observation noise covariance matrix, for Time nonlinear measurement function right The Jacobian matrix of for time right The Jacobian matrix of ;

[0063] 4) Update phase: update the state estimate according to the following equation (13), and update the error covariance matrix according to the following equation (14);

[0064] (13)

[0065] (14)

[0066] in, is the updated state estimate; is the updated error covariance matrix; is a nonlinear measurement function; is the identity matrix.

[0067] In order to make up for the defects of DeepSORT algorithm, the present invention proposes a multi-target tracking method for sea surface scenes. The method is mainly based on DeepSORT algorithm, uses camera motion compensation algorithm to reduce or eliminate the influence of camera motion, uses Mahalanobis distance-based matching to deal with the possible failure of associating target detection frame and tracking frame, and uses extended Kalman filter algorithm to fit the nonlinear motion problem of tracking target. Compared with the prior art, the multi-target tracking method for sea surface scenes proposed by the present invention realizes stable tracking of multiple targets in sea surface scenes, reduces target ID switching caused by camera motion, and improves the stable tracking capability of unmanned boats for obstructing targets in sea surface scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 is a flow chart of a multi-target tracking method for sea surface scenes involved in the present invention;

[0069] Figure 2 It is a flow chart of a camera motion compensation algorithm in a multi-target tracking method for sea surface scenes involved in the present invention;

[0070] Figure 3 It is a flow chart of a target tracking algorithm in a multi-target tracking method for sea surface scenes involved in the present invention;

[0071] Figure 4 It is a flow chart of the Mahalanobis distance matching algorithm in the multi-target tracking method for sea surface scenes involved in the present invention;

[0072] FIG. 5 is a flow chart of an extended Kalman filter algorithm in a multi-target tracking method for sea surface scenes according to the present invention. DETAILED DESCRIPTION

[0073] In order to facilitate the understanding of those skilled in the art, the present invention is further described below in conjunction with embodiments and drawings. The contents mentioned in the implementation modes are not intended to limit the present invention.

[0074] A multi-target tracking method for sea surface scenes, comprising:

[0075] Step S1, using a high-definition industrial camera carried by an unmanned boat to collect sea scene images, first annotate the image sequence in MOT format using semi-automatic annotation software, then convert it into a YOLO format file, and divide it into a training set, a validation set, and a test set according to a certain ratio, and then select a suitable training strategy to train the YOLOv8m detection model. Specifically, the training strategy includes:

[0076] 1) Use data enhancement methods including image flipping, translation, cropping, HSV saturation enhancement, and HSV value enhancement;

[0077] 2) Select SGD as the optimizer, set the initial learning rate to 0.01, the decay rate to 0.0005, and the momentum to 0.937;

[0078] 3) Use the pre-trained weights provided by YOLOv8, use the warm-up strategy for the first three training rounds, and use the cosine annealing strategy for the subsequent rounds;

[0079] 4) Use Binary Cross Entropy Loss (BCELoss) as the category loss function, judge "whether it is this category" for each category, and output the confidence level; the calculation formula of Binary Cross Entropy Loss is as follows:

[0080] (1)

[0081] in, is the binary cross entropy loss; is the number of prediction samples; is the number of categories; For the The samples belong to Probability of class; is the number of positive samples, Indicates Whether the class target exists in the trajectory prediction box In; use the Sigmoid function to calculate the probability , the calculation formula is as follows:

[0082] (2)

[0083] in, The output of the model is The samples belong to The raw score of the class;

[0084] 5) The target probability loss and bounding box loss use the CIoU+DFL loss function;

[0085] CIoU loss combines the advantages of IoU's cross-area loss, DIOU's center point distance loss, and aspect ratio loss. The calculation formula is as follows:

[0086] (3)

[0087] in, represents the intersection-union ratio (intersection-union ratio) of the target detection box and the true box, where the true box is the image box annotated in MOT format by the semi-automatic annotation software in step S1; Represent the center points of the target detection box and the real box respectively; Represents the Euclidean distance between two center points; Represents the diagonal distance of the minimum enclosed area between the target detection box and the real box; represents the weight parameter, Used to measure the consistency of aspect ratio, Represents the penalty factor, which makes the aspect ratio of the true box and the target detection box tend to be consistent in the positive direction.

[0088] The expression of DFL loss is as follows:

[0089] (4)

[0090] in, Indicates DFL loss; , They are the predicted value and near-predicted value output by the YOLOv8m detection model network respectively; , , They are the actual value of the label, the integral value of the label, and the integral value of the adjacent labels.

[0091] Step S2: Send the image to be detected into the trained YOLOv8m detection model to perform sea scene target detection and output the target detection frame information.

[0092] Step S3, determine whether the current image is the first frame. If so, initialize the multi-target tracker and then enter step S4. If not, use the camera motion compensation algorithm to perform image registration between two adjacent frames and calculate the radiation matrix, so that the radiation matrix can be used to correct the prior estimated position predicted by the extended Kalman filter, and then enter step S4. Figure 2As shown, the specific operation of the camera motion compensation algorithm is:

[0093] 1) Calculate the feature points in the background of the image through corner detection, extract key points, use sparse optical flow for feature tracking, and delete local outliers to more accurately estimate background motion;

[0094] 2) Calculate the radiation matrix using the random sampling consensus algorithm (RANSAC) , thereby extending the trajectory prediction frame obtained by the extended Kalman filter from The coordinate system of the frame is transformed to In the frame's coordinate system; the radiation matrix The calculation formula is as follows:

[0095] (5)

[0096] in, are the rotation and scale matrices in the radial matrix, represents a 2×2 matrix; is the translation matrix in the radiation matrix, represents a 2×1 column vector; is the corresponding coordinate transformation element, ;

[0097] 3) Consider the state vector in the tracking algorithm ,in Represents the center point position coordinates; Represents the aspect ratio, i.e. width / height; Represents height; Respectively represent The rates of change of the four variables are defined as , for:

[0098] (6)

[0099] The prediction covariance matrix of the extended Kalman filter after camera motion compensation is calculated according to the following formula (7) and in the The predicted state vector at the frame time is used to complete the image registration between two adjacent frames;

[0100] (7)

[0101] in, , They are the front and back extended Kalman filters of camera motion compensation in the first The predicted state vector at the frame time; , They are the front and back extended Kalman filters of camera motion compensation in the first Prediction covariance matrix at frame time.

[0102] It is worth mentioning here that the predicted state vector obtained by formula (7) is and the prediction covariance matrix It will be used for subsequent extended Kalman filter updates.

[0103] Step S4, the image and target detection frame information are sent to the multi-target tracker, so that the target detection frame and the trajectory prediction frame are matched based on IoU, ReID features, and Mahalanobis distance, thereby generating new trajectory information and updating the extended Kalman filter. The working process of the multi-target tracker is as follows Figure 3 As shown, the specific steps include:

[0104] 1) No. t -1 frame trajectory is formed into trajectory prediction box through the prediction stage of extended Kalman filter;

[0105] 2) The information of the confirmed trajectory prediction box and the target detection box is sent to the cascade matching, and the cost matrix is ​​constructed according to the motion information and appearance feature (ReID feature) information, and the first matching is completed through the Hungarian algorithm;

[0106] 3) The predicted frame of the unconfirmed dynamic trajectory is matched based on IoU, and the cost matrix is ​​constructed based on IoU together with the trajectory and target that failed to match the first time, and the second matching is completed through the Hungarian algorithm;

[0107] 4) The second failed matching trajectory and the second failed matching target are matched based on the Mahalanobis distance, and the cost matrix is ​​constructed according to the Mahalanobis distance between the trajectory prediction box and the target detection box, thereby completing the third matching based on the Mahalanobis distance;

[0108] 5) The successful matching trajectories in the three matching processes are sent to the extended Kalman filter update stage. The unmatched trajectories in the third matching process, including the unconfirmed trajectories and the confirmed trajectories that exceed the maximum number of unmatched matches, are directly deleted. The new trajectories formed by the trajectories that do not exceed the maximum number of unmatched matches and the matching failed targets are sent to the trajectory manager, and a new ID is assigned to the new trajectory. Combined with the trajectory that has completed the extended Kalman filter update stage, they form the first t The trajectory set of the frame;

[0109] 6) When the next frame arrives, continue with step 1) and repeat this cycle until the tracking is completed.

[0110] The third matching mentioned above is based on the Mahalanobis distance matching, such as Figure 4 As shown, the matching process includes the following steps:

[0111] 1) After ReID feature matching and IoU-based matching, a third matching is performed for the remaining unmatched target detection boxes and tracking trajectories, i.e., matching based on Mahalanobis distance;

[0112] 2) Calculate the target detection box And trajectory prediction box Mahalanobis distance , the formula is as follows:

[0113] (8)

[0114] in, represents the covariance matrix;

[0115] 3) Define the cost matrix , the cost matrix element The calculation formula is as follows:

[0116] (9)

[0117] Among them, the parameters is the maximum Mahalanobis matching distance. This threshold is used to limit the Mahalanobis distance matching between the detection box and the tracking box to avoid some obvious false associations.

[0118] 4) Complete the cost matrix calculation based on Mahalanobis distance, use the Hungarian algorithm for matching, obtain the global optimal solution, and complete the third matching.

[0119] In the aforementioned trajectory prediction and updating process, the present invention introduces an extended Kalman filter to replace the Kalman filter to fit the nonlinear motion problem of the target, such as Figure 5 As shown, the process is as follows:

[0120] 1) Initialize state variables and error covariance matrix;

[0121] 2) Prediction phase, including calculation of the system state prediction equation as shown in equation (10) and the error covariance matrix as shown in equation (11): ;

[0122] (10)

[0123] (11)

[0124] in, is the observation noise, is the process noise, is the nonlinear state transfer function of the system; is the process noise covariance matrix, for time right The Jacobian matrix of for time right The Jacobian matrix of ; is the extended Kalman filter after camera motion compensation in the first The predicted state vector at the frame time; is the extended Kalman filter after camera motion compensation in the first The prediction covariance matrix at the frame time;

[0125] 3) Calculate Kalman gain :

[0126] (12)

[0127] in, is the observation noise covariance matrix, for Time nonlinear measurement function right The Jacobian matrix of for time right The Jacobian matrix of ;

[0128] 4) Update phase: Update the state estimate according to the following formula (13): , and update the error covariance matrix according to the following formula (14): ;

[0129] (13)

[0130] (14)

[0131] in, is the updated state estimate; is the updated error covariance matrix; is a nonlinear measurement function; is the identity matrix.

[0132] Step S5, determine whether the current picture is the last frame of the picture, if so, end the tracker and output the final tracking trajectory set, if not, go to step S2.

[0133] The above embodiments are preferred implementation schemes of the present invention. In addition, the present invention may also be implemented in other ways. Any obvious replacement without departing from the concept of the present technical solution is within the protection scope of the present invention.

[0134] In order to make it easier for ordinary technicians in the field to understand the improvements of the present invention over the prior art, some drawings and descriptions of the present invention have been simplified, and for the sake of clarity, some other elements are omitted in this application document. Ordinary technicians in the field should realize that these omitted elements may also constitute the content of the present invention.

Claims

1. A multi-target tracking method for sea surface scenes, characterized in that: include: Step S1, using the camera carried by the unmanned boat to collect sea scene images, and training the YOLOv8m detection model after preprocessing the images; When training the YOLOv8m detection model on an image: 1) Use data enhancement methods including image flipping, translation, cropping, HSV saturation enhancement, and HSV value enhancement; 2) Select SGD as the optimizer, set the initial learning rate to 0.01, the decay rate to 0.0005, and the momentum to 0.937; 3) Use the pre-trained weights provided by YOLOv8, use the warm-up learning strategy for the first three training rounds, and use the cosine annealing strategy for the subsequent rounds; 4) Use binary cross entropy loss as the category loss function, judge "whether it is this category" for each category, and output the confidence; the calculation formula of the binary cross entropy loss is as follows: （1） in, is the binary cross entropy loss; is the number of prediction samples; is the number of categories; For the The samples belong to Probability of class; is the number of positive samples, Indicates Whether the class target exists in the trajectory prediction box In; use the Sigmoid function to calculate the probability , the calculation formula is as follows: （2） in, The output of the model is The samples belong to The raw score of the class; 5) The target probability loss and bounding box loss use the CIoU+DFL loss function; CIoU loss The calculation formula is as follows: （3） in, Represents the intersection-over-union ratio of the target detection box and the real box; Represent the center points of the target detection box and the real box respectively; Represents the Euclidean distance between two center points; Represents the diagonal distance of the minimum enclosed area between the target detection box and the real box; represents the penalty factor, which makes the aspect ratio of the real frame and the target detection frame tend to be consistent in the positive direction; The expression of DFL loss is as follows: （4） in, Indicates DFL loss; , They are the predicted value and near-predicted value output by the YOLOv8m detection model network respectively; , , They are the actual value of the label, the integral value of the label, and the integral value of the adjacent labels; Step S2, sending the image to be detected into the trained YOLOv8m detection model to perform sea scene target detection and output target detection frame information; Step S3, determine whether the current image is the first frame. If so, initialize the multi-target tracker and then enter step S4. If not, use the camera motion compensation algorithm to perform image registration between two adjacent frames of the image, calculate the radiation matrix, and use the radiation matrix to correct the prior estimated position predicted by the extended Kalman filter, and then enter step S4; Step S4, sending the image and target detection frame information to the multi-target tracker, the extended Kalman filter generates a trajectory prediction frame, and the target detection frame and the trajectory prediction frame are matched based on IoU, ReID features, and Mahalanobis distance, so as to generate new trajectory information and update the extended Kalman filter; Step S5, determine whether the current picture is the last frame of the picture, if so, end the multi-target tracker and output the final tracking trajectory set, if not, go to step S2.

2. The multi-target tracking method for sea surface scenes according to claim 1, characterized in that: In step S1, image preprocessing includes: firstly annotating the image sequence in MOT format by semi-automatic annotation software, then converting the image into a YOLO format file, and dividing it into a training set, a validation set, and a test set according to a certain ratio.

3. The multi-target tracking method for sea surface scenes according to claim 2 is characterized in that: In step S3, when the camera motion compensation algorithm is used to process the image: 1) Calculate the feature points in the image background through corner detection, extract key points, use sparse optical flow for feature tracking, and delete local outliers; 2) Calculate the radiation matrix using the random sampling consensus algorithm , thereby extending the trajectory prediction frame obtained by the extended Kalman filter from The coordinate system of the frame is transformed to k In the coordinate system of the frame; the radiation matrix The calculation formula is as follows: （5） in, are the rotation and scale matrices in the radial matrix, represents a 2×2 matrix; is the translation matrix in the radiation matrix, represents a 2×1 column vector; is the corresponding coordinate transformation element, ; 3) Consider the state vector in the multi-target tracking algorithm ,in Represents the center point position coordinates; Represents the aspect ratio, i.e. width / height; Represents height; Respectively represent The rates of change of the four variables are defined as , for: （6） The prediction covariance matrix of the extended Kalman filter after camera motion compensation is calculated according to the following formula (7) and in the The predicted state vector at the frame time completes the image registration between two adjacent frames; （7） in, , They are the front and back extended Kalman filters of camera motion compensation in the first The predicted state vector at the frame time; , They are the front and back extended Kalman filters of camera motion compensation in the first Prediction covariance matrix at frame time.

4. The multi-target tracking method for sea surface scenes according to claim 3 is characterized in that: In step S4, when the target detection frame and the Kalman filter prediction frame are matched based on ReID features, IoU, and Mahalanobis distance: 1) No. The frame trajectory is passed through the prediction stage of the extended Kalman filter to form a trajectory prediction box; 2) The information of the confirmed trajectory prediction box and the target detection box is sent to the cascade matching, and the cost matrix is ​​constructed according to the motion information and ReID feature information, and the first matching is completed through the Hungarian algorithm; 3) The predicted frame of the unconfirmed dynamic trajectory is matched based on IoU, and the cost matrix is ​​constructed based on IoU together with the trajectory and target that failed to match the first time, and the second matching is completed through the Hungarian algorithm; 4) The second failed matching trajectory and the second failed matching target are matched based on the Mahalanobis distance, and the cost matrix is ​​constructed according to the Mahalanobis distance between the trajectory prediction box and the target detection box, thereby completing the third matching based on the Mahalanobis distance; 5) The successful matching trajectories in the three matching processes are sent to the extended Kalman filter update stage. The unmatched trajectories in the third matching process, including the unconfirmed trajectories and the confirmed trajectories that exceed the maximum number of unmatched matches, are directly deleted. The new trajectories formed by the trajectories that do not exceed the maximum number of unmatched matches and the matching failed targets are sent to the trajectory manager, and a new ID is assigned to the new trajectory. Combined with the trajectory that has completed the extended Kalman filter update stage, they form the first t The trajectory set of the frame; 6) When the next frame arrives, continue with step 1) and repeat this cycle until the tracking is completed.

5. The multi-target tracking method for sea surface scenes according to claim 4 is characterized in that: In step S4, the matching process based on the Mahalanobis distance is as follows: 1) After ReID feature matching and IoU-based matching, a third matching is performed for the remaining unmatched target detection boxes and tracking trajectories, i.e., matching based on Mahalanobis distance; 2) Calculate the target detection box And trajectory prediction box Mahalanobis distance , the formula is as follows: （8） in, represents the covariance matrix; 3) Define the cost matrix , the cost matrix element The calculation formula is as follows: （9） Among them, the parameters is the maximum Mahalanobis matching distance; 4) Complete the cost matrix calculation based on Mahalanobis distance, use the Hungarian algorithm for matching, obtain the global optimal solution, and complete the third matching.

6. The multi-target tracking method for sea surface scenes according to claim 5, characterized in that: In step S4, during the trajectory prediction and update process, when using the extended Kalman filter to fit the target nonlinear motion problem: 1) Initialize state variables and error covariance matrix; 2) Prediction phase, including calculation of the system state prediction equation as shown in equation (10) and the error covariance matrix as shown in equation (11): ; （10） （11） in, is the observation noise, is the process noise, is the nonlinear state transfer function of the system; is the process noise covariance matrix, for time right The Jacobian matrix of for time right The Jacobian matrix of ; is the extended Kalman filter after camera motion compensation in the first The predicted state vector at the frame time; is the extended Kalman filter after camera motion compensation in the first The prediction covariance matrix at the frame time; 3) Calculate Kalman gain : （12） in, is the observation noise covariance matrix, for Time nonlinear measurement function right The Jacobian matrix of for time right The Jacobian matrix of ; 4) Update phase: update the state estimate according to the following equation (13), and update the error covariance matrix according to the following equation (14); （13） （14） in, is the updated state estimate; is the updated error covariance matrix; is a nonlinear measurement function; is the identity matrix.

Images