Visual anti-occlusion tracking method based on AIS and video fusion

Abstract

The invention discloses a visual anti-occlusion tracking method based on AIS and video fusion. The method comprises the following steps: synchronously obtaining AIS data and corresponding video data; preprocessing the AIS data, predicting latitude and longitude coordinates of the AIS data, projecting the latitude and longitude coordinates to a video pixel plane, and generating a ship AIS track; meanwhile, identifying a ship bounding box in a video frame by using a video detection algorithm, and tracking by using a three-stage association anti-occlusion tracking algorithm; the similarity between the AIS trajectory and the video trajectory is calculated through a dynamic time warping algorithm, and the correlation matching of the AIS trajectory and the video trajectory is realized by using a Hungary matching algorithm, so that a stable and anti-shielding ship tracking trajectory fusing AIS and video information is obtained. According to the invention, the problem that pure visual tracking is easy to fail under the shielding condition is effectively solved.

Description

Technical Field

[0001] This invention relates to the field of intelligent maritime and transportation engineering, and in particular to a visual anti-occlusion tracking method based on AIS and video fusion. Background Technology

[0002] In water transportation, ships move slowly and are large in size, which can cause visual obstruction for a long time when they meet. Without the marking of detection boxes, the monitoring perception ability is weakened, which is very detrimental to situational awareness. Existing anti-occlusion strategies of pure vision multi-object tracking algorithms mostly fully explore the detection boxes, but they still fail to track for long-term occlusion problems. In addition, anti-occlusion strategies using AIS and vision fusion tracking mostly predict the position of the occluded target by using the velocity information provided by AIS, and then forcibly replace the occluded detection box. Moreover, due to the error of the prediction method, the effect will also be biased. Summary of the Invention

[0003] This invention provides a visual anti-occlusion tracking method based on AIS and video fusion to overcome the problems of tracking failure caused by long-term occlusion and deviation in performance due to errors in prediction methods.

[0004] To achieve the above objectives, the technical solution of the present invention is as follows: A visual anti-occlusion tracking method based on AIS and video fusion includes: S1. Obtain AIS data and video data corresponding to the AIS data; S2. Preprocess the AIS data to obtain preprocessed AIS data; predict and project the latitude and longitude coordinates of the preprocessed AIS data onto the video pixel plane to obtain the ship's AIS trajectory. S3. Based on the video data corresponding to the AIS data, identify the ships in the video data using a video detection algorithm to obtain the ship's bounding box; use a three-stage association anti-occlusion tracking algorithm to perform anti-occlusion tracking on the ship's bounding box to obtain the ship's video trajectory of the bounding box; The processing procedure of the three-stage correlation anti-occlusion tracking algorithm is as follows: S31. Perform occlusion detection on the bounding boxes of ships in the video data; S32. If there is bounding box occlusion, the ship bounding box is subjected to anti-occlusion processing based on its historical trajectory, and a three-stage association is performed. The next position of the ship bounding box is predicted by Kalman filtering, thereby obtaining the anti-occlusion ship video trajectory of the bounding box. S33. If no occlusion is detected, perform two-stage association to obtain the unoccluded ship video trajectory of the bounding box. S4. Based on the ship's AIS trajectory and the ship's video trajectory with the bounding box, the trajectory similarity is calculated using the dynamic time warping algorithm to obtain the similarity between the ship's AIS trajectory and the ship's video trajectory with the bounding box. Based on the similarity between the ship's AIS trajectory and the ship's video trajectory with the bounding box, the matching result between the ship's AIS trajectory and the ship's video trajectory with the bounding box is obtained using the Hungarian matching algorithm, thereby obtaining the trajectory associated with the AIS data and the video data.

[0005] Furthermore, preprocessing of AIS data includes removing abnormal AIS information and aligning data time. The data time alignment is achieved by using a preset constant velocity model to predict the latitude and longitude coordinates of the AIS data, thus aligning the AIS data with the video data in time. The expression for the constant velocity model is: (1) In the formula, The predicted ship coordinates; The AIS update time to be predicted; for Ship coordinates at that moment This is the current AIS update time; for The ship's speed at any given moment.

[0006] Furthermore, bounding box occlusion detection is performed on the trajectory associated with AIS data and video data. The occlusion detection condition is: by calculating the intersection-union ratio of the overlapping areas between bounding boxes, if the intersection-union ratio is greater than a set threshold, it is judged as occlusion.

[0007] Furthermore, the prediction of the ship's bounding box position at the next moment is based on Kalman filtering, and its execution steps are as follows: S3211. Construct the state vector of the bounding box based on Kalman filtering, with the following expression: (2) In the formula, It is the state vector of the bounding box; It is the x-coordinate of the center point of the bounding box; It is the ordinate of the center point of the bounding box; It is the aspect ratio of the bounding box; It is the bounding box height. , These are the horizontal and vertical velocities of the bounding box, respectively. , These are the rate of change of the aspect ratio and the rate of change of the height, respectively. S3212. Determine the match between AIS data and video trajectory; If the AIS data matches the video trajectory, the velocity components of the bounding box's state vector are updated using the pixel velocities in the AIS data. Kalman filtering is then used to estimate and predict the velocity components in the updated state vector, and the bounding box position when occluded is calculated from these velocity components. The expression for updating the velocity components of the bounding box's state vector is: (3) In the formula, The horizontal axis pixel velocity of AIS data; The vertical axis represents pixel velocity in AIS data; If there is no match between the AIS data and the video trajectory, the velocity component in the state vector is estimated and predicted using Kalman filtering based on the video trajectory data, and the bounding box position when occluded is calculated using the velocity component.

[0008] Furthermore, estimation and prediction using Kalman filtering involves two stages; The first stage is the prediction stage, which can be expressed by the following formula: (4) In the formula, The predicted prior value of the state vector of the bounding box at time k is the predicted position of the tracked ship. This is the optimal estimate of the bounding box at time k-1; The state transition matrix can be represented as: (5) The second stage is the update stage, which is used to update the estimated value of the state vector. The specific steps include: S3221. Predict the covariance matrix to obtain the predicted prior covariance matrix at time k, expressed as: (6) In the formula, Let be the prediction prior covariance matrix at time k; Let be the prediction covariance matrix at time k-1; For transpose; The noise covariance matrix of the prediction process at time k can be expressed as: (7) In the formula, It is a human-preset position noise weight; It is a human-preset speed noise weight; This represents the optimal estimated bounding box height at time k-1; S3222. Based on the predicted prior covariance matrix at time k, the Kalman gain at time k is obtained, expressed as: (8) In the formula, The Kalman gain at time k; The observation matrix; The covariance matrix of the observation noise; Among them, the observation matrix The specific representation is as follows: (9) Covariance matrix of observation noise The specific representation is as follows: (10) In the formula, Predict the prior height of the bounding box at time k; S3223. Based on the Kalman gain at time k and the observed values, the optimal estimate at time k is obtained, expressed as: (11) In the formula, This is the optimal estimate at time k; For the observed values, they can be represented as: (12) In the formula, Let x be the x-coordinate of the center point of the observed bounding box at time k; The ordinate of the center point of the observed bounding box at time k; The aspect ratio of the bounding box observed at time k; The height of the bounding box is observed at time k; Let k be the state vector of the observed bounding box at time k.

[0009] Furthermore, the three-stage association includes a high-confidence bounding box association stage, a supplementary predicted bounding box association stage, and a low-confidence bounding box association stage; The high-confidence bounding box association stage is used to perform the first IOU-based Hungarian matching between bounding boxes with confidence levels higher than a preset threshold and the ship trajectories in the video data, so as to obtain matching pairs between the associated ship trajectories and bounding boxes with confidence levels higher than the threshold. The supplementary predicted bounding box association stage is used to perform a second IOU-based Hungarian matching on the ship trajectories that failed to match in the high-confidence box association stage, so as to obtain matching pairs between the associated ship trajectories and the supplementary predicted bounding boxes; the supplementary predicted bounding boxes are the predicted ship bounding boxes obtained by the anti-occlusion algorithm. In the low-confidence bounding box association stage, the bounding boxes with confidence levels below a preset threshold are matched with the ship trajectories that failed to match in the supplementary predicted bounding box association stage for a third time using IOU-based Hungarian matching to obtain matching pairs between the associated ship trajectories and the bounding boxes with confidence levels below the threshold. If there is a bounding box with a confidence level higher than the threshold and an IOU with any ship trajectory lower than the matching threshold, then initialize it as a new tracking trajectory; The two-stage association includes a high-confidence bounding box association stage and a low-confidence bounding box association stage; The high-confidence bounding box association stage is used to perform the first IOU-based Hungarian matching between bounding boxes with confidence levels higher than a preset threshold and the ship trajectories in the video data, so as to obtain matching pairs between the associated ship trajectories and bounding boxes with confidence levels higher than the threshold. In the low-confidence bounding box association stage, the bounding boxes with confidence levels below a preset threshold are matched with the ship trajectories that failed to match in the high-confidence bounding box association stage, and a second IOU-based Hungarian matching is performed to obtain matching pairs between the associated ship trajectories and the bounding boxes with confidence levels below the threshold. If a bounding box exists with a confidence level higher than the threshold and an IOU with any ship trajectory lower than the matching threshold, it is initialized as a new tracking trajectory.

[0010] Beneficial Effects: This invention presents a visual anti-occlusion tracking method based on AIS and video fusion. By designing a three-stage association anti-occlusion tracking algorithm, it actively detects occlusion. When a target is detected to be temporarily or partially occluded, by combining anti-occlusion processing of historical trajectories with Kalman filter prediction, it can maintain the continuity of the trajectory when the target is temporarily or partially occluded, significantly reducing mismatches and trajectory fragmentation caused by occlusion. Even if the ship is completely occluded, it can still be accurately marked, improving the monitoring visual capability and ensuring that the tracking trajectory is not interrupted during occlusion, greatly reducing the possibility of target loss. In the absence of occlusion, the algorithm automatically switches to a more computationally efficient two-stage association, achieving a good balance between tracking accuracy and computational overhead, enabling the system to adapt to different real-time requirements and application scenarios. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of the visual anti-occlusion tracking method of the present invention; Figure 2 This is a flowchart of the three-stage tracking algorithm in an embodiment of the present invention; Figure 3 This is a diagram illustrating the effect of the visual anti-occlusion tracking method in an embodiment of the present invention. Figure 4 This is a comparison diagram of the visual anti-occlusion tracking method and the method without anti-occlusion algorithm in the embodiments of the present invention; Figure 5 This is a comparison diagram of the visual anti-occlusion tracking method and the simple occlusion box replacement algorithm in the embodiments of the present invention. Detailed Implementation

[0013] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0014] This embodiment provides a visual anti-occlusion tracking method based on the fusion of Automatic Identification System (AIS) and video, including: S1. Obtain AIS data and video data corresponding to the AIS data; S2. Preprocess the AIS data to obtain preprocessed AIS data; predict and project the latitude and longitude coordinates of the preprocessed AIS data onto the video pixel plane to obtain the ship's AIS trajectory. S3. Based on the video data corresponding to the AIS data, identify the ships in the video data using a video detection algorithm to obtain the ship's bounding box; then, use a three-stage correlation anti-occlusion tracking algorithm to perform anti-occlusion tracking on the ship's bounding box to obtain the ship's video trajectory, such as... Figure 1 As shown; The processing procedure of the three-stage correlation anti-occlusion tracking algorithm is as follows: S31. Perform occlusion detection on the bounding boxes of ships in the video data; S32. If there is bounding box occlusion, the ship bounding box is subjected to anti-occlusion processing based on its historical trajectory, and a three-stage association is performed. The next position of the ship bounding box is predicted by Kalman filtering, thereby obtaining the anti-occlusion ship video trajectory of the bounding box. S33. If no occlusion is detected, perform two-stage association to obtain the unoccluded ship video trajectory of the bounding box. S4. Based on the ship's AIS trajectory and the ship's video trajectory with the bounding box, the trajectory similarity is calculated using the dynamic time warping algorithm to obtain the similarity between the ship's AIS trajectory and the ship's video trajectory with the bounding box. Based on the similarity between the ship's AIS trajectory and the ship's video trajectory with the bounding box, the matching result between the ship's AIS trajectory and the ship's video trajectory with the bounding box is obtained using the Hungarian matching algorithm, thereby obtaining the trajectory associated with the AIS data and the video data.

[0015] Specifically, the dynamic time warping algorithm and the Hungarian matching algorithm are existing technologies available to those skilled in the art, and will not be described in detail here; In this embodiment, after obtaining the matching results of the ship's AIS trajectory and the ship's video trajectory of the bounding box, outliers in the matching results are extracted and removed; the outliers are matching results with opposite motion directions, matching results with excessive distances, and matching results with excessive trajectory angles.

[0016] Preferably, preprocessing of AIS data includes removing abnormal AIS information and aligning data time. The data time alignment is achieved by using a preset constant velocity model to predict the latitude and longitude coordinates of the AIS data, thus aligning the AIS data with the video data in time. The expression for the constant velocity model is: (1) In the formula, The predicted ship coordinates; The AIS update time to be predicted; for Ship coordinates at that moment This is the current AIS update time; for The ship's speed at any given moment.

[0017] Specifically, the abnormal AIS information includes: data with incorrect Maritime Mobile Service Identity (MMSI) format, data with significant changes in latitude and longitude within a short period of time, data that is too far from the camera, and data that is behind the camera; The step of converting latitude and longitude coordinates to a pixel plane includes: The expression for transforming the latitude and longitude of AIS information from the WGS84 world coordinate system to the camera coordinate system is: (2) In the formula, These are the three-dimensional coordinates of AIS data in the world coordinate system; These are the three-dimensional coordinates of the AIS data in the camera coordinate system; It is the rotation matrix from the camera coordinate system to the world coordinate system; It is the translation vector from the camera coordinate system to the world coordinate system; Convert the 3D coordinates in the camera coordinate system to 2D coordinates in the pixel plane. The expression is: (3) In the formula, It is the pixel coordinate vector of the projection; and These are the pixel coordinates of the AIS data information; It is the camera's intrinsic parameter matrix; yes The depth axis coordinate value.

[0018] Preferably, bounding box occlusion detection is performed on the trajectory associated with AIS data and video data. The occlusion detection condition is: by calculating the intersection-union ratio of the overlapping areas between bounding boxes, if the intersection-union ratio is greater than a set threshold, it is judged as occlusion.

[0019] Preferably, the prediction of the next position of the ship's bounding box is based on Kalman filtering, and its execution steps are as follows: S3211. Construct the state vector of the bounding box based on Kalman filtering, with the following expression: (4) In the formula, It is the state vector of the bounding box; It is the x-coordinate of the center point of the bounding box; It is the ordinate of the center point of the bounding box; It is the aspect ratio of the bounding box; It is the bounding box height. , These are the horizontal and vertical velocities of the bounding box, respectively. , These are the rate of change of the aspect ratio and the rate of change of the height, respectively. S3212. Determine the match between AIS data and video trajectory; If the AIS data matches the video trajectory, the velocity components of the bounding box's state vector are updated using the pixel velocities in the AIS data. Kalman filtering is then used to estimate and predict the velocity components in the updated state vector, and the bounding box position when occluded is calculated from these velocity components. The expression for updating the velocity components of the bounding box's state vector is: (5) In the formula, The horizontal axis pixel velocity of AIS data; The vertical axis represents pixel velocity in AIS data; If there is no match between the AIS data and the video trajectory, the velocity component in the state vector is estimated and predicted using Kalman filtering based on the video trajectory data, and the bounding box position when occluded is calculated using the velocity component.

[0020] In this embodiment, if the three-stage correlation anti-occlusion tracking algorithm is executed for the first time and occlusion occurs, it directly enters the branch where there is no match between AIS data and video trajectory. Based on the video trajectory data, the velocity component in the state vector is estimated and predicted by Kalman filtering, and the bounding box position when occluded is calculated by the velocity component.

[0021] Preferably, estimation and prediction using Kalman filtering includes two stages; The first stage is the prediction stage, which can be expressed by the following formula: (6) In the formula, The predicted prior value of the state vector of the bounding box at time k is the predicted position of the tracked ship. This is the optimal estimate of the bounding box at time k-1; The state transition matrix can be represented as: (7) The second stage is the update stage, which is used to update the estimated value of the state vector. The specific steps include: S3221. Predict the covariance matrix to obtain the predicted prior covariance matrix at time k, expressed as: (8) In the formula, Let be the prediction prior covariance matrix at time k; Let be the prediction covariance matrix at time k-1; For transpose; The noise covariance matrix of the prediction process at time k can be expressed as: (9) In the formula, It is a human-preset position noise weight; It is a human-preset speed noise weight; This represents the optimal estimated bounding box height at time k-1; S3222. Based on the predicted prior covariance matrix at time k, the Kalman gain at time k is obtained, expressed as: (10) In the formula, The Kalman gain at time k; The observation matrix; The covariance matrix of the observation noise; Among them, the observation matrix The specific representation is as follows: (11) Covariance matrix of observation noise The specific representation is as follows: (12) In the formula, Predict the prior height of the bounding box at time k; S3223. Based on the Kalman gain at time k and the observed values, the optimal estimate at time k is obtained, expressed as: (13) In the formula, This is the optimal estimate at time k; For the observed values, they can be represented as: (14) In the formula, Let x be the x-coordinate of the center point of the observed bounding box at time k; The ordinate of the center point of the observed bounding box at time k; The aspect ratio of the bounding box observed at time k; The height of the bounding box is observed at time k; Let k be the state vector of the observed bounding box at time k.

[0022] Preferably, the three-stage association includes a high-confidence bounding box association stage, a supplementary predicted bounding box association stage, and a low-confidence bounding box association stage; The high-confidence bounding box association stage is used to perform the first IOU-based Hungarian matching between bounding boxes with confidence levels higher than a preset threshold and the ship trajectories in the video data, so as to obtain matching pairs between the associated ship trajectories and bounding boxes with confidence levels higher than the threshold. The supplementary predicted bounding box association stage is used to perform a second IOU-based Hungarian matching on the ship trajectories that failed to match in the high-confidence box association stage, so as to obtain matching pairs between the associated ship trajectories and the supplementary predicted bounding boxes; the supplementary predicted bounding boxes are the predicted ship bounding boxes obtained by the anti-occlusion algorithm. In the low-confidence bounding box association stage, the bounding boxes with confidence levels below a preset threshold are matched with the ship trajectories that failed to match in the supplementary predicted bounding box association stage for a third time using IOU-based Hungarian matching to obtain matching pairs between the associated ship trajectories and the bounding boxes with confidence levels below the threshold. If there is a bounding box with a confidence level higher than the threshold and an IOU with any ship trajectory lower than the matching threshold, then initialize it as a new tracking trajectory; The two-stage association includes a high-confidence bounding box association stage and a low-confidence bounding box association stage; The high-confidence bounding box association stage is used to perform the first IOU-based Hungarian matching between bounding boxes with confidence levels higher than a preset threshold and the ship trajectories in the video data, so as to obtain matching pairs between the associated ship trajectories and bounding boxes with confidence levels higher than the threshold. In the low-confidence bounding box association stage, the bounding boxes with confidence levels below a preset threshold are matched with the ship trajectories that failed to match in the high-confidence bounding box association stage, and a second IOU-based Hungarian matching is performed to obtain matching pairs between the associated ship trajectories and the bounding boxes with confidence levels below the threshold. If a bounding box exists with a confidence level higher than the threshold and an IOU with any ship trajectory lower than the matching threshold, it is initialized as a new tracking trajectory.

[0023] In this embodiment, the specific steps for obtaining the ship's video trajectory are as follows: First, based on the ship trajectory of the video data to be associated, a bounding box representing the ship trajectory is obtained; Secondly, the intersection-union matrix (IOU) is calculated between the bounding box representing the ship's trajectory and all unassigned bounding boxes in the current frame: Finally, based on the IOU matrix, the optimal match is obtained by using the Hungarian algorithm to obtain the bounding box that best matches the ship trajectory, thus obtaining the ship video trajectory of the bounding box.

[0024] In this embodiment, the AIS data and the corresponding video data are the FVessel dataset, which mainly includes 26 video clips and the corresponding AIS data collected by the Hikvision DS-2DC4423IW-D spherical camera and the Celeron AIS9000-08 type BAIS receiver in the Wuhan section of the Yangtze River.

[0025] In a specific embodiment, such as Figure 3 As shown, the visual anti-occlusion tracking method of this embodiment can correctly identify occluded ships in the image; ships not associated with AIS data are also identified. like Figure 4 As shown, in images without occlusion algorithms, no occluded ships are identified; however, the visual anti-occlusion tracking method in this embodiment can still identify and mark occluded ships even under severe occlusion, greatly enhancing the monitoring capabilities of the camera. like Figure 5As shown, the simple replacement occlusion box algorithm does not mark the bounding box on the ship when there is occlusion, and it does not accurately track the target ship; the visual anti-occlusion tracking method in this embodiment can accurately track the target ship.

[0026] The present invention has the following beneficial effects: This invention presents a visual anti-occlusion tracking method based on AIS and video fusion. By designing a three-stage correlation anti-occlusion tracking algorithm, it actively detects occlusion. When a target is detected to be temporarily or partially occluded, it combines anti-occlusion processing of historical trajectories with Kalman filter prediction to maintain trajectory continuity during temporary or partial occlusion, significantly reducing mismatches and trajectory fragmentation caused by occlusion. Even if a ship is completely occluded, it can still be accurately marked, improving monitoring visual capabilities and ensuring uninterrupted tracking during occlusion, greatly reducing target loss. In the absence of occlusion, the algorithm automatically switches to a more computationally efficient two-stage correlation, achieving a good balance between tracking accuracy and computational overhead, enabling the system to adapt to different real-time requirements and application scenarios.

[0027] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A visual anti-occlusion tracking method based on AIS and video fusion, characterized in that, include: S1. Obtain AIS data and video data corresponding to the AIS data; S2. Preprocess the AIS data to obtain preprocessed AIS data; The latitude and longitude coordinates of the preprocessed AIS data are predicted and projected onto the video pixel plane to obtain the ship's AIS trajectory. S3. Based on the video data corresponding to the AIS data, identify the ships in the video data using a video detection algorithm to obtain the ship's bounding box; use a three-stage association anti-occlusion tracking algorithm to perform anti-occlusion tracking on the ship's bounding box to obtain the ship's video trajectory of the bounding box; The processing procedure of the three-stage correlation anti-occlusion tracking algorithm is as follows: S31. Perform occlusion detection on the bounding boxes of ships in the video data; S32. If there is bounding box occlusion, the ship bounding box is subjected to anti-occlusion processing based on its historical trajectory, and a three-stage association is performed. The next position of the ship bounding box is predicted by Kalman filtering, thereby obtaining the anti-occlusion ship video trajectory of the bounding box. S33. If no occlusion is detected, perform two-stage association to obtain the unoccluded ship video trajectory of the bounding box. S4. Based on the ship's AIS trajectory and the ship's video trajectory with the bounding box, the trajectory similarity is calculated using the dynamic time warping algorithm to obtain the similarity between the ship's AIS trajectory and the ship's video trajectory with the bounding box. Based on the similarity between the ship's AIS trajectory and the ship's video trajectory with the bounding box, the matching result between the ship's AIS trajectory and the ship's video trajectory with the bounding box is obtained using the Hungarian matching algorithm, thereby obtaining the trajectory associated with the AIS data and the video data.

2. The visual anti-occlusion tracking method based on AIS and video fusion according to claim 1, characterized in that, Preprocessing of AIS data includes removing abnormal AIS information and aligning data time. The data time alignment is achieved by using a preset constant velocity model to predict the latitude and longitude coordinates of the AIS data, thus aligning the AIS data with the video data in time. The expression for the constant velocity model is: （1） In the formula, The predicted ship coordinates; The AIS update time to be predicted; for Ship coordinates at that moment This is the current AIS update time; for The ship's speed at any given moment.

3. The visual anti-occlusion tracking method based on AIS and video fusion according to claim 1, characterized in that, Boundary box occlusion detection is performed on the trajectory associated with AIS data and video data. The occlusion detection condition is: by calculating the intersection-union ratio of the overlapping areas between the bounding boxes, if the intersection-union ratio is greater than a set threshold, it is judged as occlusion.

4. The visual anti-occlusion tracking method based on AIS and video fusion according to claim 1, characterized in that, The prediction of the ship's next bounding box position is based on Kalman filtering, and its execution steps are as follows: S3211. Construct the state vector of the bounding box based on Kalman filtering, with the following expression: （2） In the formula, It is the state vector of the bounding box; It is the x-coordinate of the center point of the bounding box; It is the ordinate of the center point of the bounding box; It is the aspect ratio of the bounding box; It is the bounding box height. , These are the horizontal and vertical velocities of the bounding box, respectively. , These are the rate of change of the aspect ratio and the rate of change of the height, respectively. S3212. Determine the match between AIS data and video trajectory; If the AIS data matches the video trajectory, the velocity components of the bounding box's state vector are updated using the pixel velocities in the AIS data. Kalman filtering is then used to estimate and predict the velocity components in the updated state vector, and the bounding box position when occluded is calculated from these velocity components. The expression for updating the velocity components of the bounding box's state vector is: （3） In the formula, The horizontal axis pixel velocity of AIS data; The vertical axis represents pixel velocity in AIS data; If there is no match between the AIS data and the video trajectory, the velocity component in the state vector is estimated and predicted using Kalman filtering based on the video trajectory data, and the bounding box position when occluded is calculated using the velocity component.

5. The visual anti-occlusion tracking method based on AIS and video fusion according to claim 4, characterized in that, Estimation and prediction using Kalman filtering involves two stages; The first stage is the prediction stage, which can be expressed by the following formula: （4） In the formula, The predicted prior value of the state vector of the bounding box at time k is the predicted position of the tracked ship. This is the optimal estimate of the bounding box at time k-1; The state transition matrix can be represented as: ；（5） The second stage is the update stage, which is used to update the estimated value of the state vector. The specific steps include: S3221. Predict the covariance matrix to obtain the predicted prior covariance matrix at time k, expressed as: （6） In the formula, Let be the prediction prior covariance matrix at time k; Let be the prediction covariance matrix at time k-1; For transpose; The noise covariance matrix of the prediction process at time k can be expressed as: （7） In the formula, These are artificially preset location noise weights; It is a human-preset speed noise weight; This represents the optimal estimated bounding box height at time k-1; S3222. Based on the predicted prior covariance matrix at time k, the Kalman gain at time k is obtained, expressed as: （8） In the formula, The Kalman gain at time k; The observation matrix; The covariance matrix of the observation noise; Among them, the observation matrix The specific representation is as follows: ；（9） Covariance matrix of observation noise The specific representation is as follows: （10） In the formula, Predict the prior height of the bounding box at time k; S3223. Based on the Kalman gain at time k and the observed values, the optimal estimate at time k is obtained, expressed as: （11） In the formula, This is the optimal estimate at time k; For the observed values, they can be represented as: （12） In the formula, Let x be the x-coordinate of the center point of the observed bounding box at time k; The ordinate of the center point of the observed bounding box at time k; The aspect ratio of the bounding box observed at time k; The height of the bounding box is observed at time k; Let k be the state vector of the observed bounding box at time k.

6. The visual anti-occlusion tracking method based on AIS and video fusion according to claim 1, characterized in that, The three-stage association includes a high-confidence bounding box association stage, a supplementary predicted bounding box association stage, and a low-confidence bounding box association stage; The high-confidence bounding box association stage is used to perform the first IOU-based Hungarian matching between bounding boxes with confidence levels higher than a preset threshold and the ship trajectories in the video data, so as to obtain matching pairs between the associated ship trajectories and bounding boxes with confidence levels higher than the threshold. The supplementary predicted bounding box association stage is used to perform a second IOU-based Hungarian matching on the ship trajectories that failed to match in the high-confidence box association stage, so as to obtain matching pairs between the associated ship trajectories and the supplementary predicted bounding boxes; the supplementary predicted bounding boxes are the predicted ship bounding boxes obtained by the anti-occlusion algorithm. In the low-confidence bounding box association stage, the bounding boxes with confidence levels below a preset threshold are matched with the ship trajectories that failed to match in the supplementary predicted bounding box association stage for a third time using IOU-based Hungarian matching to obtain matching pairs between the associated ship trajectories and the bounding boxes with confidence levels below the threshold. If there is a bounding box with a confidence level higher than the threshold and an IOU with any ship trajectory lower than the matching threshold, then initialize it as a new tracking trajectory; The two-stage association includes a high-confidence bounding box association stage and a low-confidence bounding box association stage; The high-confidence bounding box association stage is used to perform the first IOU-based Hungarian matching between bounding boxes with confidence levels higher than a preset threshold and the ship trajectories in the video data, so as to obtain matching pairs between the associated ship trajectories and bounding boxes with confidence levels higher than the threshold. In the low-confidence bounding box association stage, the bounding boxes with confidence levels below a preset threshold are matched with the ship trajectories that failed to match in the high-confidence bounding box association stage, and a second IOU-based Hungarian matching is performed to obtain matching pairs between the associated ship trajectories and the bounding boxes with confidence levels below the threshold. If a bounding box exists with a confidence level higher than the threshold and an IOU with any ship trajectory lower than the matching threshold, it is initialized as a new tracking trajectory.

Images