Fusion method of AIS information and video data

Abstract

The invention discloses an AIS information and video data fusion method. The method comprises the following steps: collecting and preprocessing AIS and video data; constructing a constant acceleration model to interpolate the AIS data, and projecting AIS latitude and longitude coordinates to a video pixel coordinate system to obtain a target trajectory; detecting a video ship by using a YOLOX model, and tracking through a MotionTrack algorithm to obtain a video trajectory; ship motion features of the tracks in the AIS and the video data are extracted respectively; constructing a comprehensive measurement function fusing the Euclidean distance, the course and the navigational speed similarity; performing data matching by adopting a Hungary algorithm to obtain a matching result of the AIS and the track in the video data; verifying a matching result through abnormal value elimination and category consistency verification; and according to a matching result, fusing and displaying the matched AIS information and the video image. The AIS information and the visual target information are comprehensively utilized, and the information matching result is determined by using multiple clues, so that the fusion accuracy is improved, and the precision and reliability of multi-source information fusion in maritime affair monitoring are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of ship monitoring, in particular to an AIS information and video data fusion method. BACKGROUND

[0002] There are many devices in the field of navigation to ensure the safety of navigation, but at the same time, the attention of the lookout is also being competed for. In the practice of maritime supervision, the automatic identification system (AIS) and the monitoring camera are two commonly used devices, and the two devices naturally have complementary characteristics. That is, the AIS describes the characteristics of the ship in detail, including static information such as ship name, mobile service identification, and dynamic information such as heading, speed, latitude and longitude. The video is more in line with human intuition and is the most intuitive supervision method. Therefore, it is necessary to fuse the two.

[0003] The AIS and video fusion method is roughly divided into three steps: data processing, space-time alignment, and similarity matching. The data processing step includes denoising, cleaning and filtering of AIS data, target detection and classification algorithm of video, and multi-target tracking algorithm, and the position, category and trajectory of the visual target are obtained. In the space-time alignment step, the existing technical solutions of the AIS and video fusion method mostly use the constant speed model to interpolate the AIS information. Finally, in the similarity matching step, the methods based on Euclidean distance, Frechet distance, dynamic time warping, etc. are fused.

[0004] The existing technology has the following problems: the existing AIS information constant speed model interpolation method cannot accurately correspond to the visual target, because the acceleration information of the ship movement is ignored, and the movement of the ship when the AIS is not updated cannot be reflected. When the ship target is not detected, that is, missed detection, or the ship closes the AIS device, the distance or trajectory-based method will mismatch. The reason is that these methods only use the position as the only clue, and if there is not enough information, it is impossible to distinguish whether the matching result is correct or not; the existing method has high computational complexity and slow fusion speed. SUMMARY

[0005] The present application provides an AIS information and video data fusion method to overcome the above technical problems.

[0006] In order to achieve the above purpose, the technical scheme of the present application is: An AIS information and video data fusion method, comprising: S1, collecting AIS data and video data corresponding to the AIS data through the FVessel data set; cleaning and filtering the abnormal data of the AIS data to obtain preprocessed AIS data; S2, construct a constant acceleration model, apply the constant acceleration model to the preprocessed AIS data for interpolation processing, and obtain a ship target track of the AIS data time-aligned with the video data; S3, project the latitude and longitude coordinates of the preprocessed AIS data to the pixel plane coordinate system of the video data to obtain a projection track of the AIS data on the pixel plane; S4, use the YOLOX model to detect the ships in the video data to obtain the bounding boxes and class information of the ships; track the detected ships by the MotionTrack algorithm, and assign a unique ID to the bounding box of the tracked ship to ensure that the target ID is unchanged in the continuous frames, record the positions of the ships in the continuous frames, and obtain the video data track of the tracked ships; S5, extract the ship motion features in the projection track of the AIS data on the pixel plane and the ship motion features in the video data track of the tracked ships respectively through the projection track of the AIS data on the pixel plane and the video data track of the tracked ships; wherein the ship motion features include ship heading features and ship speed features; S6, construct a comprehensive similarity measurement function to calculate the ship motion feature similarity value between the projection track of the AIS data on the pixel plane and the video data track of the tracked ships; the comprehensive similarity measurement function fuses the Euclidean distance similarity, heading similarity and speed similarity of the ships in the AIS data and the video data; S7, based on the ship motion feature similarity value, use the Hungarian matching algorithm to match the projection track of the AIS data on the pixel plane and the video data track of the tracked ships to obtain the matching result of the projection track of the AIS data on the pixel plane and the video data track of the tracked ships; S8, design outlier rejection and class consistency test to verify the matching result of the projection track of the AIS data on the pixel plane and the video data track of the tracked ships, and take the matching result passing the outlier rejection and class consistency test as the successful matching result; S9, draw an image according to the matching result, fuse the AIS data with the bounding box of the tracked ship, and display the information of the detected ship in the video image to realize the supervision of water transportation.

[0007] Further, the expression of the constant acceleration model is: (1) In the formula, is the current AIS update time; is the last AIS update time; is the time to be interpolated; is the coordinate to be interpolated; is the time ship speed; is the time ship acceleration; is the time coordinate.

[0008] Further, the latitude and longitude coordinates of the preprocessed AIS data are projected to the pixel plane coordinate system of the video data, and the method is as follows: S31, the latitude and longitude coordinates of the preprocessed AIS data are converted to the camera coordinate system, and the expression is: (2) In the formula, is the three-dimensional coordinate of the AIS data in the world coordinate system; is the three-dimensional coordinate of the AIS data in the camera coordinate system; is the rotation matrix of the camera coordinate system to the world coordinate system; is the translation vector of the camera coordinate system to the world coordinate system; S32, according to the parameters of the camera, the three-dimensional coordinate of the AIS data in the camera coordinate system is converted to the pixel plane coordinate system, and the expression is: (3) In the formula, is the projected pixel coordinate vector; , respectively, the pixel coordinates of the AIS data information; is the internal parameter matrix of the camera; is the depth axis coordinate value of

[0009] Further, the expression of the comprehensive similarity measure function is: (4) In the formula, is the Euclidean distance similarity; is the heading similarity; is the speed similarity; , , is the balance weight constant; Wherein, the expression for calculating the Euclidean distance similarity is: (5) In the formula, is the sliding window size, , is the pixel coordinate of the ship visual target at the time, , is Pixel coordinates of the AIS data at time t; The expression for calculating the heading similarity is: (6) In the formula, is the moving direction of the AIS data in the image; is the moving direction of the visual target in the image; is a constant parameter; The expression for calculating the velocity similarity is: (7) In the formula, is the pixel velocity of the AIS data; is the pixel velocity of the visual target; is the width of the bounding box of the visual target; is a scaling factor.

[0010] Further, the outlier rejection is performed by comparing the threshold value obtained by calculation with the value of the penalty corresponding to the matching similarity of the matching result of the projection trajectory of the AIS data in the pixel plane and the video data trajectory of the tracked ship; if the value of the penalty is greater than the threshold value, the corresponding matching result is rejected; otherwise, the corresponding matching result is retained; The category consistency check is performed on the category of the ship in the AIS data and the category of the ship in the video data of the tracked ship; if the categories of the ships are inconsistent, the matching result is deleted; otherwise, the matching result is retained; The matching result obtained by the outlier rejection and the category consistency check is taken as the matching success result.

[0011] Further, the expression of the threshold value is: (8) In the formula, , respectively, are the width and height of the video data; W d is a balance weight constant.

[0012] Further, the abnormal data of the AIS data in step S1 includes: data with incorrect format of water mobile service identification, data with relatively large change in latitude and longitude in a short time, data with too far distance from the camera, and data behind the camera.

[0013] Advantages: The application constructs and applies a constant acceleration model to interpolate the preprocessed AIS data, fully considers the acceleration motion of the ship in the AIS data update interval through the constant acceleration model, and compared with the traditional constant speed model, the ship target trajectory of the AIS data time-aligned with the video data is obtained through the constant acceleration model, which can more truly reflect the actual motion state (such as acceleration, deceleration, turning, etc.) of the ship when the AIS data is not updated. The application not only depends on position information, but also fuses the motion characteristics (heading characteristics and speed characteristics) of the ship for matching; the heading characteristics and speed characteristics are extracted from the AIS projection trajectory and the video detection tracking trajectory respectively, and the comprehensive similarity is calculated, which provides more abundant and more robust matching clues than pure position or trajectory distance; this makes the system still be able to effectively associate by using continuous and stable motion characteristics in the complex scene of missing detection of the ship target in the video or the ship closing the AIS device and the like, and greatly reduces the mismatch risk caused by relying on single position information; The application deletes the matching results with the matching similarity corresponding to the cost greater than the threshold as unreliable matching (abnormal value), which directly filters out the false association caused by serious information loss, noise interference or too large feature difference; at the same time, the ship class detected by the video is forced to be consistent with the ship class recorded in the AIS data, which increases the constraint at the semantic level and effectively prevents the accidental similarity of the motion characteristics but the inconsistency of the actual ship type; the dual verification mechanism of abnormal value deletion and class consistency verification constitutes a powerful error filter, which significantly improves the confidence and accuracy of the final matching result. BRIEF DESCRIPTION OF DRAWINGS

[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0015] Figure 1 The step flow chart of the fusion method of the present application; Figure 2 The implementation flow chart of the fusion method of the present application; Figure 3a The video 1 result graph output in the embodiment of the present application; Figure 3b The video 2 result graph output in the embodiment of the present application; Figure 3c The video 3 result graph output in the embodiment of the present application; Figure 3dThis is the output video 4 result diagram in the embodiment of the present invention; Figure 3e This is the output video 5 result diagram in this embodiment of the invention; Figure 3f This is a video output result diagram (6) from an embodiment of the present invention. Figure 3g This is the output video 7 result diagram in the embodiment of the present invention; Figure 3h This is a video output image (8) from an embodiment of the present invention. Figure 3i This is the output video 9 result diagram in the embodiment of the present invention; Figure 3j This is a video 10 output in an embodiment of the present invention; Figure 4a This is a comparison diagram of the first embodiment of the present invention with the prior art; Figure 4b This is a second comparison result diagram between the present invention and the prior art; Figure 4c This is a comparison diagram of the third result of the present invention with the prior art. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] This embodiment provides a method for fusing AIS information and video data, such as Figure 1 As shown, it includes: S1. Collect AIS data and corresponding video data using the FVessel dataset; clean and filter out abnormal data in the AIS data to obtain preprocessed AIS data; S2. Construct a constant acceleration model, and apply the constant acceleration model to interpolate the preprocessed AIS data to obtain the ship target trajectory of the AIS data that is time-aligned with the video data. S3. Project the latitude and longitude coordinates of the preprocessed AIS data onto the pixel plane coordinate system of the video data to obtain the projection trajectory of the AIS data on the pixel plane. S4, using the YOLOX model to detect the ships in the video data to obtain the boundary box and class information of the ships; using the MotionTrack algorithm to track the detected ships and assigning a unique ID to the boundary box of the tracked ships, ensuring that the target ID is unchanged in the continuous frames, recording the position of the ships in the continuous frames to obtain the video data trajectory of the tracked ships; S5, extracting the ship motion features in the projection trajectory of the AIS data on the pixel plane and the ship motion features in the video data trajectory of the tracked ships respectively based on the projection trajectory of the AIS data on the pixel plane and the video data trajectory of the tracked ships; the ship motion features include ship heading features and ship speed features; S6, constructing a comprehensive similarity measurement function to calculate the ship motion feature similarity value between the projection trajectory of the AIS data on the pixel plane and the video data trajectory of the tracked ships; the comprehensive similarity measurement function fuses the Euclidean distance similarity, heading similarity and speed similarity of the ships in the AIS data and video data; S7, based on the ship motion feature similarity value, using the Hungarian matching algorithm to match the projection trajectory of the AIS data on the pixel plane and the video data trajectory of the tracked ships to obtain the matching result of the projection trajectory of the AIS data on the pixel plane and the video data trajectory of the tracked ships; S8, designing an outlier rejection and category consistency test to verify the matching result of the projection trajectory of the AIS data on the pixel plane and the video data trajectory of the tracked ships, and taking the matching result that passes the outlier rejection and category consistency test as the successful matching result; S9, drawing an image according to the matching result, fusing the AIS data and the boundary box of the tracked ships, and displaying the information of the detected ships in the video image to realize the supervision of water transportation.

[0018] Specifically, as shown in the figure, Figure 2 firstly, the AIS data and the video data are synchronously collected through the FVessel dataset, the abnormal data of the AIS data is cleaned and filtered, the signal noise and transmission error are removed to ensure the reliability of the data source; a constant acceleration model is constructed to interpolate and fill the timestamp gap of the preprocessed AIS data, generate a ship target trajectory strictly aligned with the video frame, and solve the time sequence mismatch problem in the fusion of heterogeneous asynchronous data; through coordinate projection transformation, the AIS latitude and longitude trajectory is mapped to the video pixel plane to establish the correspondence between the geographic coordinates and the image space, providing a spatial basis for subsequent cross-modal association; secondly, the YOLOX model is used to detect the video ship target to obtain high-precision boundary box and class label, and the MotionTrack algorithm is used to realize multi-target continuous tracking to generate ID consistent video trajectory sequence, ensuring the continuity of the target identity; Again, from the AIS projection track and video tracking track, the ship motion features of the AIS data and the video data are extracted respectively to depict the consistency of the ship motion mode; a comprehensive similarity measurement function is constructed to fuse the three constraints of Euclidean distance (spatial consistency), heading difference (motion direction consistency) and speed ratio (motion state consistency) to quantify the correlation confidence of the AIS and video tracks; the Hungarian matching algorithm is used to solve the optimal matching to realize one-to-one mapping of multi-target cross-modal data; a double-checking mechanism is designed to eliminate low-confidence matches with matching similarity corresponding cost exceeding the threshold, and then the semantic conflict matches are excluded through category consistency test to improve the robustness of the fusion result; Finally, based on the matching result, the superposition visualization of the AIS attribute information and the video bounding box is realized to form a spatiotemporally unified ship matching picture.

[0019] Preferably, the expression of the constant acceleration model is: (1) In the formula, is the current time of AIS update; is the time of the last AIS update; is the time to be interpolated; is the coordinate to be interpolated; is the ship speed at time t; is the ship acceleration at time t; is the coordinate at time t.

[0020] Specifically, when the ship acceleration is zero, the constant acceleration model will automatically degenerate into the constant speed model, i.e., the ship moves at a constant speed; therefore, the constant acceleration model of the embodiment has more advantages than the constant speed model and has a wider range of application.

[0021] Preferably, the latitude and longitude in the AIS information are the coordinates of the World Geodetic System-1984 Coordinate System (WGS84), so in order to fuse the AIS and video information, it is necessary to convert the latitude and longitude of the AIS information from the WGS84 world coordinate system to the camera coordinate system; In the embodiment, the latitude and longitude coordinates of the pretreated AIS data are projected to the pixel plane coordinate system of the video data, and the method adopted is as follows: S31, convert the latitude and longitude of the pretreated AIS data to the camera coordinate system, and the expression is: (2) In the formula, is the three-dimensional coordinate of the AIS data in the world coordinate system; These are the three-dimensional coordinates of the AIS data in the camera coordinate system; It is the rotation matrix from the camera coordinate system to the world coordinate system; It is the translation vector from the camera coordinate system to the world coordinate system; S32. Based on the camera parameters, transform the three-dimensional coordinates of the AIS data in the camera coordinate system to two-dimensional coordinates in the pixel plane. The expression is: (3) In the formula, It is the pixel coordinate vector of the projection; , These are the pixel coordinates of the AIS data information; It is the camera's intrinsic parameter matrix; yes The depth axis coordinate value.

[0022] In a specific embodiment, after the MotionTrack algorithm finishes tracking the ship's bounding box, it simultaneously performs tracking filtering; if the position or area change between consecutive frames of the same ID target is greater than a set threshold, the ID target is deleted; otherwise, the ID target is retained.

[0023] In this embodiment, the ship motion features of the extracted AIS data projection trajectory on the pixel plane and the video data trajectory of the tracked ship, including the ship's heading in the AIS data image, the ship's pixel velocity in the AIS data image, the ship's heading in the tracked ship image, and the ship's pixel velocity, are used to calculate a comprehensive similarity based on the extracted motion features. The method for extracting ship motion features is prior art to those skilled in the art and will not be described in detail here.

[0024] Preferably, the expression for the comprehensive similarity measurement function is: (4) In the formula, It is the Euclidean distance similarity; It is the similarity of the course; It is speed similarity; , , It is the balance weight constant; The expression for calculating Euclidean distance similarity is as follows: (5) In the formula, It's the size of the sliding window. , yes The pixel coordinates of the visual target on the ship at any given time. , is the pixel coordinate of the AIS data at time A; The expression for calculating the heading similarity is: (6) wherein, is the moving direction of the AIS data in the image; is the moving direction of the visual target in the image; is a constant parameter; The expression for calculating the velocity similarity is: (7) wherein, is the pixel velocity of the AIS data; is the pixel velocity of the visual target; is the width of the bounding box of the visual target; is a scaling factor.

[0025] Specifically, the matching similarity of the projection trajectory of the AIS data on the pixel plane and the video data trajectory of the tracked ship is obtained by the Hungarian matching algorithm, and the corresponding cost value is obtained through the matching similarity, the total cost sum of the overall matching result is calculated, and the overall matching result corresponding to the minimum total cost sum is taken as the optimal matching result. When the Hungarian matching algorithm is implemented to match the minimum total cost, all the projection trajectories of the AIS data on the pixel plane and the video data trajectories of the tracked ship are matched, which can produce unreasonable matching results. In order to ensure the accuracy of the matching, the matching results are subjected to outlier rejection and category consistency test. The outlier rejection compares the cost value corresponding to the matching similarity of the matching result of the projection trajectory of the AIS data on the pixel plane and the video data trajectory of the tracked ship with the threshold value obtained by calculation; if the cost value is greater than the threshold value, the corresponding matching result is rejected; otherwise, the corresponding matching result is retained. The category consistency test performs consistency test on the category of the ship in the AIS data and the category of the ship in the video data of the tracked ship; if the categories of the ships are inconsistent, the matching result is deleted; otherwise, the matching result is retained. The matching result subjected to the outlier rejection and the category consistency test is taken as the matching success result.

[0026] Preferably, the threshold value The expression of the threshold value is: (8) wherein, , are the width and height of the video data, respectively; W d is a balance weight constant.

[0027] Preferably, the abnormal data of the AIS data in step S1 includes: data with format error of the water mobile service identifier, data with relatively large change in latitude and longitude in a short time, data with too long distance from the camera, and data behind the camera.

[0028] Specifically, the fused video data is used for water traffic safety monitoring, and real-time monitoring of ship navigation in the water area is realized; for example, suspicious ships (AIS signal is closed) are screened and tracked, and the efficiency of water traffic safety supervision is improved.

[0029] In specific embodiments, the fusion method of the present embodiment is tested by using the FVessel dataset, which mainly includes 26 videos and the corresponding AIS data of the videos collected in the Wuhan section of the Yangtze River; the test results are as shown in Figures 3a to 3j .

[0030] At the same time, the fusion method of the present embodiment is compared with the prior art for testing, and the comparison results are as shown in Figures 4a to 4b ; in Figure 4a and Figure 4b , the ship on the leftmost side of the image shows incorrect matching information in the prior art, while the information shown in the fusion method of the present embodiment is correct; in Figure 4c , after the ship on the leftmost side of the image appears, the fusion method of the present embodiment can quickly display the matching information.

[0031] The present application has the following beneficial effects: The present application constructs and applies a constant acceleration model to interpolate the preprocessed AIS data, and through the constant acceleration model, the acceleration motion of the ship in the AIS data update interval is fully considered; compared with the traditional constant speed model, the constant acceleration model can more truly reflect the actual motion state (such as acceleration, deceleration, turning, etc.) of the ship when the AIS is not updated; The present application not only depends on position information, but also fuses the motion characteristics (heading and speed) of the ship for matching; by extracting the heading and speed characteristics from the AIS projection track and the video detection and tracking track respectively, and calculating the comprehensive similarity, more abundant and robust matching clues than pure position or track distance are provided; this makes the system still be able to use continuous and stable motion characteristics for effective association in the complex scene of missing detection of the ship target in the video or the ship closing the AIS device, etc., greatly reducing the mismatch risk caused by relying on single position information; The present application directly filters out the false association caused by serious information loss, noise interference or too large feature difference by regarding the matching result with the cost corresponding to the matching similarity greater than the threshold as unreliable matching (outlier); meanwhile, the video detected ship class is forced to be consistent with the ship class recorded in the AIS data, which increases the constraint on the semantic level and effectively prevents the accidental similarity of the motion features but the inconsistency of the actual ship type; the dual verification mechanism of outlier deletion and class consistency verification constitutes a powerful error filter, which significantly improves the confidence and accuracy of the final matching result.

[0032] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for fusing AIS information and video data, characterized in that, include: S1. Collect AIS data and video data corresponding to the AIS data using the FVessel dataset; Abnormal data in AIS data is cleaned and filtered to obtain preprocessed AIS data. S2. Construct a constant acceleration model, and apply the constant acceleration model to interpolate the preprocessed AIS data to obtain the ship target trajectory of the AIS data that is time-aligned with the video data. S3. Project the latitude and longitude coordinates of the preprocessed AIS data onto the pixel plane coordinate system of the video data to obtain the projection trajectory of the AIS data on the pixel plane. S4. Use the YOLOX model to detect ships in the video data and obtain the bounding boxes and category information of the ships; use the MotionTrack algorithm to track the detected ships and assign a unique ID to the bounding box of the tracked ship. Ensure that the target ID of the tracked ship remains unchanged in consecutive frames, record the position of the ship in consecutive frames, and obtain the video data trajectory of the tracked ship. S5. Using the projection trajectory of AIS data on the pixel plane and the video data trajectory of the tracked ship, extract the ship motion features from the projection trajectory of AIS data on the pixel plane and the ship motion features from the video data trajectory of the tracked ship, respectively; the ship motion features include ship heading features and ship speed features. S6. Construct a comprehensive similarity measurement function to calculate the similarity value of ship motion features between the projection trajectory of AIS data on the pixel plane and the trajectory of video data tracking the ship; The comprehensive similarity metric function integrates the Euclidean distance similarity, heading similarity, and speed similarity of ships in AIS data and video data; S7. Based on the similarity value of ship motion features, the Hungarian matching algorithm is used to match the projection trajectory of AIS data on the pixel plane with the trajectory of video data tracking the ship, so as to obtain the matching result of the projection trajectory of AIS data on the pixel plane and the trajectory of video data tracking the ship. S8. Design outlier removal and category consistency test: verify the matching results of the projection trajectory of AIS data on the pixel plane and the video data trajectory of the tracked ship, and take the matching result that passes the outlier removal and category consistency test as the matching success result. S9. Draw an image based on the matching results, fuse the AIS data with the bounding box of the tracked vessel, and display the information of the detected vessel in the video image to achieve the supervision of water traffic.

2. The method for fusing AIS information and video data according to claim 1, characterized in that, The expression for the constant acceleration model is: （1） In the formula, This is the current AIS update time; This is the last time AIS was updated; The moment when interpolation is needed; These are the coordinates to be interpolated; yes Ship speed at any given time; yes Ship acceleration at any given moment; yes The coordinates of the moment.

3. The method for fusing AIS information and video data according to claim 1, characterized in that, The latitude and longitude coordinates of the preprocessed AIS data are projected onto the pixel plane coordinate system of the video data using the following method: S31. Transform the latitude and longitude coordinates of the preprocessed AIS data to the camera coordinate system. The expression is: （2） In the formula, These are the three-dimensional coordinates of AIS data in the world coordinate system; These are the three-dimensional coordinates of the AIS data in the camera coordinate system; It is the rotation matrix from the camera coordinate system to the world coordinate system; It is the translation vector from the camera coordinate system to the world coordinate system; S32. Based on the camera parameters, transform the three-dimensional coordinates of the AIS data in the camera coordinate system to the pixel plane coordinate system. The expression is: （3） In the formula, It is the pixel coordinate vector of the projection; , These are the pixel coordinates of the AIS data information; It is the camera's intrinsic parameter matrix; yes The depth axis coordinate value.

4. The method for fusing AIS information and video data according to claim 1, characterized in that, The expression for the comprehensive similarity measurement function is: （4） In the formula, It is the Euclidean distance similarity; It is the similarity of the course; It is speed similarity; , , It is the balance weight constant; The expression for calculating Euclidean distance similarity is as follows: （5） In the formula, It's the size of the sliding window. , yes The pixel coordinates of the visual target on the ship at any given time. , yes Pixel coordinates of AIS data at any given time; The expression for calculating heading similarity is: （6） In the formula, It represents the direction of motion of AIS data in the image; It is the direction of motion of the visual target in the image; It is a constant parameter; The expression for calculating velocity similarity is: （7） In the formula, It is the pixel speed of AIS data; It is the pixel speed of the visual target; It is the width of the bounding box of the visual target; It is the scaling factor.

5. The method for fusing AIS information and video data according to claim 1, characterized in that, The outlier removal is achieved by comparing a calculated threshold with the cost corresponding to the matching similarity between the projection trajectory of AIS data on the pixel plane and the trajectory of video data tracking the ship. If the cost value is greater than the threshold, the corresponding matching result is removed; otherwise, the corresponding matching result is retained. The category consistency check verifies the consistency between the vessel category in the AIS data and the vessel category in the video data of the tracked vessel. If the vessel categories do not match, the matching results will be deleted. Otherwise, retain the matching results; Matching results that pass outlier removal and category consistency checks are considered successful matches.

6. The method for fusing AIS information and video data according to claim 5, characterized in that, The expression for the threshold is: （8） In the formula, , These are the width and height of the video data, respectively; W d It is the balancing weight constant.

7. The method for fusing AIS information and video data according to claim 1, characterized in that, Abnormal data in AIS data in step S1 includes: data with incorrect waterborne mobile service identifier format, data with large changes in latitude and longitude within a short period of time, data that is too far away from the camera, and data that is behind the camera.

Images