Shipborne radar, AIS and video data association and fusion method and system

Abstract

The application provides a shipborne radar, AIS and video data association and fusion method and system, belongs to multi-source data fusion and ship auxiliary driving technology, and the method comprises the following steps: acquiring spatio-temporal alignment radar, AIS and video data containing target ship; constructing an AIS and radar candidate association cost matrix and solving, obtaining a matching target and a single-source target; establishing a data source state identifier including double-source effectiveness, single-source and source switching; fusing AIS and radar data when the double-source is effective, splicing the track with time decay when the source is switched, and obtaining a continuous fusion track; determining a relative real azimuth angle and a relative visual azimuth angle of the target according to the continuous fusion track and the video data respectively; constructing a comprehensive association cost matrix based on the azimuth angle deviation and the relative distance and solving, and obtaining an optimal matching result of the video and the fusion track. The application solves the track mutation problem when the data source is switched, and improves the accuracy and robustness of the cross-modal association in the dense scene.

Description

Technical Field

[0001] This invention relates to multi-source data fusion and ship assisted driving technology, specifically to a method and system for associating and fusing shipborne radar, AIS and video data. Background Technology

[0002] As the shipping industry moves towards intelligentization, the safe navigation and autonomous collision avoidance of ships in complex waters place extremely high demands on the comprehensive and accurate perception of the navigation environment around the vessel. In existing shipborne perception systems, AIS (Automatic Identification System), shipborne radar, and visual sensors are commonly used core equipment. AIS provides the ship's identification information and dynamic motion data; radar can acquire the precise relative position and distance of targets and has the ability to detect non-cooperative targets; visual sensors provide two-dimensional image information containing rich texture, color, and contour features, intuitively reflecting the navigation environment and the ship's external characteristics.

[0003] However, all of the aforementioned single sensors have significant limitations in practical applications. AIS has limitations in data update frequency and non-cooperative target detection; radar data updates are greatly affected by environmental noise and cannot obtain target identification information; visual sensors, lacking geospatial reference, struggle to directly obtain the target's precise latitude, longitude, speed, and identification attributes, and are susceptible to interference from environmental factors such as lighting changes, severe weather, and water surface reflection. Data acquired by a single sensor is insufficient to independently meet the needs of operators for accurate and comprehensive assessment of the on-site environment. Therefore, combining the physical location detection capabilities of AIS and radar with the intuitive image information provided by visual sensors can create significant complementary advantages. Through multi-source data fusion, not only can the data blind spots of a single sensor be compensated for, improving the system's overall perception of surrounding vessels, but the key identification and dynamic information of target vessels in the location perception data can also be overlaid and displayed on the corresponding target in the video surveillance screen. This fusion method can meet the needs of operators for visual confirmation of the on-site navigation environment, intuitively showcasing potential navigation risks, and providing more comprehensive situational support to assist ship operators in making safety decisions. Existing technologies include pairwise association and fusion methods for AIS and radar, AIS and video, and radar and video. However, fusing data from only two sensors still presents the following problems: First, multi-source data is prone to signal loss or source switching during actual navigation. Existing fusion algorithms lack a smooth transition processing mechanism when dealing with data source changes, leading to sudden changes and breaks in target tracks. Second, visual images and radar / AIS position data exhibit cross-modal heterogeneity. Traditional coordinate projection matching methods are extremely sensitive to changes in ship attitude, resulting in significant pixel position drift and alignment errors. Third, ships frequently meet and overlap in complex waters. Traditional methods rely solely on single azimuth features, which easily leads to directional ambiguity and target confusion, resulting in high mismatch rates and poor robustness in association and fusion results in dense scenes.

[0004] In summary, the core technical problems of existing multi-source data association technologies are: existing fusion algorithms cannot smoothly transition when multi-source data signals are jittery; the hull attitude changes cause large matching errors in heterogeneous cross-modal data; and the association fusion mismatch rate is high in dense scenarios. Summary of the Invention

[0005] In view of this, it is necessary to provide a method and system for associating and fusing shipborne radar, AIS and video data, in order to solve the technical problems in the existing technology, such as the inability to smoothly transition when multi-source data signals are jittery and the high mismatch rate of association and fusion in dense scenes.

[0006] To address the aforementioned technical problems, in a first aspect, the present invention provides a method for associating and fusing shipborne radar, AIS, and video data, used for multi-source matching of ship targets, including: Acquire spatiotemporally aligned radar data, AIS data, and video data including that of the target ship; Construct and solve a candidate association cost matrix that represents the possibility of association between the AIS data and the radar data to obtain the matching target and the single-source target; A data source status identifier is established for the matched target and the single-source target, including AIS-radar dual-source effective, AIS single-source, radar single-source, and source switching. When the AIS-radar dual-source effective state is reached, AIS data and radar data are fused. When the AIS single-source state or the radar single-source state is reached, the current single-source data is kept as the track output. When the source is switched, time-effect attenuation is used to stitch the track to obtain the AIS-radar continuous fused track. Based on the AIS-radar continuous fusion track, the relative true azimuth of the target is determined; based on the video data, the relative visual azimuth of the target is determined; based on the azimuth deviation between the relative true azimuth and the relative visual azimuth and the relative distance of the target relative to the ship, a comprehensive correlation cost matrix is ​​constructed and solved to obtain the multi-source target fusion result of optimal matching between the video target and the AIS-radar continuous fusion track.

[0007] In one possible implementation, the step of constructing and solving a candidate association cost matrix characterizing the association probability between the AIS data and the radar data to obtain the matching target and the single-source target includes: An AIS target set and a radar target set are constructed based on the AIS data and the radar data, respectively, and an AIS-radar target pair is constructed based on the AIS target set and the radar target set; The AIS-radar target pairs that simultaneously satisfy the preset spatial distance difference, preset speed difference, preset heading difference and preset relative bearing difference are selected to obtain candidate target association pairs; A candidate association cost matrix is ​​constructed based on the spatial distance, speed, heading, orientation, time extrapolation error, and data quality of the candidate target association pairs; The Hungarian algorithm is used to solve the global optimal allocation of the candidate association cost matrix, and the AIS-radar matched target, AIS single-source target and radar single-source target are obtained and output; the AIS-radar matched target output also includes the corresponding association confidence.

[0008] In one possible implementation, establishing data source status identifiers for the matched target and the single-source target, including AIS-radar dual-source active, AIS single-source, radar single-source, and source switching, further includes: Maintain the target state vector for all targets; the target state vector includes the target number at the current moment, the data source status, the time of the most recent dual-source fusion, the set of historical fusion tracks, and the correlation confidence.

[0009] In one possible implementation, the fusion of AIS data and radar data during the AIS-radar dual-source effective state includes: The AIS measurement error is determined based on the inherent observation noise of the global navigation and positioning system; the radar measurement error is determined based on the inherent measurement error of the radar and the distance error between the ship and the target. Based on the AIS measurement error and the radar measurement error, the AIS dynamic fusion weights and radar dynamic fusion weights are determined using the variance minimization principle. The current AIS-radar dual-source observation fusion point is obtained by weighted summation of the original AIS trajectory point, the AIS dynamic fusion weight corresponding to the original AIS trajectory point, the radar original trajectory point, and the radar dynamic fusion weight corresponding to the original radar trajectory point; and the AIS-radar fusion track confidence is generated based on the correlation confidence, the AIS measurement error, and the radar measurement error.

[0010] In one possible implementation, the method for determining the source switching state is as follows: The current source state is determined based on the correlation result between the current AIS data and radar data; when the current source state changes relative to the previous source state, the target is determined to have entered a source switching state; the source switching state includes switching from an AIS-radar dual-source effective state to an AIS single-source state or a radar single-source state, and switching from an AIS single-source state or a radar single-source state to an AIS-radar dual-source effective state; the use of time-attenuation for track stitching during the source switching state includes: The historical predicted track points for the current moment are obtained by using linear extrapolation on the first two AIS-radar continuous fusion track points at the current moment; For the source switching state where the AIS-radar dual-source effective state switches to AIS single-source state or radar single-source state: the current AIS observation point or the current radar observation point is used as the current single-source observation track point. A time-effect attenuation weight function related to the source switching duration is constructed. The historical predicted track point, the time-effect attenuation weight function, the current single-source observation track point and its corresponding single-source data weight are weighted and summed to obtain the stitched track point at the current moment. For the source switching state where the AIS single-source state or radar single-source state switches to the AIS-radar dual-source effective state: first, the current AIS observation point and the current radar observation point are adaptively weighted based on distance to obtain the current AIS-radar dual-source observation fusion point. Then, the historical predicted track points, the time-effect attenuation weight function, the current AIS-radar dual-source observation fusion point and its corresponding dual-source fusion weight are weighted and summed to obtain the stitched track point at the current moment. When the duration of the source switching state reaches a preset source switching time threshold, or when the time-attenuation weight is lower than a preset weight threshold, the source switching state is exited, and the corresponding AIS single-source state, radar single-source state, or AIS-radar dual-source effective state is entered.

[0011] In one possible implementation, the AIS-radar continuous fusion track includes at least: target number, target latitude and longitude, stitched track points, current data source status, and the confidence level of the AIS-radar fusion track.

[0012] In one possible implementation, determining the target's relative true azimuth angle based on the AIS-radar continuous fusion track includes: Using a standard spherical geometry model, the true azimuth of the target relative to the ship is determined based on the current latitude and longitude of the ship and the target. The relative true azimuth of the target from the ship's perspective is obtained by subtracting the true azimuth from the ship's bow angle.

[0013] In one possible implementation, the video data is acquired via a visual sensor, which performs installation parameter calibration relative to the ship's coordinate system. The bow reference line in the image is corrected based on the installation offset angle and / or lateral installation offset of the optical principal axis relative to the bow direction obtained from the calibration. The geometric centerline of the video data is equivalent to the ship's heading reference line. Determining the relative visual azimuth angle of the target based on the video data includes: Target detection is performed on the target ship in the video data to obtain a target detection box; the horizontal coordinate of the midpoint of the bottom edge of the target detection box is used as the horizontal reference anchor point of the target in the picture; the pixel distance of the target anchor point from the central axis is determined based on the difference between the horizontal reference anchor point and the horizontal coordinate of the geometric central axis of the picture. Based on the equivalent physical focal length of the visual sensor and the pixel distance of the target anchor point from the central axis, the two-dimensional pixel features of the target are mapped into a three-dimensional relative visual azimuth angle.

[0014] In one possible implementation, the step of constructing and solving a comprehensive correlation cost matrix based on the azimuth deviation between the relative true azimuth and the relative visual azimuth and the relative distance of the target relative to the ship, to obtain the multi-source target fusion result of optimal matching between the video target and the AIS-radar continuous fusion track, includes: The azimuth deviation between the relative true azimuth and the relative visual azimuth is determined based on the minimum circumferential angular distance. The actual distance between the target and the ship is determined by latitude and longitude. The ratio of the actual distance between the target and the ship to the preset maximum observation distance is used as the relative true distance coefficient between the target and the ship. The relative visual distance coefficient between the target and the ship is determined by the ordinate of the midpoint of the bottom edge of the video target detection box and the pixel height of the image. The distance feature deviation is constructed based on the relative true distance coefficient and the relative visual distance coefficient. The azimuth deviation and the distance feature deviation are weighted and fused to obtain a comprehensive correlation cost matrix; elements in the comprehensive correlation cost matrix that exceed a preset range are assigned to infinity to obtain an updated comprehensive correlation cost matrix. The Hungarian algorithm is used to solve the updated integrated correlation cost matrix for global optimal allocation, and the target matching result of the optimal matching between the video target and the AIS-radar continuous fusion track is obtained. Based on the target matching result, the information in the AIS-radar continuous fusion track is superimposed onto the video target to obtain the multi-source target fusion result.

[0015] Secondly, the present invention also provides a shipborne radar, AIS, and video data association and fusion system, comprising: The AIS receiver is used to receive AIS data broadcast by targets around the ship. Shipborne radar, used to detect radar data of targets around the ship; Video sensors are used to collect video data directly in front of the ship; A compass is used to provide the ship's real-time heading angle; GNSS receiver, used to provide the ship's latitude and longitude, speed above ground and heading above ground; The memory is used to store the program and the AIS data, the radar data, the video data, the real-time bow angle, the ship's latitude and longitude, ground speed and ground heading; The processor is communicatively connected to the AIS receiver, the shipborne radar, and the video sensor, and coupled to the memory, respectively, for executing the program stored in the memory to implement the steps in the shipborne radar, AIS and video data association and fusion method described in any of the above implementations; A display, coupled to the processor, is used to output the multi-source target fusion result of superimposing information from the AIS-radar continuous fusion track onto the video target.

[0016] The beneficial effects of this invention are as follows: The shipborne radar, AIS, and video data association and fusion method provided by this invention firstly eliminates the differences in spatial description benchmarks and sampling times between different sensors by acquiring spatiotemporally aligned radar data, AIS data, and video data including the target ship, thus laying a unified spatiotemporal foundation for subsequent multi-source data association. Secondly, by constructing and solving a candidate association cost matrix characterizing the association possibility of AIS data and radar data, matching targets and single-source targets are obtained, effectively avoiding the mismatch problem that easily occurs in local nearest neighbor matching in dense scenes, and significantly improving the association accuracy between heterogeneous sensors. Next, data source status identifiers including AIS-radar dual-source effective, AIS single-source, radar single-source, and source switching are established for matching targets and single-source targets. In the AIS-radar dual-source effective state, AIS data and radar data are fused; in the AIS single-source state or radar single-source state... The system maintains the current single-source data as the output track. During source switching, time-dependent attenuation is used for track stitching to obtain a continuous fused track from AIS and radar. This enables soft switching when the data source is lost or recovered, effectively suppressing abrupt changes and breaks in the target track's position, and ensuring the smoothness and continuity of the output track. Finally, the relative true azimuth of the target is determined based on the continuous fused track from AIS and radar, while the relative visual azimuth of the target is determined based on video data. A comprehensive correlation cost matrix is ​​constructed based on azimuth deviation and relative distance for global optimization. This introduces dual spatial features into cross-modal matching, overcoming the directional ambiguity problem of traditional single features in ship encounter and overlap scenarios. This significantly improves the accuracy and robustness of the optimal correspondence between video targets and fused tracks. Finally, position, identity, and other information are superimposed onto the video target, providing intuitive, comprehensive, and reliable situational awareness support for ship intelligent perception and collision avoidance decision-making. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic flowchart of an embodiment of the shipborne radar, AIS and video data association and fusion method provided by the present invention; Figure 2 Provided by the present invention Figure 1 A schematic diagram of an embodiment of S102; Figure 3 A schematic diagram of an embodiment of the fusion rule provided by the present invention; Figure 4This is a schematic flowchart of an embodiment of the present invention for fusing AIS data and radar data when AIS-radar dual-source is effective; Figure 5 This is a schematic diagram of an embodiment of the present invention that uses time-effect attenuation for track stitching during source switching; Figure 6 A schematic diagram illustrating an embodiment of the present invention for determining the relative true azimuth of a target based on AIS-radar continuous fusion track; Figure 7 A schematic diagram illustrating an embodiment of the present invention for determining the relative visual azimuth of a target based on video data; Figure 8 This is a schematic diagram of an embodiment of the present invention for obtaining the optimal matching multi-source target fusion result by comprehensively considering the correlation cost matrix; Figure 9 A schematic diagram illustrating an embodiment of the visualization effect of wide water area association and fusion provided by the present invention; Figure 10 A schematic diagram illustrating an embodiment of the visualization effect of narrow waterway association fusion provided by the present invention; Figure 11 A schematic diagram illustrating an embodiment of the port / anchorage association and fusion visualization effect provided by the present invention; Figure 12 This is a schematic diagram of an embodiment of the shipborne radar, AIS and video data association and fusion method provided by the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0020] In the description of the embodiments of the present invention, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone.

[0021] The terms "first," "second," etc., used in the embodiments of this invention are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature defined with "first" or "second" may explicitly or implicitly include at least one of that feature.

[0022] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0023] Before demonstrating the embodiments, the following terms will be explained.

[0024] AIS (Automatic Identification System) is a maritime safety and communication aid system applied between ships and shore, and between ships. It consists of a VHF communicator, a GPS locator, and a communication controller, and achieves automatic exchange of dynamic and static information of ships by connecting with shipboard displays and sensors.

[0025] MMSI (Maritime Mobile Service Identity): A nine-digit code transmitted by a ship's radio communication system on its radio channels, which can uniquely identify various types of stations and groups of calling stations.

[0026] This invention provides a method and system for associating and fusing shipborne radar, AIS, and video data, which will be described below.

[0027] Figure 1 The shipborne radar, AIS, and video data association and fusion method provided by this invention is used for multi-source matching of ship targets, such as... Figure 1 As shown, it includes: S101. Acquire spatiotemporally aligned radar data, AIS data, and video data including the target ship; S102. Construct and solve a candidate association cost matrix that represents the possibility of association between AIS data and radar data to obtain the matching target and the single-source target. S103. Establish data source status identifiers for matching targets and single-source targets, including AIS-radar dual-source effective, AIS single-source, radar single-source, and source switching; when AIS-radar dual-source effective, fuse AIS data and radar data; when AIS single-source or radar single-source, keep the current single-source data as the track output; when the source is switched, use time-effect attenuation to stitch the track to obtain a continuous fused AIS-radar track. S104. Based on the AIS-radar continuous fusion track, determine the target's relative true azimuth angle; based on the video data, determine the target's relative visual azimuth angle; based on the azimuth angle deviation between the relative true azimuth angle and the relative visual azimuth angle and the target's relative distance to the ship, construct a comprehensive correlation cost matrix and solve it to obtain the multi-source target fusion result of the optimal matching between the video target and the AIS-radar continuous fusion track.

[0028] It should be noted that in this invention, "target" refers to the target vessel, that is, other vessels within the observation range of this vessel.

[0029] In some embodiments of the present invention, the method for associating and fusing shipborne radar, AIS, and video data provided by the present invention includes the following steps: Step 1: Multi-source data preprocessing and spatiotemporal synchronization: First, the multi-sensor perception data is cleaned and optimized, including the removal of abnormal and redundant data, AIS coordinate center compensation, and processing of short-term radar target loss. Second, the images acquired by the visual sensor are geometrically distorted, and the bounding boxes of ship targets are extracted in real time using a target detection model. Finally, multi-sensor spatiotemporal synchronization is performed. Spatially, the radar polar coordinates are uniformly mapped to the World Geodetic System 1984 (WGS-84). Temporally, a kinematic model is introduced to smoothly extrapolate the asynchronous data of each sensor to a time reference consistent with the video frame, thereby achieving spatiotemporal alignment of multi-source data.

[0030] Step 2, Candidate Association of AIS Targets and Radar Targets: First, threshold screening is performed on the target set after spatiotemporal synchronization to eliminate invalid candidate target pairs that do not conform to physical characteristics; second, a candidate association cost matrix reflecting the association possibility is constructed by comprehensively considering spatial distance, motion characteristics and data validity quality factors; finally, the Hungarian algorithm is used to solve the global optimal allocation problem, distinguishing between matching targets and single-source targets, and the corresponding association confidence is calculated simultaneously.

[0031] Step 3: Adaptive Weighted Fusion and Track Stitching: First, establish a data source status identifier for the target and formulate multi-source fusion rules covering four typical cases (dual-source effective, single-source independent, and source switching); second, when dual-source data is complete, establish a dynamic variance model related to the target distance and execute adaptive weighted fusion based on the principle of minimizing variance; finally, when facing changes in observation sources, introduce a track stitching algorithm based on time decay to achieve soft data switching, ensuring the smoothness and continuity of the output fused track.

[0032] Step 4: Multi-source perception data association and fusion based on azimuth and distance matching: First, a cross-modal spatial mapping model is established to calculate the target's true azimuth in physical space and its visual relative azimuth on the two-dimensional image plane. Second, the azimuth deviation and relative distance features between the two are combined to construct a weighted association cost matrix that reflects the comprehensive positional deviation of the two-dimensional plane. Finally, an association threshold is introduced to eliminate abnormal combinations with excessively large spatial feature differences, and the Hungarian algorithm is used to minimize the global cost. Ultimately, the optimal correspondence between the fused track target and the visually detected target is established.

[0033] The shipborne radar, AIS, and video data association and fusion method provided in this invention first eliminates the differences in spatial description benchmarks and sampling times between different sensors by acquiring spatiotemporally aligned radar data, AIS data, and video data including the target ship, thus laying a unified spatiotemporal foundation for subsequent multi-source data association. Second, by constructing and solving a candidate association cost matrix characterizing the association probability of AIS data and radar data, matching targets and single-source targets are obtained, effectively avoiding the mismatch problem that easily occurs in local nearest neighbor matching in dense scenes, and significantly improving the association accuracy between heterogeneous sensors. Next, data source status identifiers are established for matching targets and single-source targets, including AIS-radar dual-source effective, AIS single-source, radar single-source, and source switching. When the AIS-radar dual-source effective state is reached, AIS data and radar data are fused; when the AIS single-source state or radar single-source state is reached, the current... The initial single-source data is used as the track output. During source switching, time-dependent attenuation is applied to stitch the track together to obtain the AIS-radar continuous fused track. This enables soft switching when the data source is lost or recovered, effectively suppressing abrupt changes and breaks in the target track's position, and ensuring the smoothness and continuity of the output track. Finally, the relative true azimuth of the target is determined based on the AIS-radar continuous fused track, while the relative visual azimuth of the target is determined based on the video data. A comprehensive correlation cost matrix is ​​constructed based on the azimuth deviation and relative distance for global optimization. This introduces dual spatial features into cross-modal matching, overcoming the directional ambiguity problem of traditional single features in ship encounter and overlap scenarios. This significantly improves the accuracy and robustness of the optimal correspondence between the video target and the fused track. Finally, position, identity, and other information are superimposed onto the video target, providing intuitive, comprehensive, and reliable situational awareness support for ship intelligent perception and collision avoidance decision-making.

[0034] It should be noted that after obtaining the raw multi-source data, the data needs to be preprocessed.

[0035] In some embodiments of the present invention, in step 1, multi-sensor data preprocessing is performed to address the heterogeneous and asynchronous nature of shipborne multi-source sensing data and the requirements for visual image distortion and target extraction, in order to establish a unified data benchmark. First, the multi-sensor data is optimized by cleaning up abnormal and redundant data through multi-level verification, compensating the AIS coordinates to the ship's geometric center using ship dimensional parameters, and addressing the short-term loss of radar targets based on a motion continuity mechanism. Second, a calibration method is used to correct geometric distortion in the images extracted by the visual sensors, and the ship's bounding box is extracted simultaneously using a target detection model. Finally, multi-sensor spatiotemporal synchronization is performed, uniformly mapping the radar polar coordinates to the WGS-84 geographic coordinate system in the spatial dimension, and introducing a kinematic model in the temporal dimension to smoothly extrapolate the asynchronous observation data from each sensor to a system global benchmark time consistent with the video frame timestamp. Specifically, the following steps are included: Step 1-1 Multi-sensor data optimization: First, multi-level data cleaning is performed to remove invalid and erroneous information. For the parsed sensor data, this invention adopts a tiered processing strategy based on the importance of each field: First, regarding data integrity verification, if core fields such as AIS MMSI, radar target number, or positioning coordinates are missing, the record is directly removed; for missing non-core static fields, an attempt is made to retrieve and complete them from the historical database or mark them as missing. Second, regarding numerical rationality screening, location coordinates exceeding the experimental water area, as well as outlier data that violates physical logic, such as headings exceeding the 0° to 360° range or negative speeds, are removed. Third, regarding outlier removal, amplitude-limiting filtering is introduced to verify target motion; if adjacent sampling points are displaced... Exceeding the physical limits set based on the ship's maximum speed (i.e. > If a certain condition is detected, it is identified as an abnormal transition point, and linear prediction compensation is performed using the state of the previous moment. In addition, the system uses a time window mechanism to merge duplicate and redundant messages at the same moment to reduce the computational load.

[0036] Secondly, ship center position correction is performed to eliminate spatial errors caused by antenna installation offset. Since the latitude and longitude provided in the AIS message represent the physical installation location of the antenna, while radar systems typically track the physical centroid of the ship target, direct correlation will produce significant systematic spatial deviations. Based on the ship dimensional parameters defined in the standard protocol, the longitudinal and lateral relative offsets of the ship's geometric center relative to the antenna position can be calculated. and As shown in Equation 1: (1) in, The distance from the antenna to the bow of the ship. Distance to the stern Distance to port side This refers to the distance to the starboard side.

[0037] Based on this, combined with the ship's real-time true bow angle and the Earth's average radius The aforementioned offset is compensated to the geographic coordinate system using a rotation matrix to obtain the corrected coordinates of the ship's geometric center. As shown in Equation 2: (2) in, These represent the latitude and longitude of the antenna installation location. This processing successfully and accurately restored the AIS target location to the ship's geometric center, laying the foundation for subsequent spatial alignment of multi-source data.

[0038] Finally, radar target status discrimination and short-term loss handling are performed based on a motion continuity mechanism. Shipborne radar target status is categorized into three types: query (Q), track (T), and lost (L). The system only extracts targets under stable tracking (T) status for the fusion process. To prevent track breakage, when a target changes from T to L status due to environmental obstruction or strong sea clutter interference, a target identity (ID) preservation mechanism is activated. By comparing historical trajectories, if within a preset time window, the Euclidean distance between a newly acquired query (Q) target and the previously lost (L) target is less than the maximum displacement threshold, and their speed and heading deviations are within allowable ranges, they are determined to be the same physical target. This mechanism effectively solves the problem of short-term radar target loss and maintains the uniqueness and continuity of the perceived target number.

[0039] Steps 1-2: Visual Image Correction and Object Detection First, a mathematical model of geometric distortion in visual images is established. The geometric distortion of a camera mainly manifests as radial distortion caused by lens shape defects and tangential distortion caused by the lens not being parallel to the photosensitive plane. Let... The coordinates are the normalized image coordinates under ideal distortion-free conditions; the coordinates that actually produce distortion are... ,and The mathematical model of radial distortion is approximated by Taylor series expansion as shown in Equation 3: (3) The tangential distortion model is represented as shown in Equation 4: (4) In the formula, , , Radial distortion coefficient; , denoted as the tangential distortion coefficient.

[0040] Secondly, parameters are solved based on the calibration algorithm, and image correction is performed. Zhang Zhengyou's checkerboard calibration method is used to calibrate the camera parameters. By extracting the coordinates of the interior corner points of the standard checkerboard calibration board from multiple perspectives, a mapping relationship between the two-dimensional image and the three-dimensional spatial points is established, and the camera's intrinsic parameter matrix and the aforementioned distortion coefficients are calculated. The obtained parameters are used to perform reverse mapping and interpolation compensation on the real-time acquired raw video frames to eliminate the influence of nonlinear distortion and obtain a corrected image that conforms to the ideal perspective projection law, providing a reliable geometric plane for subsequent azimuth mapping.

[0041] Finally, a lightweight model is deployed on the corrected image to extract ship targets. The set of target detection boxes in the current video frame is shown in Equation 5: (5) in, The set of visual objects in the current video frame. The number of video targets.

[0042] For any video target Its visual features include at least those shown in Equation 6: (6) in, The coordinates of the center of the target detection box; The width of the detection frame; The height of the detection frame; Confidence level for the target category.

[0043] Steps 1-3: Multi-sensor spatiotemporal synchronization: First, spatial coordinate alignment is performed to unify the radar's local polar coordinates to the WGS-84 geographic coordinate system. The target position measured by the shipborne radar is set with the ship as the pole, and the primary output is the distance from the target to the ship. With azimuth (Based on true north). Combined with the ship's current absolute latitude and longitude ( , and Earth's radius Introducing angular distance The latitude and longitude of the target ship in the WGS-84 coordinate system were calculated using the spherical geometric transformation formula. , As shown in Equation 7: (7) This mapping formula can eliminate the differences between radar and AIS data in terms of spatial description reference.

[0044] Secondly, time alignment of dynamic observation data is performed based on a kinematic model. To address the time asynchrony issue caused by inconsistent sampling frequencies of different sensors, the system employs an extrapolation prediction method. Let the system's global reference time be... The most recent effective observation time of a certain sensor is The sampling time difference is Using the target's ground speed at the time of observation. With ground heading , its observation location ( , Smoothly extrapolating the predicted coordinates to the reference time. , As shown in Equation 8: (8) Finally, a global time synchronization strategy based on video images was established. In multimodal systems incorporating visual sensors, image sequences are typically output at a constant frame rate. The system uses the timestamps of image frames output by the video acquisition module as the absolute visual reference benchmark. Utilizing the aforementioned extrapolation method and high-frequency pulse alignment mechanism, radar spot data and AIS message data are aligned to the corresponding video frame time nodes. This mechanism ensures that geospatial data and visual perception results remain strictly consistent on the time axis when subsequently associating each frame across modalities, eliminating sensor sampling time lag bias.

[0045] In some embodiments of the present invention, a candidate association cost matrix characterizing the correlation probability of AIS data and radar data is constructed and solved to obtain the matching target and the single-source target, such as... Figure 2 As shown, it includes: S201. Construct AIS target sets and radar target sets based on AIS data and radar data respectively, and construct AIS-radar target pairs based on AIS target sets and radar target sets; S202. Select AIS-radar target pairs that simultaneously meet the preset spatial distance difference, preset speed difference, preset heading difference and preset relative bearing difference to obtain candidate target association pairs; S203. Construct a candidate association cost matrix based on the spatial distance, velocity, heading, azimuth, time extrapolation error, and data quality of candidate target association pairs; S204. The Hungarian algorithm is used to solve the global optimal allocation of the candidate association cost matrix, and the AIS-radar matched target, AIS single-source target and radar single-source target are obtained and output. The AIS-radar matched target output also includes the corresponding association confidence.

[0046] This embodiment achieves precise one-to-one association between AIS and radar targets by fusing candidate association cost matrices based on spatial distance, motion characteristics, and data quality factors, and employing the Hungarian algorithm for global optimal allocation. In dense ship-centric scenarios, the Hungarian global optimization avoids the chain of mismatches that can easily occur with local nearest neighbor matching, significantly reducing the matching error rate. It clearly distinguishes between AIS-radar matched targets (dual-source), AIS single-source targets, and radar single-source targets, providing clear input for subsequent adaptive fusion and source state switching. Simultaneously, it outputs association confidence, enabling the system to quantitatively evaluate the credibility of each matching pair, providing a basis for dynamic fusion weight adjustment and track confidence generation.

[0047] In some embodiments of the present invention, a data source status identifier is established for matching targets and single-source targets, including AIS-radar dual-source active, AIS single-source, radar single-source, and source switching, and further includes: Maintain the target state vector for all targets; the target state vector includes the target number at the current moment, the data source status, the time of the most recent dual-source fusion, the set of historical fusion tracks, and the correlation confidence.

[0048] This embodiment achieves continuous tracking and management of multi-source sensing targets by maintaining a target state vector for each target, including target number, data source status, most recent dual-source fusion time, historical fusion track set, and correlation confidence. Firstly, it provides complete historical information support for subsequent adaptive fusion and track stitching, enabling the system to accurately determine the current data source status and switching timing. Secondly, by recording the most recent dual-source fusion time and historical fusion tracks, the time-degradation stitching algorithm can obtain reliable prior prediction points, thus achieving a smooth transition during source switching and avoiding track abrupt changes. Thirdly, incorporating correlation confidence into the state vector provides a quantitative basis for dynamic fusion weight adjustment and fusion track credibility assessment.

[0049] In some embodiments of the present invention, in step 2, target candidate association and matching calculations are performed on the spatiotemporally synchronized AIS and radar multi-source sensing data to establish the observation source status of different sensors and their association confidence. First, a set of AIS and radar targets under a unified spatiotemporal reference is constructed, and candidate association thresholds are used for screening based on spatial distance, speed, heading, and relative azimuth difference to eliminate invalid target pairs that clearly do not belong to the same ship. Second, a candidate association cost matrix for AIS and radar targets is constructed by comprehensively considering the spatial and motion characteristic deviations between targets, time extrapolation errors, and data validity quality factors. Finally, the candidate association is modeled as a one-to-one global optimal allocation problem, solved using the Hungarian algorithm, and AIS-radar matched targets, AIS single-source targets, and radar single-source targets are divided based on the association confirmation threshold, and the corresponding association confidence is calculated synchronously. Specifically, the following steps are included: Step 2-1: Construction of AIS and Radar Target Sets: At the unified reference time corresponding to the current video frame Based on the spatiotemporal synchronization results of step 1, the AIS target set and radar target set are constructed as shown in Equations 9 and 10 respectively: (9) (10) in, This represents the current set of AIS targets. This represents the set of radar targets at the current moment. For the target number of AIS, This represents the number of radar targets.

[0050] For any AIS target Its status characteristics include at least the ship's MMSI, latitude and longitude position, speed over land, heading over land, and data timestamp.

[0051] For any radar target Its status characteristics include at least radar target number, latitude and longitude position, relative distance, relative azimuth angle, target speed, target heading, tracking status, and data timestamp.

[0052] By constructing the above target set, AIS targets and radar targets are unified under the same time reference and the same WGS-84 spatial coordinate system, enabling the two types of targets to be judged as candidates.

[0053] Step 2-2 Candidate Association Threshold Filtering: To avoid invalid calculations for AIS and radar targets that are clearly unlikely to belong to the same physical vessel, we first consider the AIS targets... With radar target Perform threshold filtering.

[0054] For any pair of targets The spatial distance difference, speed difference, heading difference, and relative bearing difference are calculated and expressed as shown in Equations 11 to 14: (11) (12) (13) (14) in, For AIS target The geographical location after compensation by the center For radar target Geographical location after conversion to the WGS-84 coordinate system; and The speeds of AIS targets and radar targets are respectively; and The headings are for AIS targets and radar targets, respectively. and These are the azimuth angles of the AIS target and the radar target relative to the ship, respectively.

[0055] A target pair is considered a candidate association pair if it satisfies the following conditions as shown in Equations 15 to 17: (15) (16) (17) in, As a spatial distance threshold, For speed threshold, The heading threshold. Target pairs that do not meet the above conditions are directly determined as non-candidate association pairs and will not proceed to the subsequent global matching process. Spatial distance threshold. It can be dynamically adjusted according to the target distance, the ship's speed, the sensor sampling interval, and the radar measurement error to avoid false screening caused by a fixed threshold in close-range or long-range scenarios.

[0056] Step 2-3: Construction of candidate association cost matrix: For candidate target pairs that pass the threshold screening The candidate correlation feature vector is constructed as shown in Equation 18: (18) in, This is the normalized value of the spatial distance difference. This is the normalized value of the speed difference. This is the normalized value of the heading difference. This is the normalized value of the relative azimuth difference. This is the normalized value for the time extrapolation error between AIS data and radar data. This is the AIS data validity quality factor. This is the radar data validity quality factor.

[0057] Based on the above candidate association feature vectors, the association cost between AIS targets and radar targets is constructed as shown in Equation 19: (19) in, For AIS target With radar target The associated costs; These are the weighting coefficients for spatial distance, speed, heading, bearing, time extrapolation error, and data quality, respectively.

[0058] The AIS-radar candidate association cost matrix, composed of the association costs of all candidate target pairs, is shown in Equation 20: (20) For target pairs that fail the threshold screening, their corresponding cost is set to infinity so that they do not participate in subsequent global optimal allocation.

[0059] Steps 2-4: One-to-one global candidate matching: Based on the candidate association cost matrix obtained in steps 2-3 The candidate association between AIS targets and radar targets is modeled as a global optimal allocation problem under one-to-one constraints.

[0060] The candidate matching matrix is ​​defined as shown in Equation 21: (twenty one) Among them, when When, it indicates the AIS target With radar target Determined as a candidate matching target; when When, it indicates that the two do not match.

[0061] The global optimization objective is as shown in Equation 22: (twenty two) The constraints are shown in Equations 23 and 24: (twenty three) (twenty four) The above global minimum cost matching problem is solved by using the Hungarian algorithm to obtain candidate matching results between AIS targets and radar targets.

[0062] To avoid forcing matching of weakly related targets, this invention further sets an association confirmation threshold. When the target pair satisfies the conditions shown in Equation 25: (25) And this objective satisfies the following in the global allocation result. If the AIS target and the radar target are confirmed to be the same physical ship target, then the relationship between the two is not confirmed.

[0063] Steps 2-5: Calculate the association confidence score and output the source state. For a confirmed matching AIS-radar target pair, its association confidence is calculated as shown in Equation 26: (26) in, Used to characterize AIS targets With radar target The degree of confidence in a connection pointing to the same physical ship target. The lower the association cost, the higher the association confidence.

[0064] Based on the matching results of steps 2-4, the targets at the current moment are divided into the following three categories: The first category is the AIS-radar matched target set, as shown in Equation 27: (27) This type of target possesses both AIS identity information and radar observation information, and its initial source state is marked as a dual-source effective state. .

[0065] The second category is the AIS single-source target set, as shown in Equation 28: (28) This type of target only has AIS observation information, and its initial source state is labeled as an AIS single-source state. .

[0066] The third category is a set of radar single-source targets, as shown in Equation 29: (29) This type of target only has radar observation information, and its initial source state is marked as a single radar source state. .

[0067] The aforementioned AIS-radar matched target set, AIS single-source target set, radar single-source target set, and corresponding associated confidence scores are used as inputs for step 3, which are then used for subsequent adaptive fusion based on source state, track stitching, and source switching judgment.

[0068] In some embodiments of the present invention, in step 3, to address the problem of incomplete data sources caused by changes in equipment status or detection range during actual navigation, adaptive fusion and track stitching based on source status are performed to ensure the smoothness and continuity of the target track. First, a data source status identifier is established for each perceived target, and fusion rules and state transition mechanisms covering four typical cases—dual-source effective, single-source independent, and source switching—are formulated. Second, in the dual-source effective stage where AIS and radar data are complete, a dynamic error variance model related to the relative physical distance to the target is established, and adaptive weighted fusion based on the principle of variance minimization is performed to balance the short-range accuracy of radar and the long-range positioning capability of AIS. Finally, when facing source switching states due to changes in the observed source, a track stitching and fusion algorithm based on time decay is introduced. A weight function is dynamically used to achieve soft switching between historical prior prediction information and the current observed track point, ultimately outputting a continuous fused track set containing state variables and confidence levels. Specifically, the steps include the following: 3-1 Source State Establishment and State Transition: For each target output in step 2, establish a corresponding data source state identifier (SourceState) and maintain state information such as target number, historical fusion trajectory, most recent dual-source fusion time, current valid observation source, and fusion confidence. The fusion rules are as follows: Figure 3 As shown. Among them, If the current observation source is different from the previous one, then the system enters the source switching state.

[0069] SourceState includes at least the following four types of states: (1) AIS-radar dual-source effective status: This indicates that both the AIS data and the radar data are valid, and that the association between the two has been successfully confirmed through step 2.

[0070] (2) AIS single-source status: This indicates that only AIS observation data exists, while the corresponding radar target does not exist, was missed by the radar, or was not confirmed through association.

[0071] (3) Radar single-source status: This indicates that only radar observation data exists, while the corresponding AIS target does not exist, the AIS equipment is turned off, or the AIS data has not been confirmed through association.

[0072] (4) Source Switching State: This refers to the transitional fusion state that the target enters after the data source state changes at adjacent time points. This includes switching from AIS-radar dual-source effective state to AIS single-source state or radar single-source state, and recovering from AIS single-source state or radar single-source state to AIS-radar dual-source effective state. The source switching state is used to trigger time-attenuation weighted fusion between historical predicted track points and current observed track points. When the duration of the source switching state reaches the preset source switching time threshold, or the time-attenuation weight is lower than the preset weight threshold, the target exits the source switching state and enters the corresponding stable source state.

[0073] For any target The target state vector is maintained as shown in Equation 30: (30) in, Number the target; This represents the current source state. This refers to the most recent dual-source fusion time; A collection of historical navigation routes; This represents the current fusion confidence level.

[0074] When step 2 outputs the AIS-radar matched target, it is as shown in Equation 31: (31) The target then enters the dual-source effective state as shown in Equation 32: (32) When the target is only observed by AIS, it is as shown in Equation 33: (33) The corresponding target then enters the AIS single-source state as shown in Equation 34: (34) When the target is only observable by radar, it is as shown in Equation 35: (35) The target then enters the radar single-source state as shown in Equation 36: (36) When the target changes from a dual-source state to a single-source state, or recovers from a single-source state to a dual-source state, it is as shown in Equation 37: (37) In the source switching state, the system does not directly replace the target track, but instead initiates a track stitching mechanism based on time decay to achieve a smooth transition between different observation sources.

[0075] In some embodiments of the present invention, AIS data and radar data are fused when the AIS-radar dual-source is active, such as... Figure 4 As shown, it includes: S401. Determine AIS measurement error based on the inherent observation noise of the global navigation and positioning system; determine radar measurement error based on the inherent measurement error of the radar and the distance error between the ship and the target; S402. Based on the AIS measurement error and the radar measurement error, the AIS dynamic fusion weight and the radar dynamic fusion weight are determined using the principle of minimizing variance. S403. The current AIS-radar dual-source observation fusion point is obtained by weighted summation based on the original AIS trajectory point, the AIS dynamic fusion weight corresponding to the original AIS trajectory point, the radar original trajectory point, and the radar dynamic fusion weight corresponding to the original radar trajectory point; the AIS-radar fusion track confidence is generated based on the correlation confidence, AIS measurement error, and radar measurement error.

[0076] In some embodiments of the present invention, step 3 further includes: 3-2 Distance-based adaptive weighted fusion: When the target is in AIS-radar dual-source effective state At the same time, it has both AIS observation data and radar observation data.

[0077] First, the center position of the AIS data is corrected using ship dimensional parameters to eliminate errors caused by antenna installation offset. The positional error of AIS data mainly stems from the inherent observation noise of the Global Navigation Satellite System (GNSS), therefore the variance of the equivalent measurement error of AIS is... The calculation formula is shown in Equation 38: (38) in, This represents the inherent reference error variance of the global navigation and positioning system.

[0078] Secondly, radar data is corrected using target distance and equivalent measurement error. Limited by the physical beamwidth of the radar antenna, beam divergence causes the radar's lateral resolution to decrease with increasing detection range, leading to a quadratic amplification of the position observation error variance. Therefore, the radar's equivalent measurement error variance... It is modeled as a function of spatial distance, and the calculation formula is shown in Equation 39: (39) in, This represents the inherent, minimum measurement error variance of the radar. For distance-related error terms, This represents the relative physical distance between the ship and the target. This is the error coefficient.

[0079] Finally, based on the aforementioned dynamic error variance model, the fusion algorithm assigns weights according to the principle of minimizing variance. Since the equivalent measurement error variance of AIS is relatively stable, while radar error varies with distance... The weighting increases significantly with the increase in [the specific element], and this mechanism enables adaptive weight adjustment that favors radar observation at close range and AIS positioning at long range. Dynamic fusion weights The calculation is shown in Equation 40: (40) in, and These are the dynamic fusion weights for AIS and radar, respectively. After obtaining the adaptive weights, the target fusion state variables are obtained through weighted averaging. The calculation formula for the current AIS-radar dual-source observation fusion point is shown in Equation 41: (41) in, and These are the original trajectory points of AIS and radar, respectively. The current AIS-radar dual-source observation fusion point includes at least: fusion position, fusion velocity, fusion heading, and fusion timestamp. This point represents the current dual-source fusion result obtained by adaptively weighting the AIS observation point and the radar observation point based on distance at the current moment. In a stable dual-source state, it can be directly used as the current continuous fusion track output. During the source switching process from a single-source state to a dual-source effective state, it is used as the current dual-source observation fusion point to continue to undergo time-attenuation weighted fusion with historical predicted track points.

[0080] Furthermore, based on the association confidence score output in step 2... In addition to AIS and radar measurement errors, a fusion track confidence score is generated. As shown in Equation 42: (42) This embodiment dynamically allocates weights for AIS and radar measurement errors using the principle of minimizing variance, achieving dual-source fusion of AIS and radar. This fusion mechanism allows for greater trust in the relatively stable AIS positioning when the target is far from the ship, and greater trust in the high-precision shipborne radar when the target is close to the ship, thus obtaining optimal fusion accuracy across the entire range. Simultaneously, the current AIS-radar dual-source observation fusion point is obtained through weighted summation, and the fusion track confidence is output synchronously. This dual-source observation fusion point can be directly used as the continuous fusion track output in a stable dual-source state, and can be used as the current observation track point to participate in time-attenuation stitching in a source switching state from single-source to dual-source recovery.

[0081] In some embodiments of the present invention, the method for determining the source switching state is as follows: The current source state is determined based on the correlation result between the current AIS data and radar data; when the current source state changes relative to the previous source state, the target is determined to have entered the source switching state; the source switching state includes switching from an AIS-radar dual-source effective state to an AIS single-source state or a radar single-source state, and switching from an AIS single-source state or a radar single-source state to an AIS-radar dual-source effective state; time-sensitivity attenuation is used for track stitching during the source switching state, such as... Figure 5 As shown, it includes: S501. The linear extrapolation method is used to obtain the historical predicted track points for the current time by taking the first two AIS-radar continuous fusion track points at the current moment. S502. For the source switching state when the AIS-radar dual-source effective state is switched to the AIS single-source state or the radar single-source state: take the current AIS observation point or the current radar observation point as the current single-source observation track point, construct the time-effect attenuation weight function related to the source switching duration, and perform weighted summation based on the historical predicted track point, the time-effect attenuation weight function, the current single-source observation track point and its corresponding single-source data weight to obtain the stitched track point at the current moment. S503. For source switching states where AIS single-source state or radar single-source state is switched to AIS-radar dual-source effective state: First, perform distance-based adaptive weighting on the current AIS observation point and the current radar observation point to obtain the current AIS-radar dual-source observation fusion point. Then, perform weighted summation based on historical predicted track points, time-effect attenuation weight function, the current AIS-radar dual-source observation fusion point and its corresponding dual-source fusion weight to obtain the stitched track point at the current moment. S504. When the duration of the source switching state reaches the preset source switching time threshold, or the time attenuation weight is lower than the preset weight threshold, exit the source switching state and enter the corresponding AIS single source state, radar single source state, or AIS-radar dual source effective state.

[0082] This embodiment constructs a time-degradation weight function related to the source switching duration to achieve soft switching between historical predicted track points and the current single-source observed track point when switching from dual-source to single-source, and soft switching between historical predicted track points and the current AIS-radar dual-source observation fusion point when switching back from single-source to dual-source. This avoids sudden track abrupt changes caused by instantaneous changes in data source status, achieving a smooth track transition. It not only solves the problem of track abrupt changes and breaks caused by sensor signal loss or recovery, ensuring the continuity of track output, but also generates historical predicted track points at the current moment as prior states through linear extrapolation, enabling the system to maintain reasonable track predictions even when signals are lost or recovered. This significantly improves the reliability and robustness of multi-source fusion tracks in complex navigation environments.

[0083] In some embodiments of the present invention, the AIS-radar continuous fusion track includes at least: target number, target latitude and longitude, stitched track points, current data source status, and AIS-radar fusion track confidence.

[0084] This embodiment clarifies the core elements of AIS-radar fusion tracks, including at least: target number, target latitude and longitude, stitched track points, current data source status, and fusion track confidence. It not only provides the target with complete identity, location, motion status, and data source status, enabling the direct acquisition of latitude and longitude information when calculating the true relative angle, but also lays the data foundation for the global optimal matching of video targets and fusion tracks through structured fusion track information, ensuring the accuracy and completeness of the subsequent construction of the comprehensive association cost matrix.

[0085] In some embodiments of the present invention, step 3 further includes: 3-3 Track stitching and fusion based on time-decrease: When the target is in a source switching state, this invention employs a track stitching method based on time decay to achieve a smooth transition. First, for the prior estimation of historical fused tracks, a linear extrapolation method is used to calculate historical predicted track points. Let the sampling time of the fused track be T, and the first two times T1 and T2, the obtained fused target information are as follows: The fused target information at system time T is As shown in Equation 43: (43) Where D is the distance of the target relative to the ship, B is the angle relative to the ship, SOG is the speed over the ground, and COG is the heading over the ground.

[0086] Secondly, the duration of the build and source switching The relevant time-decrease weighting function satisfies the following condition: When the value approaches 0, the weight of the historical fusion track points approaches 1; when As the duration increases, the weight smoothly decays to 0. Correspondingly, the weight of the current observed track point gradually increases with the duration of the source switch. (Time-decrease weight) The calculation formula is shown in Equation 44: (44) in, This refers to the duration of the source switching. It is a decay constant used to control the rate at which the weights decay. Correspondingly, it represents the weight of the currently observed track point. The calculation is shown in Equation 45: (45) Final stitching track point By analyzing historical predicted flight paths Compared with the current observation track point The result is obtained by dynamic weighting, and the calculation formula is shown in Equation 46: (46) Among them, the current observation track point is the current single-source observation track point when the dual-source effective state switches to the single-source state, and the current AIS-radar dual-source observation fusion point when the single-source state returns to the dual-source effective state.

[0087] When switching from AIS-radar dual-source effective state to AIS single-source state or radar single-source state, the current AIS observation point or the current radar observation point is used as the current single-source observation track point, and the historical predicted track point is fused with the current single-source observation track point by time-effect attenuation weighting to obtain the stitched track point at the current moment.

[0088] For situations where AIS single-source status or radar single-source status is switched to AIS-radar dual-source effective status, firstly, based on the AIS measurement error and the radar measurement error, the current AIS observation point and the current radar observation point are subjected to distance-based adaptive weighted fusion to obtain the current AIS-radar dual-source observation fusion point; then, the historical predicted track points and the current AIS-radar dual-source observation fusion point are subjected to time-attenuation weighted fusion to obtain the stitched track point at the current moment.

[0089] When the duration of the source switching state reaches a preset source switching time threshold, or the time-degradation weight is lower than a preset weight threshold, the target exits the source switching state and enters the corresponding stable source state. In one embodiment, the preset source switching time threshold is 12 seconds.

[0090] Step 3 outputs the final set of continuously fused tracks as shown in Equation 47: (47) Each consecutive fused track includes at least the following as shown in Equation 48: (48) in, Number the target; To stitch together waypoints; This represents the current data source status. To fuse track confidence. Continuous fused track set. This will be used as the physical input for cross-modal video target association in step 4, instead of directly using a single AIS track or a single radar track for video matching.

[0091] In some embodiments of the present invention, the relative true azimuth of the target is determined based on the AIS-radar continuous fusion track, such as... Figure 6 As shown, it includes: S601. Using a standard spherical geometry model, determine the true azimuth of the target relative to the ship based on the current latitude and longitude of the ship and the target. S602. Subtract the true azimuth from the bow angle of the ship to obtain the relative true azimuth of the target from the ship's perspective.

[0092] This embodiment converts the absolute geographical bearing into a relative true bearing with the bow as the reference by subtracting the true azimuth from the ship's bow angle. This unifies the observation perspective of the visual sensor and provides an accurate reference space for subsequent cross-modal azimuth deviation calculations.

[0093] In some embodiments of the present invention, in step 4, firstly, spatial mapping models are established to calculate the target's true azimuth relative to the ship in physical space and its visual relative azimuth on the image plane. Secondly, combining the azimuth deviation and relative distance features between the two, a weighted association cost matrix reflecting the comprehensive positional deviation is constructed. Finally, an association threshold is introduced to exclude abnormal matching pairs with excessively large spatial feature differences, and the Hungarian algorithm is applied again to minimize the global cost in the bipartite graph, ultimately establishing the globally optimal correspondence between the track target and the visual target. Specifically, the steps include: 4-1 Calculation of relative azimuth angle in physical space: First, based on the AIS-radar fusion trajectory information output in step 3, the true bearing (TB) of the target ship relative to the ship is calculated, thus unifying it with the observation perspective of the ship's visual sensors. Let the ship's current moment... The latitude and longitude are Target ship The latitude and longitude are The difference in longitude between the two is shown in Equation 49: (49) Secondly, based on the standard spherical geometric model, intermediate variables are introduced. and As shown in Equations 50 and 51: (50) (51) True azimuth The calculation formula is shown in Equation 52: (52) The true azimuth is set to 0° in the true north direction and increases clockwise, with the value ranging from [0° to 360°].

[0094] Finally, the real-time bow heading provided by the ship's compass should be recorded as follows, based on the obtained true azimuth. This converts the absolute angle into a relative bearing (RB) with the bow as the reference point. The calculation formula is shown in Equation 53: (53) This formula allows the geographical location of a target acquired by AIS radar in the world coordinate system to be accurately mapped to the relative azimuth angle of the physical space from the ship's perspective.

[0095] In some embodiments of the present invention, video data is acquired through a visual sensor, which performs installation parameter calibration relative to the ship's coordinate system. The bow reference line in the image is corrected based on the installation offset angle and / or lateral installation offset of the optical principal axis relative to the bow direction obtained from the calibration. The geometric centerline of the video data frame is equivalent to the ship's heading reference line. The relative visual azimuth angle of the target is determined based on the video data, such as... Figure 7 As shown, it includes: S701. Detect the target ship in the video data to obtain the target detection box; take the horizontal coordinate of the midpoint of the bottom edge of the target detection box as the horizontal reference anchor point of the target in the picture, and determine the pixel distance of the target anchor point from the central axis based on the difference between the horizontal reference anchor point and the horizontal coordinate of the geometric central axis of the picture. S702: Based on the equivalent physical focal length of the visual sensor and the pixel distance of the target anchor point from the central axis, the two-dimensional pixel features of the target are mapped into a three-dimensional relative visual azimuth angle.

[0096] This embodiment calibrates the installation parameters of the vision sensor relative to the ship's coordinate system and corrects the bow reference line in the image based on the installation offset angle and / or lateral installation offset, ensuring that the corrected image reference line is consistent with the ship's heading reference. This eliminates the pixel drift problem caused by the sensitivity of traditional coordinate projection methods to changes in ship attitude. In a preferred embodiment, the vision sensor is fixedly installed at the center of the ship's bow with its optical axis parallel to the bow. Simultaneously, the relative visual azimuth angle mapped from the midpoint offset of the bottom edge of the detection frame and the equivalent physical focal length is under the same bow reference as the relative true azimuth angle in physical space, providing a stable and accurate cross-modal spatial correspondence for subsequent azimuth angle deviation calculations.

[0097] In some embodiments of the present invention, step 4 preferably further includes: 4-2 Calculation of Visual Azimuth Angle in Image Space: For the video target detected by the target detection algorithm in step 1, its visual azimuth angle relative to the bow direction needs to be calculated on the two-dimensional plane of the image. The shipborne visual sensor used in this invention completes installation parameter calibration relative to the ship's coordinate system. Based on the installation offset angle and / or lateral installation offset of the optical principal axis relative to the bow direction obtained from the calibration, the bow reference line in the image is corrected. Preferably, the visual sensor is fixedly installed at the exact center of the ship's bow, ensuring that its optical principal axis remains parallel to the ship's bow direction without any lateral offset. Based on this spatial relationship, in subsequent algorithm processing, the geometric centerline of the image can be directly equivalent to the ship's heading reference line in physical space.

[0098] First, taking the center point at the bottom of the image as the origin of the local coordinate system, the vertically upward central axis, i.e., the horizontal coordinate... This location serves as the reference baseline for the bow direction. For the first point output by the target detection model... For each detected target, extract the x-coordinate of the midpoint of the bottom edge of its bounding box. As a reference anchor point for the horizontal position of the target in the image, the anchor point is offset from the central axis of the image by a pixel distance. As shown in Equation 54: (54) in, Total pixel width of the image Secondly, by combining the camera's true physical horizontal field of view (HFOV) obtained from calibration, the camera's equivalent physical focal length at the current resolution can be derived. As shown in Equation 55: (55) Finally, based on perspective projection geometry, the two-dimensional pixel features are precisely mapped to the visual relative azimuth angle in three-dimensional physical space. Its mapping formula is shown in Equation 56: (56) This calculation process fully compensates for the nonlinear mapping deviation of the perspective projection plane, and its geometric mapping relationship is shown in the figure below. The calculated visual relative azimuth angle will be directly used as the core input dimension for constructing the multi-sensor correlation cost matrix.

[0099] In some embodiments of the present invention, a comprehensive correlation cost matrix is ​​constructed and solved based on the azimuth deviation between the relative true azimuth and the relative visual azimuth and the relative distance of the target relative to the ship. This yields the multi-source target fusion result of optimal matching between the video target and the AIS-radar continuous fusion track, such as... Figure 8 As shown, it includes: S801. Determine the azimuth deviation between the relative true azimuth and the relative visual azimuth based on the minimum circumferential angular distance; S802. Determine the actual distance between the target and the ship using latitude and longitude, and use the ratio of the actual distance between the target and the ship to the preset maximum observation distance as the relative true distance coefficient between the target and the ship; determine the relative visual distance coefficient between the target and the ship using the ordinate of the midpoint of the bottom edge of the video target detection box and the pixel height of the image; construct the distance feature deviation based on the relative true distance coefficient and the relative visual distance coefficient. S803. Weighted fusion of azimuth deviation and distance feature deviation to obtain a comprehensive correlation cost matrix; assigning infinity to elements in the comprehensive correlation cost matrix that exceed the preset range to obtain an updated comprehensive correlation cost matrix. S804. The Hungarian algorithm is used to solve the global optimal allocation of the updated integrated correlation cost matrix, and the target matching result of the optimal matching between the video target and the AIS-radar continuous fusion track is obtained. S805. Based on the target matching results, the information in the AIS-radar continuous fusion track is superimposed onto the video target to obtain the multi-source target fusion result.

[0100] This embodiment constructs a comprehensive correlation cost matrix by weighted fusion of azimuth deviation and range feature deviation, and uses the Hungarian algorithm to solve for the global optimal allocation, effectively overcoming the directional ambiguity and mismatch problem caused by a single feature when ships intersect and overlap. At the same time, by removing combinations with excessive spatial differences through threshold elimination, forced mismatch is avoided. Finally, the information in the AIS-radar continuous fusion track is superimposed on the video target, realizing high-precision fusion of multi-source targets from AIS, radar and video.

[0101] In some embodiments of the present invention, step 4 further includes: 4-3 Construction of the Related Cost Matrix: First, for fused track targets With visual target The difference in azimuth angle between the two is calculated and used as the first matching cost. Since the azimuth angle has a 360° periodic closed characteristic in geometric space, its angular deviation needs to be calculated using the minimum circumferential angle distance formula to avoid numerical calculation errors when crossing true north, as shown in Equation 57: (57) Secondly, for fused track targets The actual physical distance between it and the ship is calculated using latitude and longitude coordinates. To facilitate comparison with image features, a maximum observation distance is preset. As a normalization scaling factor, physical distance is converted into a relative distance coefficient. As shown in Equation 58: (58) The relative distance coefficient ranges from 0 to 1, with a smaller value indicating that the target is closer to the ship.

[0102] For visual targets In a scene with a flat water surface, the vertical coordinate of a target in the image has a strict monotonic correspondence with its actual distance. Let the total pixel height of the image be... The ordinate of the midpoint of the bottom edge of the detection box is Since the origin of the pixel coordinate system is located at the top left corner of the image, the ordinate... A higher value indicates that the target is closer to the bottom of the image, meaning it is closer to the ship. Based on this, a visual relative distance coefficient is constructed. As shown in Equation 59: (59) The visual relative distance coefficient is also distributed between 0 and 1.

[0103] Finally, based on the two coefficients mentioned above, the distance feature deviation cost between the track and the visual target is constructed. As shown in Equation 60: (60) After obtaining the azimuth deviation relative distance deviation Later, due to the different numerical ranges of the two, in constructing the final... Comprehensive correlation cost matrix At this time, it is necessary to numerically scale the azimuth deviation and fuse these two features using a weighted summation method, as shown in Equation 61: (61) in, This is the maximum allowable angle error threshold for the system. and These are the weighting coefficients for the azimuth cost and the distance cost, respectively, and they satisfy... This formula quantifies the overall positional deviation of the multi-source sensing targets on the two-dimensional plane at the current moment.

[0104] 4-4 Global optimal matching based on the Hungarian algorithm: First, to avoid the Hungarian algorithm forcibly pairing targets with excessively different spatial features, this invention introduces an association threshold. If the elements in the cost matrix Greater than the threshold This indicates the target of the flight path. With visual target The overall positional deviation exceeds a reasonable range and lacks physical spatial consistency. In this case, the cost element at that position is assigned infinity to exclude this combination in bipartite graph matching, thus obtaining the updated cost matrix after threshold filtering. As shown in Equation 62: (62) Subsequently, based on the updated cost matrix The Hungarian algorithm is applied to find the optimal allocation. The core idea of ​​this algorithm is based on the augmenting path theorem, which continuously expands the matching set through alternating search and state updates, ultimately finding a matching set with a dimension of... Boolean assignment matrix Its elements The goal of this allocation process is to minimize the total association cost of successfully matched pairs. To achieve the global minimum, the objective function expression is shown in Equation 63: (63) Finally, a one-to-one assignment constraint is set for the optimization model, that is, each track target is matched with at most one visual target, and each visual target is also matched with at most one track target, as shown in Equations 64 and 65: (64) (65) Among them, when the solution result is At that time, determine the target of the flight path. With visual target The association was successful, achieving the correspondence and fusion of physical space coordinates and image space features; for the allocation matrix... If a target fails to be successfully associated, that is, if all elements in the corresponding row or column are 0, it is determined as an independent unmatched target and retained in the next data processing cycle as a new target or a temporarily lost target.

[0105] Furthermore, once a target match is successful, information from the corresponding continuously fused track is overlaid onto the video target, including: the ship's MMSI number, latitude and longitude information, ground speed, and ground heading. This ultimately results in a unified shipborne target display that integrates AIS, radar, and video data.

[0106] This invention addresses the problems of traditional fusion methods being unable to dynamically adjust errors, leading to abrupt changes and breaks in the trajectory; traditional coordinate projection methods being sensitive to ship attitude, resulting in large cross-modal alignment errors; and single feature association being prone to target confusion and mismatch in dense intersection scenarios. It proposes a method for associating and fusing shipborne radar, AIS, and video data.

[0107] To address the issue of inconsistent spatiotemporal references for multi-source data, this invention proposes a multi-sensor spatiotemporal synchronous preprocessing mechanism. Spatially, it uniformly maps radar polar coordinates to the WGS-84 coordinate system, and temporally, it introduces a kinematic model to achieve smooth extrapolation and alignment with video frames. To address the difficulty of high-precision association between heterogeneous AIS and radar observation sources, a candidate association mechanism based on physical threshold screening, a spatial-motion feature weighted cost matrix, and Hungarian global allocation is proposed. This mechanism outputs association confidence and single-source / dual-source status while avoiding forced matching of weakly correlated targets. To address the issue of track jumps caused by incomplete data sources in actual navigation, a track stitching strategy encompassing source state identification, adaptive weighted fusion, and time-based decay is developed to ensure track smoothness and continuity. Finally, for the cross-modal fusion of visual and positional data, this invention constructs a comprehensive association cost matrix based on physical azimuth, visual azimuth, and relative distance features. It then combines an association threshold with the Hungarian algorithm to minimize the global cost, significantly improving the accuracy and robustness of multi-source sensing target correspondence.

[0108] To objectively quantify the actual performance of the target detection model and association algorithm, this invention selects a standard evaluation metric commonly used in the fields of computer vision and multi-sensor fusion. For the task of ship target detection in two-dimensional image space, accuracy is primarily used. ), recall rate ( ) and Mean Average Precision This is used as the core evaluation indicator. Its mathematical definitions are shown in Equations 66 to 68 respectively: (66) (67) (68) In the formula, This represents the number of positive samples that were correctly predicted as ships by the model. This indicates the number of negative samples that were falsely identified as ships in the background. This indicates the actual number of ships that were missed during inspection. The integral area under the precision-recall curve is used to measure the overall detection capability of the model, since this invention mainly targets the detection of a single major category of ships.

[0109] For visual-location data association and fusion algorithms, the main approach is to use matching accuracy. The matching effect between the visual detection box and the location data is evaluated using the formula shown in Equation 69: (69) In the formula, This represents the total number of heterogeneous sensor target pairs that were correctly matched by the algorithm. This represents the total number of target pairs that are manually identified as having a real correlation.

[0110] The test data selected representative synchronously collected data segments from three typical navigation scenarios (wide waters, narrow channels, and port anchorages) and manually labeled them with ground truth.

[0111] When constructing the comprehensive cost matrix, the reasonable setting of thresholds and weights is a prerequisite for ensuring the stable operation of the matching algorithm. Based on the calibration parameters of the shipborne camera and the error distribution patterns of the position data, this invention sets the spatial correlation threshold as follows: When the overall matching cost of heterogeneous target pairs exceeds this threshold, they are directly determined to be unrelated entities. This mechanism effectively filters out forced mismatches caused by false alarms from a single sensor.

[0112] Regarding the allocation of feature fusion weights, since the physical azimuth accuracy provided by AIS and radar is generally higher than the relative distance accuracy estimated based on monocular images, a higher weight is assigned to azimuth deviation in the cost matrix. In this invention, a uniform azimuth cost weight is set. Relative distance cost weight This was done to verify the performance of the algorithm in various complex water scenarios under fixed parameter configurations.

[0113] In some embodiments of the present invention, in the fused visual image, the system not only outputs the two-dimensional bounding box of the ship extracted by the target detection model, but also accurately overlays and displays the corresponding ship's location information, identity information, and other data as text labels next to the corresponding image target using the algorithm of the present invention, such as... Figures 9 to 11 As shown, blue boxes indicate correct associations, and red boxes indicate incorrect associations. The parameters in the figure are MMSI (Unique Identifier for Vessels, i.e., Maritime Mobile Service Identifier), Lon (longitude, the longitude coordinates of the target vessel), Lat (latitude, the latitude coordinates of the target vessel), and Cost (the comprehensive association cost between the video target and the corresponding AIS-radar fusion track; the smaller the value, the higher the matching degree). Wide sea scene ( Figure 9 In this scenario, the ships are relatively dispersed. For example... Figure 9 As shown, for small targets at a distance that are near the horizon and have very inconspicuous visual features, the system relies on high-precision azimuth matching to successfully project the ship's identity and position information onto the corresponding tiny detection box, effectively extending the perception distance of monocular vision.

[0114] Narrow waterway scene Figure 10 In this scenario, ships frequently pass each other in opposite or the same direction. For example... Figure 10As shown, even if the target's scale changes drastically due to vertical movement in the image, or if there is a brief visual overlap at the moment of intersection, the matching strategy based on the dual features of azimuth and distance and the global optimal allocation still maintains the stable binding of the track ID, and there is no information jump or matching loss.

[0115] Port anchorage scene ( Figure 11 This scenario is the most complex, with a high density of anchored and sailing ships. For example... Figure 11 As shown, traditional single-feature algorithms are prone to mismatches when dealing with multiple ships at similar azimuth angles in the image. However, as... Figure 11 As shown in the magnified local area, the system successfully performed matching by combining visual relative distance cost and azimuth cost, distinguishing near and far targets in the same direction, and matching them with the correct identity and location data respectively.

[0116] To comprehensively and objectively evaluate the overall performance advantages of the proposed correlation fusion algorithm, this invention manually labeled 780 pairs of real correlation targets in test datasets of three typical general aviation scenarios, and quantitatively compared them with commonly used algorithms in traditional multi-source data fusion.

[0117] The comparative experiment selected two representative methods as references: one is the coordinate projection method relying on three-dimensional spatial geometric mapping, which uses camera intrinsic and extrinsic parameters to project latitude and longitude coordinates onto the pixel plane and then performs nearest neighbor matching; the other is the azimuth-based nearest neighbor matching algorithm commonly used in water surface sensing engineering. The matching accuracy statistics of the three algorithms in various scenarios are shown in Table 1.

[0118] Table 1: Quantitative Experimental Results of Different Association Matching Algorithms

[0119] Analysis of the data in Table 1 shows that the algorithm of this invention achieved optimal performance in all scenarios. The coordinate projection method, which relies on strict spatial geometric transformations, performed poorly, mainly because the high-frequency rolling and pitching of the ship caused by wind and waves on the water surface leads to drastic changes in camera extrinsic parameters, resulting in significant pixel position drift during direct projection. The azimuth-based matching method effectively avoids this physical limitation. Especially in the complex environment of port anchorages, relying solely on azimuth angles for correlation and fusion reduces the accuracy to 64.8%, while the algorithm of this invention maintains a high level of 90.7%, demonstrating its good robustness.

[0120] It should be noted that, to further explore the individual contributions of the two core mechanisms, "distance-azimuth feature cost matrix" and "Hungarian global allocation," to the overall performance of this method, a cross-ablation experiment was designed. By controlling variables, the system performance under different combinations of feature dimensions and allocation strategies was tested, and the experimental results are shown in Table 2.

[0121] Table 2: Experimental Results of Ablation Using the Association Matching Algorithm

[0122] By comparing the ablation experimental data longitudinally and laterally, the following conclusions can be drawn: Effectiveness analysis of the allocation strategy: Experiments show that in wide waters with sparse target distribution, the performance gain of introducing the Hungarian algorithm is not significant. However, in narrow channels and port anchorages where there are many targets and intersections occur, the advantages of the Hungarian algorithm are greatly demonstrated. For example, with the use of dual features, the Hungarian algorithm improves the accuracy of narrow channel scenarios by 6.6 percentage points. This proves that the global cost minimization model effectively overcomes the inherent defect of the local nearest neighbor algorithm, which is prone to getting trapped in local optima when targets are dense, thus leading to a chain of mismatches.

[0123] Effectiveness Analysis of Feature Dimensions: Experimental data shows that introducing relative distance features based on the image's vertical coordinate plays a crucial role in improving matching accuracy in complex navigation environments. Specifically, comparing combinations B and D, under the condition of using the Hungarian algorithm, fusing both distance and azimuth features improves the system's matching accuracy by 22.5 percentage points in port anchorage scenarios. This result fully demonstrates that when multiple ships are in similar observation positions, the introduction of distance features can effectively resolve directional ambiguities caused by single azimuth matching, further validating the rationality and necessity of the dual-feature cost matrix design.

[0124] The ablation experiments described above demonstrate that the dual constraint mechanism of azimuth angle and relative distance, along with the global optimal allocation strategy, are highly complementary in handling complex multi-target matching problems on water surfaces. The two work together to enhance the robustness and adaptability of the multi-source sensing system in complex water surface environments.

[0125] like Figure 12 As shown, the present invention also provides a shipborne radar, AIS and video data association and fusion system 1200, comprising: AIS receiver 1201 is used to receive AIS data broadcast by targets around the ship; Shipborne radar 1202 is used to detect radar data of targets around the ship; Video sensor 1203 is used to collect video data directly in front of the ship; Compass 1204 is used to provide the ship's real-time bow angle; GNSS receiver 1205 is used to provide the ship's latitude and longitude, speed over land, and heading over land; The memory 1206 is used to store programs and AIS data, radar data, video data, real-time bow angle, the ship's latitude and longitude, ground speed and ground heading; The processor 1207 is communicatively connected to the AIS receiver, the shipborne radar, and the video sensor, and coupled to the memory. It is used to execute the program stored in the memory to implement the steps in the shipborne radar, AIS and video data association and fusion method provided in the above method embodiments. Display 1208, coupled to the processor, is used to output the multi-source target fusion result of superimposing information from AIS-radar continuous fusion tracks onto video targets.

[0126] In one embodiment, when the processor 1207 executes the shipborne radar, AIS, and video data association and fusion program stored in the memory 1206, the following steps can be implemented: Acquire spatiotemporally aligned radar data, AIS data, and video data including that of the target ship; Construct and solve a candidate association cost matrix representing the possibility of association between AIS data and radar data to obtain the matched target and the single-source target; To match targets with single-source targets, data source status identifiers are established, including AIS-radar dual-source effective, AIS single-source, radar single-source, and source switching. When AIS-radar dual-source effective, AIS data and radar data are fused. When AIS single-source or radar single-source is in effect, the current single-source data is kept as the track output. When the source is switched, time-effect attenuation is used to stitch the track together to obtain a continuous fused AIS-radar track. Based on the AIS-radar continuous fusion track, the relative true azimuth of the target is determined; based on the video data, the relative visual azimuth of the target is determined; based on the azimuth deviation between the relative true azimuth and the relative visual azimuth and the relative distance of the target relative to the ship, a comprehensive correlation cost matrix is ​​constructed and solved to obtain the multi-source target fusion result of optimal matching between the video target and the AIS-radar continuous fusion track.

[0127] It should be understood that when the processor 1207 executes the shipborne radar, AIS and video data association and fusion program in the memory 1206, in addition to the functions mentioned above, it can also perform other functions, as can be found in the description of the corresponding method embodiments above.

[0128] Accordingly, this application also provides a computer-readable storage medium for storing computer-readable programs or instructions. When the programs or instructions are executed by a processor, they can implement the steps or functions of the shipborne radar, AIS and video data association and fusion methods provided in the above-described method embodiments.

[0129] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.), and the computer program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0130] The above provides a detailed description of the shipborne radar, AIS, and video data association and fusion method and system provided by this invention. Specific examples have been used to illustrate the principles and implementation methods of this invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this invention. At the same time, those skilled in the art will recognize that there will be changes in the specific implementation methods and application scope based on the ideas of this invention. Therefore, the content of this specification should not be construed as a limitation of this invention.

Claims

1. A method for associating and fusing shipborne radar, AIS, and video data, characterized in that, include: Acquire spatiotemporally aligned radar data, AIS data, and video data including that of the target ship; Construct and solve a candidate association cost matrix that represents the possibility of association between the AIS data and the radar data to obtain the matching target and the single-source target; A data source status identifier is established for the matched target and the single-source target, including AIS-radar dual-source effective, AIS single-source, radar single-source, and source switching. When the AIS-radar dual-source effective state is reached, AIS data and radar data are fused. When the AIS single-source state or the radar single-source state is reached, the current single-source data is kept as the track output. When the source is switched, time-effect attenuation is used to stitch the track to obtain the AIS-radar continuous fused track. Based on the AIS-radar continuous fusion track, the relative true azimuth of the target is determined; based on the video data, the relative visual azimuth of the target is determined; based on the azimuth deviation between the relative true azimuth and the relative visual azimuth and the relative distance of the target relative to the ship, a comprehensive correlation cost matrix is ​​constructed and solved to obtain the multi-source target fusion result of optimal matching between the video target and the AIS-radar continuous fusion track.

2. The method for associating and fusing shipborne radar, AIS, and video data according to claim 1, characterized in that, The process of constructing and solving a candidate association cost matrix characterizing the correlation probability between the AIS data and the radar data to obtain the matching target and the single-source target includes: An AIS target set and a radar target set are constructed based on the AIS data and the radar data, respectively, and an AIS-radar target pair is constructed based on the AIS target set and the radar target set; The AIS-radar target pairs that simultaneously satisfy the preset spatial distance difference, preset speed difference, preset heading difference and preset relative bearing difference are selected to obtain candidate target association pairs; A candidate association cost matrix is ​​constructed based on the spatial distance, speed, heading, orientation, time extrapolation error, and data quality of the candidate target association pairs; The Hungarian algorithm is used to solve the global optimal allocation of the candidate association cost matrix, and the AIS-radar matched target, AIS single-source target and radar single-source target are obtained and output; the AIS-radar matched target output also includes the corresponding association confidence.

3. The method for associating and fusing shipborne radar, AIS, and video data according to claim 1, characterized in that, The step of establishing data source status identifiers for the matched target and the single-source target, including AIS-radar dual-source active, AIS single-source, radar single-source, and source switching, also includes: Maintain the target state vector for all targets; the target state vector includes the target number at the current moment, the data source status, the time of the most recent dual-source fusion, the set of historical fusion tracks, and the correlation confidence.

4. The method for associating and fusing shipborne radar, AIS, and video data according to claim 3, characterized in that, The fusion of AIS data and radar data when the AIS-radar dual-source effective state is mentioned includes: The AIS measurement error is determined based on the inherent observation noise of the global navigation and positioning system; the radar measurement error is determined based on the inherent measurement error of the radar and the distance error between the ship and the target. Based on the AIS measurement error and the radar measurement error, the AIS dynamic fusion weights and radar dynamic fusion weights are determined using the variance minimization principle. The current AIS-radar dual-source observation fusion point is obtained by weighted summation of the original AIS trajectory point, the AIS dynamic fusion weight corresponding to the original AIS trajectory point, the radar original trajectory point, and the radar dynamic fusion weight corresponding to the original radar trajectory point; and the AIS-radar fusion track confidence is generated based on the correlation confidence, the AIS measurement error, and the radar measurement error.

5. The method for associating and fusing shipborne radar, AIS, and video data according to claim 4, characterized in that, The method for determining the source switching state is as follows: The current source state is determined based on the correlation result between AIS data and radar data at the current moment. When the current source state changes relative to the source state at the previous moment, the target is determined to have entered the source switching state. The source switching state includes switching from an AIS-radar dual-source effective state to an AIS single-source state or a radar single-source state, and switching from an AIS single-source state or a radar single-source state to an AIS-radar dual-source effective state. The use of time-attenuation for track stitching during the source switching state includes: The historical predicted track points for the current moment are obtained by using linear extrapolation on the first two AIS-radar continuous fusion track points at the current moment; For the source switching state where the AIS-radar dual-source effective state switches to AIS single-source state or radar single-source state: the current AIS observation point or the current radar observation point is used as the current single-source observation track point. A time-effect attenuation weight function related to the source switching duration is constructed. The historical predicted track point, the time-effect attenuation weight function, the current single-source observation track point and its corresponding single-source data weight are weighted and summed to obtain the stitched track point at the current moment. For the source switching state where the AIS single-source state or radar single-source state switches to the AIS-radar dual-source effective state: first, the current AIS observation point and the current radar observation point are adaptively weighted based on distance to obtain the current AIS-radar dual-source observation fusion point. Then, the historical predicted track points, the time-effect attenuation weight function, the current AIS-radar dual-source observation fusion point and its corresponding dual-source fusion weight are weighted and summed to obtain the stitched track point at the current moment. When the duration of the source switching state reaches a preset source switching time threshold, or when the time-attenuation weight is lower than a preset weight threshold, the source switching state is exited, and the corresponding AIS single-source state, radar single-source state, or AIS-radar dual-source effective state is entered.

6. The method for associating and fusing shipborne radar, AIS, and video data according to claim 5, characterized in that, The AIS-radar continuous fusion track includes at least: target number, target latitude and longitude, stitched track points, current data source status, and the confidence level of the AIS-radar fusion track.

7. The method for associating and fusing shipborne radar, AIS, and video data according to claim 1, characterized in that, The determination of the target's relative true azimuth angle based on the AIS-radar continuous fusion track includes: Using a standard spherical geometry model, the true azimuth of the target relative to the ship is determined based on the current latitude and longitude of the ship and the target. The relative true azimuth of the target from the ship's perspective is obtained by subtracting the true azimuth from the ship's bow angle.

8. The method for associating and fusing shipborne radar, AIS, and video data according to claim 1, characterized in that, The video data is acquired through a visual sensor, which performs installation parameter calibration relative to the ship's coordinate system. The installation offset and / or lateral installation offset of the optical principal axis relative to the bow direction are used to correct the bow reference line in the image. The geometric centerline of the video data is equivalent to the heading reference line of the ship. Determining the relative visual azimuth angle of the target based on the video data includes: Target detection is performed on the target ship in the video data to obtain a target detection box; the horizontal coordinate of the midpoint of the bottom edge of the target detection box is used as the horizontal reference anchor point of the target in the picture; the pixel distance of the target anchor point from the central axis is determined based on the difference between the horizontal reference anchor point and the horizontal coordinate of the geometric central axis of the picture. Based on the equivalent physical focal length of the visual sensor and the pixel distance of the target anchor point from the central axis, the two-dimensional pixel features of the target are mapped into a three-dimensional relative visual azimuth angle.

9. The method for associating and fusing shipborne radar, AIS, and video data according to claim 1, characterized in that, The process involves constructing and solving a comprehensive correlation cost matrix based on the azimuth deviation between the relative true azimuth and the relative visual azimuth, and the relative distance of the target to the ship. This yields the multi-source target fusion result, which optimally matches the video target with the AIS-radar continuous fusion track. The result includes: The azimuth deviation between the relative true azimuth and the relative visual azimuth is determined based on the minimum circumferential angular distance. The actual distance between the target and the ship is determined by latitude and longitude. The ratio of the actual distance between the target and the ship to the preset maximum observation distance is used as the relative true distance coefficient between the target and the ship. The relative visual distance coefficient between the target and the ship is determined by the ordinate of the midpoint of the bottom edge of the video target detection box and the pixel height of the image. The distance feature deviation is constructed based on the relative true distance coefficient and the relative visual distance coefficient. The azimuth deviation and the distance feature deviation are weighted and fused to obtain a comprehensive correlation cost matrix; elements in the comprehensive correlation cost matrix that exceed a preset range are assigned to infinity to obtain an updated comprehensive correlation cost matrix. The Hungarian algorithm is used to solve the updated integrated correlation cost matrix for global optimal allocation, and the target matching result of the optimal matching between the video target and the AIS-radar continuous fusion track is obtained. Based on the target matching result, the information in the AIS-radar continuous fusion track is superimposed onto the video target to obtain the multi-source target fusion result.

10. A system for associating and fusing shipborne radar, AIS, and video data, characterized in that, include: The AIS receiver is used to receive AIS data broadcast by targets around the ship. Shipborne radar, used to detect radar data of targets around the ship; Video sensors are used to collect video data directly in front of the ship; A compass is used to provide the ship's real-time heading angle; GNSS receiver, used to provide the ship's latitude and longitude, speed above ground and heading above ground; The memory is used to store the program and the AIS data, the radar data, the video data, the real-time bow angle, the ship's latitude and longitude, ground speed and ground heading; The processor is communicatively connected to the AIS receiver, the shipborne radar, and the video sensor, and coupled to the memory, respectively, for executing the program stored in the memory to implement the steps in the shipborne radar, AIS and video data association and fusion method according to any one of claims 1 to 9; A display, coupled to the processor, is used to output the multi-source target fusion result of superimposing information from the AIS-radar continuous fusion track onto the video target.

Images