Ship target identifying and tracking method and system based on multi-source data fusion

Abstract

The invention discloses a ship target identification tracking method and system based on multi-source data fusion, and relates to the field of electric digital data processing, and the method comprises the steps: collecting ship monitoring data of a multi-source sensor, including position coordinates, area and color features; generating an initial ship trajectory according to the monitoring data, calculating the speed, area and color similarity of a detection point, and associating to obtain a trajectory pair; carrying out abnormal point identification and trajectory optimization based on a similarity energy function containing options such as dynamics and continuity; calculating the state difference between the optimized trajectories to perform similarity test, and judging the trajectories exceeding a threshold value as the same target to form a multi-source trajectory group; and calculating a dynamic fusion weight according to the error standard deviation of the trajectory relative to the average trajectory, and performing weighted fusion to obtain a target trajectory. By implementing the method, the fusion precision of the ship target trajectory can be improved.

Description

Technical Field

[0001] The present application relates to the field of electronic digital data processing, and in particular to a ship target identification and tracking method and system based on multi-source data fusion. Background Art

[0002] In marine safety supervision, accurate identification and real-time tracking of ship targets are important means of protection. With the increasing frequency of marine ship activities and the continuous expansion of navigation range, higher requirements are placed on the accuracy and reliability of ship monitoring.

[0003] Among the related technologies, the identification and tracking of ships at sea mainly adopts radar detection, optical remote sensing and satellite remote sensing. These technologies collect ship information by setting up sensors, obtain characteristic data such as the position and speed of the ship, and then obtain the navigation track of the ship through data processing. Among them, radar detection can work all-weather but is easily disturbed by waves, optical remote sensing has high resolution but is greatly affected by weather, and satellite remote sensing has a wide coverage but a long update cycle.

[0004] In practical applications, due to the limited detection capabilities of a single sensor, the acquired ship trajectory data often has problems such as unstable accuracy, missing data, and anomalies. When the sea conditions are complex or the weather conditions change, the quality of the data collected by the sensor will significantly decrease, resulting in deviations in the generated ship trajectory, affecting subsequent monitoring and early warning. Summary of the invention

[0005] The present application provides a ship target recognition and tracking method and system based on multi-source data fusion, which is used to improve the fusion accuracy of ship target trajectory.

[0006] In a first aspect, the present application provides a ship target identification and tracking method based on multi-source data fusion, which uses a ship target identification and tracking system, and the method includes: collecting ship monitoring data of the same sea area from multi-source sensors, the ship monitoring data including ship position coordinates, area characteristics and color characteristics; generating an initial ship trajectory of each sensor according to the ship monitoring data, calculating the detection point similarity of the initial ship trajectory and associating it to obtain an associated trajectory pair, the detection point similarity including speed similarity, area similarity and color similarity; performing outlier identification and trajectory optimization on the associated trajectory pair based on a similarity energy function to obtain an optimized ship trajectory, the similarity energy function including a dynamic option, a continuity option, a fusion interval option, a color similarity option and an adjustment option; performing a similarity test by calculating the state difference between the optimized ship trajectories, judging the optimized ship trajectories with a similarity exceeding a preset threshold as the same target and associating them to form a multi-source trajectory group; calculating a dynamic fusion weight according to the standard deviation of the measurement error of each optimized ship trajectory in the multi-source trajectory group relative to its average trajectory, and performing weighted fusion to obtain a ship target trajectory.

[0007] By adopting the above technical solutions, the ship monitoring data collected by multi-source sensors contains location, area and color features, providing a comprehensive data basis for subsequent similarity calculations. Based on the association of detection point similarity and the optimization of similarity energy function, outliers can be effectively identified and eliminated. Combined with the multi-source trajectory group fusion mechanism with dynamic fusion weights, the measurement advantages of different sensors are fully utilized, and the final generated ship target trajectory has higher accuracy and reliability.

[0008] In combination with some embodiments of the first aspect, in some embodiments, the step of calculating the similarity of the detection points of the initial ship trajectory and associating them to obtain an associated trajectory pair specifically includes: calculating the speed similarity of adjacent detection points based on the position information of the detection points in the initial ship trajectory; calculating the area similarity of adjacent detection points based on the rectangular frame of the detection points in the initial ship trajectory; calculating the color similarity of adjacent detection points based on the color features of the detection points in the initial ship trajectory; and associating adjacent detection points based on a weighted combination of the speed similarity, the area similarity and the color similarity to obtain an associated trajectory pair.

[0009] By adopting the above technical solution, the similarities of the three dimensions of speed, area and color of the detection points are weighted and combined to associate. Speed ​​similarity reflects the consistency of movement, area similarity reflects the law of target scale change, and color similarity ensures the accuracy of target recognition. The synergy of multi-dimensional features significantly improves the accuracy of trajectory association and reduces the occurrence of false association and missed association.

[0010] In combination with some embodiments of the first aspect, in some embodiments, the steps of identifying outliers and optimizing the trajectory of the initial ship trajectory based on a similarity energy function to obtain an optimized ship trajectory specifically include: using a similarity energy function to calculate the similarity of adjacent trajectory points in the initial ship trajectory, the similarity energy function includes a dynamic option, a continuity option, a fusion interval option, a color similarity option and an adjustment option; constructing a linear model based on a speed drift coefficient and a disturbance variance factor to calculate the mutation probability of the trajectory points, eliminating trajectory points whose mutation probability exceeds a preset threshold, and using a moving mean method to supplement the missing fragments after eliminating the outliers to obtain an optimized ship trajectory.

[0011] By adopting the above technical solutions, the similarity energy function integrates multiple options such as dynamics and continuity to comprehensively characterize the correlation between trajectory points. The linear model constructed based on the speed drift coefficient and the disturbance variance factor accurately captures the trajectory mutation characteristics, and the moving mean method supplements the missing fragments to maintain the continuity of the trajectory. The optimized ship trajectory is smoother and more coherent, effectively eliminating abnormal fluctuations.

[0012] In combination with some embodiments of the first aspect, in some embodiments, the step of performing a similarity test by calculating the state difference between the optimized ship trajectories, determining the optimized ship trajectories with a similarity exceeding a preset threshold as the same target and associating them to form a multi-source trajectory group specifically includes: calculating the state estimation difference of the optimized ship trajectories of different sensors, where the state estimation difference includes position, heading and speed information; performing a likelihood ratio test based on the joint probability density function based on uncertain distribution according to the state estimation difference, determining whether the trajectories of different sensors belong to the same target, and obtaining the associated multi-source trajectory group.

[0013] By adopting the above technical solutions, the state estimation difference covers the position, heading and speed information, and fully reflects the motion state of the ship. Based on the joint probability density function of the uncertain distribution, the likelihood ratio test is performed to accurately evaluate the correlation degree of different sensor trajectories. The formed multi-source trajectory group fully utilizes the detection advantages of each sensor and provides a high-quality data foundation for subsequent fusion.

[0014] In combination with some embodiments of the first aspect, in some embodiments, the step of calculating the dynamic fusion weight according to the standard deviation of the measurement error of each optimized ship trajectory in the multi-source trajectory group relative to its average trajectory, and performing weighted fusion to obtain the ship target trajectory, specifically includes: calculating the standard deviation of the measurement error of each optimized ship trajectory in the multi-source trajectory group relative to the average trajectory of the multi-source trajectory group; calculating the dynamic fusion weight according to the measurement error standard deviation; and performing weighted fusion on the multi-source trajectory group according to the dynamic fusion weight to obtain the ship target trajectory.

[0015] By adopting the above technical solution, the standard deviation of the measurement error of each optimized ship trajectory in the multi-source trajectory group relative to the average trajectory reflects the reliability of the trajectory. The dynamic fusion weight is adjusted in real time with the change of measurement error, so that the trajectory with higher accuracy plays a greater role in the fusion process. The ship target trajectory after weighted fusion inherits the advantages of each sensor.

[0016] In combination with some embodiments of the first aspect, in some embodiments, the step of generating the initial ship trajectory of each sensor based on the ship monitoring data specifically includes: when no detection point associated with the target detection point can be found in the previous frame image, a new trajectory segment is generated with the detection point as the starting point of the trajectory; when no detection point associated with the target detection point can be found in the next frame image, the target detection point is used as the end point of the current trajectory segment; it is determined whether the number of frames in which the trajectory appears continuously is less than a preset threshold, and if so, it is determined to be a trajectory fragment; and the energy function value after the trajectory fragment is fused with other trajectory segments is calculated, and when the energy function value decreases, the trajectory fragment is retained, otherwise it is removed from the trajectory space to obtain the initial ship trajectory.

[0017] By adopting the above technical solution, trajectory fragments are dynamically generated and terminated according to the correlation of detection points, realizing automatic initialization of trajectory. The judgment mechanism of energy function value effectively screens trajectory fragments and eliminates false trajectories caused by noise. The generated initial ship trajectory has strong spatiotemporal continuity, laying a good foundation for subsequent trajectory optimization.

[0018] In combination with some embodiments of the first aspect, in some embodiments, the step of eliminating trajectory points whose mutation probability exceeds a preset threshold and using the moving mean method to supplement the missing fragments after eliminating the abnormal points to obtain the optimized ship trajectory specifically includes: setting a sliding time window, calculating the speed drift coefficient and the disturbance variance factor of the trajectory point in the sliding time window; constructing a linear model of the trajectory point position based on the speed drift coefficient and the disturbance variance factor, and calculating the mutation probability of the trajectory point according to the deviation between the trajectory point and the predicted position of the linear model; eliminating the abnormal points whose mutation probability exceeds the preset probability threshold to obtain the trajectory fragment after eliminating the abnormal points; using the moving mean method to supplement the missing part in the trajectory fragment after eliminating the abnormal points, and when the time interval between the trajectory fragments exceeds the preset interval threshold, using a two-way search method to supplement, so as to obtain the optimized ship trajectory.

[0019] By adopting the above technical solutions, the velocity drift coefficient and disturbance variance factor calculated in the sliding time window reflect the changing characteristics of the local trajectory. The mutation probability calculation based on the linear model accurately identifies abnormal points, and the moving mean method and the two-way search method adaptively select the supplementary strategy according to the time interval. The optimized ship trajectory not only maintains the original motion law, but also eliminates the trajectory fluctuation caused by sensor errors, and the continuity and smoothness of the trajectory are improved.

[0020] In a second aspect, an embodiment of the present application provides a ship target identification and tracking system, which includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the ship target identification and tracking system to execute the method described in the first aspect and any possible implementation method of the first aspect.

[0021] In a third aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when the computer program product is run on a ship target identification and tracking system, enables the ship target identification and tracking system to execute the method described in the first aspect and any possible implementation method of the first aspect.

[0022] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions. When the instructions are executed on a ship target identification and tracking system, the ship target identification and tracking system executes the method described in the first aspect and any possible implementation method of the first aspect.

[0023] It can be understood that the ship target identification and tracking system provided in the second aspect, the computer program product provided in the third aspect, and the computer storage medium provided in the fourth aspect are all used to execute the method provided in the embodiment of the present application. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method, which will not be repeated here.

[0024] One or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. The ship monitoring data collected by multi-source sensors in this application contains location, area and color features, providing a comprehensive data basis for subsequent similarity calculations. Based on the correlation of detection point similarity and the optimization of similarity energy function, outliers can be effectively identified and removed. The multi-source trajectory group fusion mechanism combined with dynamic fusion weights makes full use of the measurement advantages of different sensors, and the final generated ship target trajectory has higher accuracy and reliability.

[0025] 2. This application combines and associates the similarities of the three dimensions of speed, area and color of the detection points in a weighted manner. Speed ​​similarity reflects the consistency of movement, area similarity reflects the law of target scale change, and color similarity ensures the accuracy of target recognition. The synergy of multi-dimensional features significantly improves the accuracy of trajectory association and reduces the occurrence of false associations and missed associations.

[0026] 3. This application integrates multiple options such as dynamics and continuity through the similarity energy function to comprehensively characterize the correlation between trajectory points. The linear model constructed based on the speed drift coefficient and the disturbance variance factor accurately captures the trajectory mutation characteristics, and the moving mean method supplements the missing fragments to maintain the continuity of the trajectory. The optimized ship trajectory is smoother and more coherent, effectively eliminating abnormal fluctuations. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 It is a flow chart of a ship target identification and tracking method based on multi-source data fusion in an embodiment of the present application; Figure 2 It is another flow chart of the ship target recognition and tracking method based on multi-source data fusion in an embodiment of the present application; Figure 3 It is a schematic diagram of the structure of a physical device of a ship target identification and tracking system in an embodiment of the present application. DETAILED DESCRIPTION

[0028] The terms used in the following embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to be used as limitations to the present application. As used in the specification of the present application, the singular expressions "one", "a kind of", "above", "the" and "this" are intended to also include plural expressions, unless there is a clear indication to the contrary in the context. It should also be understood that the term "and / or" used in the present application refers to any or all possible combinations comprising one or more of the listed items.

[0029] In the following, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as suggesting or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features, and in the description of the embodiments of the present application, unless otherwise specified, "plurality" means two or more.

[0030] For ease of understanding, the application scenarios of the embodiments of the present application are introduced below.

[0031] Identification and tracking of ships at sea is one of the important means to maintain marine safety. With the stronger maneuverability of the position of ships at sea and the diversification of the environment, a single sensor is difficult to meet the needs of accurate tracking of ships. Multi-source information fusion is mainly used in the field of target tracking by coordinating the operation of multiple sensors to perform information fusion processing, so as to achieve more accurate estimation and judgment of the actual situation of the target. At present, methods such as point trace association fusion based on AIS and satellite-borne SAR data, association fusion based on electronic and optical remote sensing, point trace fusion based on ground wave radar and AIS, and point trace fusion based on satellite-borne SAR, ground wave radar and AIS are mostly used for ship tracking. With the increase of ship detection sensors, how to fuse multi-source sensors to carry out ship identification and tracking to further improve the accuracy of ship identification and tracking has become a hot and difficult point in research. The present invention is based on the target and trajectory data obtained by electronic information, satellite-borne SAR, and high-resolution optical recognition, and realizes high-precision ship target identification and tracking through processing technologies such as trajectory optimization, trajectory association, and trajectory information fusion.

[0032] For ease of understanding, the following describes the process of the method provided by this implementation in combination with the above scenario. Figure 1 , which is a flow chart of a ship target identification and tracking method based on multi-source data fusion in an embodiment of the present application.

[0033] S101. Collect ship monitoring data of the same sea area from multiple source sensors, where the ship monitoring data includes ship position coordinates, area features, and color features.

[0034] Among them, multi-source sensors refer to multiple different types of monitoring methods that can obtain ship information in the same sea area, including but not limited to radar satellites, optical satellites, AIS, optoelectronics, etc. Ship monitoring data refers to the collection of ship target feature information collected by sensors. The ship position coordinates are used to represent the position information of the ship on a two-dimensional plane, including longitude and latitude. Area features refer to the size of the area occupied by the ship target in the image. Color features are used to represent the appearance characteristics of the ship target, which are described by color histograms.

[0035] This step is performed when multiple source sensors monitor the sea area. Specifically, the system first starts multiple sensors to monitor the designated sea area in real time. For the data collected by each sensor, the system extracts the position coordinates (x, y) of the ship target, calculates the rectangular box area a of the ship target, and constructs a color histogram h to describe the appearance characteristics of the ship. The system organizes and stores these feature data to form a complete ship monitoring data set.

[0036] In some embodiments, multi-source sensor data acquisition can be achieved in a variety of ways: optionally, using fixed radar scanning to obtain ship echo signals, extracting ship position information through signal processing, analyzing echo intensity to determine target area, and recording echo characteristics; obtaining ship position messages based on AIS receivers, parsing messages to extract position coordinates, and calculating area based on ship size information; using optoelectronic equipment to collect ship images, and performing image segmentation to obtain target areas. Optionally, an airborne radar is used to perform a round-trip scan of the sea area to obtain dynamic information of ship targets; at the same time, an optoelectronic pod is turned on for tracking and shooting to extract ship image sequences; and ship AIS information is received as auxiliary data. It is understandable that other types of sensors or different combinations can also be used to achieve data acquisition, which is not limited here.

[0037] S102, generating an initial ship track of each sensor according to the ship monitoring data, calculating the detection point similarity of the initial ship track and associating them to obtain an associated track pair, wherein the detection point similarity includes speed similarity, area similarity and color similarity.

[0038] Among them, ship monitoring data refers to the original position, speed and other information data obtained by each sensor when detecting the ship target. The initial ship trajectory refers to the track point sequence formed according to the ship monitoring data obtained by a single sensor. The detection point similarity is used to indicate the similarity between the ship trajectory points obtained by different sensors. The associated trajectory pair refers to the trajectory point pairs of different sensors that are determined to be the same ship target through similarity calculation.

[0039] Specifically, this step is performed after obtaining the ship monitoring data, and is used to generate the initial trajectory and perform similarity judgment. First, based on the ship monitoring data collected by each sensor, the initial trajectory sequence of each sensor is formed through time series association. Then, the speed similarity, area similarity and color similarity between the trajectory points of different sensors are calculated, where the speed similarity is calculated based on the Euclidean distance of the speed vector, the area similarity is calculated based on the area ratio of the target circumscribed rectangle, and the color similarity is calculated based on the Bhattacharyya coefficient of the color histogram. Finally, according to the similarity threshold, the trajectory points with high similarity are paired to form associated trajectory pairs.

[0040] In some embodiments, initial trajectory generation and similarity calculation can be achieved in a variety of ways: optionally, through a sliding time window method, the correlation between track points is calculated in each window according to the temporal continuity of the ship monitoring data, the points with high correlation are connected to form an initial trajectory segment, and then the trajectory segments are spliced ​​to obtain a complete initial trajectory; the similarity feature values ​​are calculated for different sensor trajectory points in pairs, and the similarity is comprehensively evaluated in combination with a multi-feature fusion method, and the point pair with the highest similarity is selected as the associated trajectory pair. Optionally, a target tracking filter is used to filter the ship monitoring data to obtain a smooth trajectory; the speed, area and color feature vectors are calculated based on the trajectory segment feature extraction technology; the cosine similarity of the feature vector is used to calculate the similarity of the trajectory points, and the trajectory is associated by setting a similarity threshold. It can be understood that other initial trajectory generation and similarity calculation methods can also be used, which are not limited here.

[0041] S103, performing outlier identification and trajectory optimization on the associated trajectory pair based on a similarity energy function to obtain an optimized ship trajectory, wherein the similarity energy function includes a dynamics option, a persistence option, a fusion interval option, a color similarity option, and an adjustment option.

[0042] Among them, the similarity energy function represents the objective function used to evaluate the trajectory optimization effect. Outlier recognition refers to detecting and removing abnormal track points in the trajectory. Trajectory optimization refers to smoothing and completing the trajectory. The optimized ship trajectory refers to the high-quality trajectory obtained after outlier removal and optimization.

[0043] Specifically, this step is performed after obtaining the associated trajectory pair to optimize the trajectory quality. First, a similarity energy function is constructed that includes dynamics options, persistence options, fusion interval options, color similarity options, and adjustment options. The dynamics option describes the change in the motion state of the track, the persistence option describes the continuity of the trajectory, the fusion interval option describes the interval between trajectory segments, the color similarity option describes the change in the color characteristics of the trajectory points, and the adjustment option is used to control the number of trajectories. Then, based on the energy function, the trajectory is identified and optimized, including removing abnormal points that do not conform to the law of motion, completing missing trajectory segments, and smoothing trajectory curves.

[0044] In some embodiments, trajectory optimization can be achieved in a variety of ways: optionally, using a sliding window method to calculate the motion characteristics of local trajectory segments, identifying abnormal track points based on Mahalanobis distance; using cubic spline interpolation to complete missing trajectory segments; using Kalman filtering to smooth the trajectory; and finally obtaining the optimized trajectory by minimizing the energy function. Optionally, a trajectory anomaly detector is established based on a Bayesian probability model, and a multivariate statistical analysis method is combined to identify abnormal points; a dynamic programming algorithm is used to optimize the trajectory in segments, and the least squares method is used to fit the trajectory curve in each segment; and the trajectory morphology is adjusted by iteratively optimizing the energy function. It is understandable that other trajectory optimization methods can also be used, which are not limited here.

[0045] S104, performing a similarity test by calculating the state difference between the optimized ship trajectories, determining the optimized ship trajectories with similarity exceeding a preset threshold as the same target and associating them to form a multi-source trajectory group.

[0046] Among them, the state difference indicates the degree of difference in the motion state parameters between different optimized ship trajectories. Similarity test refers to evaluating the similarity between trajectories by calculating the trajectory state difference. The preset threshold is used to represent the similarity standard for determining that the trajectories belong to the same target. The multi-source trajectory group represents a set of optimized trajectories from different sensors that are determined to be the same ship target.

[0047] Specifically, this step is performed after the optimized ship trajectory is obtained to determine whether the trajectories from different sensors correspond to the same ship target. First, the state differences between different optimized trajectories are calculated, including differences in motion parameters such as position difference, speed difference, and heading difference. Then, a likelihood ratio statistic is constructed based on the state difference to perform a similarity test, and the similarity between the trajectories is evaluated by the Bayesian criterion. When the similarity exceeds the preset threshold, it is determined that these trajectories come from the same ship target, and they are associated together to form a multi-source trajectory group, laying the foundation for subsequent trajectory fusion.

[0048] In some embodiments, trajectory similarity testing and association can be achieved in a variety of ways: Optionally, first extract the feature vector of the trajectory, including average speed, turning angle, navigation distance, etc.; then calculate the Mahalanobis distance of the feature vector as a state difference measure; finally, use the hypothesis test method to determine the significance of the difference, and determine the trajectories with a significance level less than a threshold as the same target and associate them. Optionally, construct a trajectory similarity model based on joint probability data association; calculate the state transition probability matrix between trajectory points; perform trajectory association judgment based on a sequential probability ratio test, and dynamically update the association results using a sliding time window. It is understandable that other trajectory similarity testing and association methods can also be used, which are not limited here.

[0049] S105, calculating a dynamic fusion weight according to a standard deviation of a measurement error of each optimized ship trajectory in the multi-source trajectory group relative to its average trajectory, and performing weighted fusion to obtain a ship target trajectory.

[0050] Among them, the average trajectory represents the average state sequence of all trajectories in the multi-source trajectory group. The standard deviation of the measurement error is used to indicate the degree of deviation of a single trajectory from the average trajectory. The dynamic fusion weight refers to the trajectory fusion coefficient dynamically adjusted according to the measurement error. The ship target trajectory represents the final trajectory result obtained after the weighted fusion of multi-source trajectories.

[0051] Specifically, this step is performed after the multi-source trajectory group is formed, and is used to fuse multiple trajectories into a single high-precision trajectory. First, the average value of each trajectory point in the multi-source trajectory group is calculated to obtain the average trajectory as a reference. Then, the standard deviation of the measurement error of each trajectory relative to the average trajectory, such as position, heading, speed, etc., is calculated to evaluate the measurement accuracy of the trajectory. Based on the standard deviation of the measurement error, the minimum variance criterion is used to calculate the dynamic fusion weight of each trajectory, so that the trajectory with high measurement accuracy obtains a larger weight. Finally, each trajectory is weighted averaged according to the corresponding weight to obtain the fused ship target trajectory.

[0052] In some embodiments, trajectory fusion can be achieved in a variety of ways: Optionally, first time-align and spatially normalize the multi-source trajectories; then calculate the root mean square error of the trajectory within the sliding time window as a measurement accuracy indicator; calculate the normalized fusion weight based on the inverse of the measurement accuracy; and finally use weighted Kalman filtering to estimate and fuse the trajectory state. Optionally, establish a trajectory accuracy assessment model based on covariance cross-validation; dynamically adjust the fusion weight using an adaptive weighting algorithm; optimize the fused trajectory in combination with trajectory prediction and smoothing technology to improve the smoothness and continuity of the trajectory. It is understandable that other trajectory fusion methods can also be used, which are not limited here.

[0053] The following is a more detailed description of the process of the method provided by this implementation. Figure 2 , is another flow chart of the ship target identification and tracking method based on multi-source data fusion in an embodiment of the present application.

[0054] S201. Collect ship monitoring data of the same sea area from multiple source sensors, where the ship monitoring data includes ship position coordinates, area features, and color features.

[0055] Multi-source sensors refer to different types of monitoring methods that can obtain ship information in the same sea area, such as radar, AIS (Automatic Identification System), optoelectronic equipment, etc. These sensors each have unique monitoring principles and functions, and can obtain relevant information about ships from different dimensions. Ship monitoring data is a collection of feature information about ship targets collected by sensors, which is crucial for ship monitoring. Among them, the ship position coordinates are used to accurately represent the position of the ship on a two-dimensional plane, usually determined by longitude and latitude; the area feature refers to the size of the area occupied by the ship target in the image; the color feature is a description of the appearance color characteristics of the ship target through a color histogram.

[0056] When the ship target identification and tracking system is running, the system starts multi-source sensors to synchronously monitor ships in the same sea area. Taking radar as an example, radar can accurately obtain the ship's position coordinates after signal processing by emitting electromagnetic waves and receiving reflected waves. The AIS system parses the ship's position coordinates by receiving messages containing information such as location sent by the ship itself. For the acquisition of area features, if optoelectronic equipment is used to collect ship images, image segmentation technology can be used to separate the ship target from the background, and then calculate its occupied area in the image. In terms of color feature acquisition, based on the images collected by optoelectronic equipment, the color analysis of the ship target area is also performed, the distribution of different colors in the area is counted, and a color histogram is constructed to describe the color features. The system integrates and stores the ship's position coordinates, area features, and color feature data obtained from different sensors to form a complete ship monitoring data set, providing a key data basis for subsequent ship trajectory generation, analysis, and other operations.

[0057] S202: When no detection point associated with the target detection point can be found in the previous frame image, a new trajectory segment is generated with the detection point as the trajectory starting point.

[0058] The target detection point is a feature point related to the ship detected in the image, which represents the state of the ship at a certain moment. The previous frame image is the previous image in the time series that is immediately adjacent to the current processed image, and is used for comparative analysis with the current image to determine the motion trajectory of the ship. The associated detection points are detection points that represent the same ship target and have similar characteristics and spatiotemporal continuity in different frames. The starting point is the detection point that serves as the starting position of the new trajectory segment, and the new segment is the initial part of a ship trajectory constructed based on this specific detection point.

[0059] When processing an image sequence, the system analyzes the detection points frame by frame. For each target detection point, the system compares it with all detection points in the previous frame according to preset rules. These rules are formulated based on the similarity calculation of the detection point's position, speed, area, and color. For example, the position distance threshold, speed change threshold, area difference threshold, and color histogram similarity threshold are set. When calculating position similarity, the Euclidean distance formula can be used to measure the position difference between two detection points on a two-dimensional plane; the speed similarity calculation can refer to the position change and time interval of the detection points in the previous and next frames to obtain the speed vector, and then calculate the speed difference; the area similarity is measured by calculating the ratio of the area of ​​the ship represented by the two detection points; the color similarity is calculated based on the color histogram similarity algorithm. If after these similarity calculations, no detection point matching the current target detection point is found in the previous frame, that is, all similarity indicators of all detection points and the current target detection point do not reach the preset threshold, then the system determines the target detection point as the starting point of a new trajectory segment. This is most likely because the detection point represents a newly appeared ship target, or it was not correctly detected and associated in the previous frame due to previous detection errors. After that, the system takes this as the starting point and begins to record subsequent related detection points, thereby generating a new ship track segment, laying the foundation for subsequent accurate tracking of ship targets.

[0060] S203: When no detection point associated with the target detection point can be found in the next frame image, the target detection point is used as the end point of the current trajectory segment.

[0061] The next frame is the next image immediately following the current processed image in time sequence, and is used to analyze the subsequent state of the ship target in conjunction with the current image. The current track segment is a ship track segment that is being constructed or has been partially constructed, which records part of the ship's motion track information over a period of time. The end point refers to the position point where the current track segment ends, that is, there is no subsequent detection point associated with this detection point, marking the end of this track segment.

[0062] In the process of constructing the ship's trajectory, when a target detection point is analyzed, the system will look for detection points associated with it in the next frame of the image where it is located. The basis for association judgment is similar to the previous one, mainly based on the similarity calculation of multiple features such as position, speed, area and color. For example, when calculating the speed similarity between the current detection point and each detection point in the next frame, the absolute value of the speed vector difference is calculated according to the position change and time interval of the detection points in the previous and next frames. If the value exceeds the preset speed difference threshold, it is considered that the speed similarity does not meet the requirements and the two are not associated; the area similarity can be calculated by calculating the relative difference in the area of ​​the ship represented by the two detection points. If it exceeds the preset area difference threshold, it is considered to be unrelated; for color similarity, the similarity is calculated using algorithms such as the Bhattacharyya coefficient of the color histogram. If it is lower than the set color similarity threshold, it is determined to be unrelated. When it is determined that no detection point matching the target detection point can be found in the next frame after comprehensive similarity calculation and judgment, the system sets the target detection point as the end point of the trajectory segment currently being constructed. This may be due to the fact that the ship has left the monitoring area or is blocked by other objects, resulting in the inability to detect the ship target in subsequent images, or an error has occurred during the detection process, making it impossible to find the detection point associated with it in subsequent images, thereby ending the recording of the current trajectory segment.

[0063] S204, determining whether the number of frames in which the track appears continuously is less than a preset threshold, and if so, determining it as a track fragment.

[0064] The trajectory is formed by a series of detection points connected in chronological order, which is used to characterize the movement path of the ship over a period of time. The number of consecutive frames refers to the number of frames in which the detection points contained in the trajectory appear in sequence in the image sequence, which reflects the continuity of the trajectory in the time dimension. The preset threshold is a fixed frame number value pre-set before the system runs, which serves as a standard for judging whether the trajectory is complete or reliable. Track fragments refer to those parts of the trajectory whose consecutive frame numbers do not reach the preset threshold. Such trajectories may be caused by detection errors, temporary occlusion of the target, etc., and their integrity and reliability are relatively low.

[0065] When the system processes the ship monitoring data to generate the initial ship trajectory, each trajectory will be traversed and analyzed. The system checks the occurrence of detection points on each trajectory in the time sequence of the image. Each time a frame of image is processed, if the detection point belonging to a certain trajectory is detected, the counter of the continuous appearance frame number of the trajectory is increased by 1; if it is not detected, the counter is reset to 0. For example, the preset threshold is set to 5 frames. When the system processes a certain trajectory, the detection point of the trajectory is detected from the 1st frame to the 3rd frame, but not detected in the 4th frame. At this time, the number of consecutive frames of the trajectory is 3 frames, which is less than the preset threshold of 5 frames. The system compares the calculated number of consecutive appearance frames with the preset threshold. If it is less than the preset threshold, the trajectory is determined to be a trajectory fragment. This judgment operation is crucial to the subsequent quality assurance of the initial ship trajectory, because the trajectory fragments may interfere with the subsequent trajectory analysis and processing. By identifying and marking these trajectory fragments, the system can perform special treatment on them to improve the accuracy and reliability of the entire ship target recognition and tracking system.

[0066] S205, calculating the energy function value after the trajectory fragment is fused with other trajectory segments, and retaining the trajectory fragment when the energy function value decreases, otherwise removing it from the trajectory space to obtain the initial ship trajectory.

[0067] Track fragments refer to the part of the track that is judged to have too few consecutive frames in step S204. Other track segments are relatively complete or reliable track segments in the entire track space, except for the track fragments. The energy function value is a quantitative indicator that comprehensively considers multiple factors and is used to evaluate the rationality of track configuration. It consists of multiple parts such as dynamic options, continuity options, fusion interval options, color similarity options and adjustment options. Its numerical value can intuitively reflect the overall quality and rationality of the track. The track space is a virtual space used by the system to store and manage all detected ship track information, which includes various types of tracks and their related attributes.

[0068] After identifying the trajectory fragments, the system will try to merge each trajectory fragment with other trajectory segments in the trajectory space. Specifically, the system merges the trajectory fragment with other trajectory segments one by one, and then calculates the energy function value after fusion according to the energy function formula. The energy function formula is:

[0069] After identifying the trajectory fragments, the system will try to merge each trajectory fragment with other trajectory segments in the trajectory space. Specifically, the system merges the trajectory fragment with other trajectory segments one by one, and then calculates the energy function value after fusion according to the energy function formula. The energy function formula is Among them, the dynamic option E_{dyn}^{i}=α∑_{t=1}^{F}ξ‖v_{i}^{t}-v_{i}^{t+1}‖^2 is mainly used to constrain the range of change of the target's motion speed. When the speed changes too much, a larger penalty will be given to ensure the smoothness of the trajectory curve and prevent the target from undergoing identity transformation when crossing.

[0070] The persistence option E_{per}^{i}=β(d(Γ_{i}(e_{i}))+d(Γ_{i}(s_{i})))+χ(e_{i}-s_{i})^{-1} takes into account the rationality of the target appearing and disappearing in the image, and encourages the trajectories to merge at the break points to form longer trajectories.

[0071] The fusion interval option E_{int}^{i}=λ∑_{k}M_{k} avoids the fusion trajectory interval being too large by penalizing the situation where the fusion trajectory does not pass through the detection point for a long time.

[0072] The color similarity option E_{col}^{i}=ψ(1-P(Γ_{i}(e_{i}), Γ_{j}(s_{j}))) ensures that the similarity of color features between detection points in the trajectory segment varies little, which helps solve the identity conversion problem when multiple moving targets are close to each other.

[0073] The adjustment option E_{reg}^{i}=ϑN is used to optimize the existing trajectory space when generating the final trajectory so that the number of generated trajectories is as small as possible.

[0074] If the energy function value after fusion decreases compared to before fusion, it means that the fused trajectory is more reasonable and more in line with the actual situation, and the system will retain the trajectory fragment; conversely, if the energy function value does not decrease, the system will remove the trajectory fragment from the trajectory space. After this processing of all trajectory fragments, the system can obtain a relatively accurate and reliable initial ship trajectory, providing a better data basis for subsequent trajectory analysis, association and fusion operations.

[0075] S206: Calculate the speed similarity of adjacent detection points according to the position information of the detection points in the initial ship trajectory.

[0076] The initial ship trajectory is initially generated by the system based on the ship monitoring data. It contains a series of detection points of the ship's motion trajectory. These detection points record the position of the ship at different times and other related information. The detection point is the mark of the ship's position in the image. Each detection point contains characteristic information such as the position coordinates of the ship at that moment. Adjacent detection points refer to two detection points that are closely connected in time order in the initial ship trajectory. Speed ​​similarity is used to measure the similarity of the ship's motion speed represented by the two adjacent detection points. It is one of the important bases for judging whether adjacent detection points belong to the same trajectory segment.

[0077] After obtaining the initial ship trajectory, the system will calculate the speed similarity for each pair of adjacent detection points on the trajectory in order to further analyze and process the trajectory data. For the adjacent detection points Di(t) and Dj(t+1) on the initial ship trajectory, the speed similarity calculation formula is:

[0078] In this formula, (vxi, vyi) and (vxj, vyj) represent the speeds of the two detection points in the X-axis direction and the Y-axis direction respectively. The speed is calculated by dividing the position change of the detection point in two adjacent frames by the time interval. For example, if the position vector of the detection point in frame t is pi(t), and the position vector in frame t+1 is pi(t+1), then the speed vector vi(t) of the detection point at time t is vi(t)=pi(t+1)-pi(t). σvx² and σvy² represent the variance of the speed in the two directions in the current trajectory segment. Taking the calculation of the speed variance in the X-axis direction as an example, first, it is necessary to find the difference between the speed of each detection point in the trajectory segment in the X-axis direction and the average speed, square these differences respectively, sum them, and then divide them by the number of detection points to get σvx². The calculation method of the speed variance in the Y-axis direction is the same. For example, in an initial ship trajectory, there are adjacent detection points A and B, A's speed in the X-axis direction is vxA, and its speed in the Y-axis direction is vyA, B's speed in the X-axis direction is vxB, and its speed in the Y-axis direction is vyB, and the speed variance of this trajectory segment in the X-axis direction is σvx², and the speed variance in the Y-axis direction is σvy². Substituting these values ​​into the speed similarity calculation formula, the speed similarity of adjacent detection points A and B can be obtained.

[0079] S207, calculating the area similarity of adjacent detection points according to the rectangular frame of the detection points in the initial ship trajectory.

[0080] When processing the initial ship trajectory, the system uses the rectangular box information of each pair of adjacent detection points on the trajectory to calculate the area similarity. First, the system obtains the rectangular boxes corresponding to the adjacent detection points. For the adjacent detection point Di(t), the system obtains its area by calculating the product of the length and width of the rectangular box, which are recorded as si and sj respectively.

[0081] The area similarity calculation formula is:

[0082] Among them, σs² represents the variance of the rectangular box area of ​​all detection points in the current trajectory segment. When calculating the variance, the system first calculates the average value s̄ of the rectangular box area of ​​all detection points in the trajectory segment. For example, there are 5 detection points in an initial ship trajectory segment, and their rectangular box areas are s1, s2, s3, s4, and s5, respectively, then s̄=(s1+s2+s3+s4+s5) / 5. Next, calculate the square of the difference between the rectangular box area of ​​each detection point and the average value, that is, (s1-s̄)², (s2-s̄)², (s3-s̄)², (s4-s̄)², (s5-s̄)², sum these square values, and then divide them by the number of detection points 5 to get σs². Substituting the area values ​​corresponding to Di(t) and Dj(t+1) and the calculated σs² into the formula, the area similarity of the two adjacent detection points can be obtained. The system will perform this calculation process for each pair of adjacent detection points on the initial ship trajectory, thereby obtaining a series of area similarity data. This data can be used to subsequently determine the relevance of adjacent detection points and provide support for accurately constructing the ship trajectory.

[0083] S208: Calculate the color similarity of adjacent detection points according to the color features of the detection points in the initial ship trajectory.

[0084] When calculating the color similarity of adjacent detection points, the system uses a color histogram to quantify the color characteristics of the detection points. For the adjacent detection points Di(t) and Dj(t+1) in the initial ship trajectory, the color similarity calculation formula is:

[0085] Among them, N represents the level of the color histogram, which determines the degree of refinement of the color feature description. For example, if N=32, it means that the color space is divided into 32 levels to describe the color features. q(d1)n and p(d2)n respectively represent the proportion of the nth level in the color histogram of the two detection points. The system first constructs a color histogram for each detection point, counts the distribution of different colors in the target area of ​​the ship, and then obtains the proportion of each color level in the histogram. For example, for the detection point Di(t), after statistics, the proportion of the first level color in its color histogram is q(d1)1, the proportion of the second level color is q(d1)2, and so on; for the detection point Dj(t+1), there are also corresponding color proportions of each level p(d2)n. Then, substitute the corresponding proportions of each level into the above formula for calculation. For example, if the similarity contribution of the first level color is calculated as sqrt(q(d1)1p(d2)1), the similarity contributions of the 32 levels of color are added together, that is, sqrt(q(d1)np(d2)n) is summed from n=1 to 32, and the result is the color similarity of the two adjacent detection points. The system calculates the color similarity for each pair of adjacent detection points on the initial ship trajectory according to this method. The generated color similarity data helps to more accurately judge the relationship between adjacent detection points and further improve the accuracy of ship trajectory construction.

[0086] S209 , associating adjacent detection points based on a weighted combination of the speed similarity, the area similarity, and the color similarity to obtain an associated trajectory pair, wherein the detection point similarity includes speed similarity, area similarity, and color similarity.

[0087] After obtaining the speed similarity, area similarity and color similarity data of adjacent detection points in the initial ship trajectory, the system will perform a weighted combination to determine whether the adjacent detection points belong to the same trajectory, and then obtain the associated trajectory pair. Assuming that the speed similarity is Avel, the area similarity is Aare, and the color similarity is Acol, they are assigned weights w1, w2, and w3 respectively (w1+w2+w3=1, and the weight setting can be determined according to actual conditions and experience. For example, in a complex sea environment, the ship speed varies greatly, and the speed similarity has a greater impact on the accuracy of trajectory judgment, so w1=0.5 can be assigned; if the area feature of the ship in the image is less disturbed, the reliability of the area similarity is higher, and w2=0.3 can be set; the color similarity has a relatively small impact, w3=0.2), and the comprehensive similarity calculation formula is P(Di(t), Dj(t+1))=w1Avel+w2Aare+w3*Acol. The system calculates this comprehensive similarity index for each pair of adjacent detection points on the initial ship trajectory. For example, for a pair of adjacent detection points, the calculated speed similarity Avel=0.7, area similarity Aare=0.6, and color similarity Acol=0.5. According to the above weight settings, the comprehensive similarity P(Di(t), Dj(t+1))=0.5×0.7+0.3×0.6+0.2×0.5=0.63. The system will compare this comprehensive similarity with the preset threshold (such as 0.6). If the comprehensive similarity is greater than the threshold, it is determined that the pair of adjacent detection points belong to the same trajectory and they are recorded as an associated trajectory pair; if it is less than the threshold, it is considered that they do not belong to the same trajectory. By performing such calculations and judgments on all adjacent detection points on the initial ship trajectory, the system obtains a series of associated trajectory pairs, which lay the foundation for the subsequent generation of more accurate ship trajectories.

[0088] S210, using a similarity energy function to calculate the similarity of adjacent track points in the initial ship track, the similarity energy function comprising a dynamics option, a persistence option, a fusion interval option, a color similarity option and an adjustment option.

[0089] When the system processes the initial ship trajectory, it uses the similarity energy function to calculate the similarity of each pair of adjacent trajectory points in the trajectory. The formula of the similarity energy function is:

[0090] Among them, the dynamics option Edyn(i)=α∑(t=1 to F)ξ||vi(t)-vi(t+1)||², vi(t) represents the velocity vector of target i in the tth frame, ξ will take values ​​according to the magnitude of the velocity change (when ||vi(t)-vi(t+1)||²>5, ξ=10; otherwise, ξ=1), α is a constant, and this option is used to penalize trajectories with sudden velocity changes to make the trajectory smoother. For example, if the velocity of an adjacent trajectory point changes greatly between two frames, resulting in ||vi(t)-vi(t+1)||²=8, then the value of this part of the dynamics option is relatively large, which has a greater impact on the overall similarity energy function value. The persistence option Eper(i)=β(d(Γi(ei))+d(Γi(si)))+χ(ei-si)^(-1), d(Γi(ei)) and d(Γi(si)) represent the distance from the head node or tail node of the trajectory to the edge of the image, ei and si are the frames where the end and start of the trajectory are located, respectively. This option promotes the fusion of trajectories at the break and generates longer trajectories. The fusion interval option Eint(i)=λ∑Mk, Mk represents the total number of frames that do not pass through any detection points for a long time in the fused trajectory, λ is a constant, and the validity of the trajectory is guaranteed by penalizing trajectories without detection points for a long time. The color similarity option Ecol(i)=ψ(1-P(Γi(ei), Γj(sj))), P(Γi(ei), Γj(sj)) represents the similarity of the color histogram between the end point of trajectory Γi and the start point of trajectory Γj, ψ is a constant. This option is combined with the dynamics option to solve the identity transformation problem when multiple targets are close. Adjust the option Ereg(i)=ϑN, where N is the total number of trajectory segments in the current trajectory space and ϑ is a constant used to adjust the number of trajectories generated in the end. The system adds the results of these options and the resulting Etal value is the similarity of adjacent trajectory points. By performing such calculations on all adjacent trajectory points in the initial ship trajectory, the system can evaluate the quality of the trajectory and provide a basis for subsequent outlier identification and trajectory optimization.

[0091] S211. Based on the speed drift coefficient and the disturbance variance factor, a linear model is constructed to calculate the mutation probability of the trajectory point, a sliding time window is set, and the speed drift coefficient and the disturbance variance factor of the trajectory point within the sliding time window are calculated.

[0092] The system first sets a sliding time window. The window length can be set according to the actual situation, for example, set to l. When processing the initial ship trajectory, the system moves this sliding time window along the trajectory in chronological order. For the trajectory point data in the window, the system calculates the speed drift coefficient and the disturbance variance factor. Assuming that the ship sails along a large circle for a period of time, the estimated value of the ship's sailing speed in the time interval (recorded as the position estimation speed) can be calculated from the position of the ship at the adjacent moments before and after, and the sailing speed in the time interval estimated from the actual sailing speed of the ship (recorded as the actual speed). A linear model is established in the window. Assuming that the position estimation speed is v̂ and the actual speed is v, the model is v̂=β0+β1v+ε, where ε is the model error, which obeys the normal distribution, and Il×l is the unit matrix of l×l, indicating that the model errors of each point are independent of each other. The coefficient β0 is the speed drift coefficient, which contains the speed drift information affected by the local ocean current velocity and sea surface wind speed; the variance factor τ^(-1) describes the information that the data is disturbed by the positioning error, that is, the disturbance variance factor. For example, in a certain trajectory, the sliding time window contains 5 trajectory points. By processing the position and velocity data of these 5 points and using fitting methods such as the least squares method, the velocity drift coefficient β0 and the disturbance variance factor τ^(-1) can be obtained. The system continuously moves the sliding time window to calculate the velocity drift coefficient and disturbance variance factor in each window, providing data support for the subsequent construction of a linear model to calculate the mutation probability of the trajectory points, thereby identifying abnormal points in the trajectory and improving the accuracy and reliability of the trajectory.

[0093] S212: construct a linear model of the trajectory point position based on the velocity drift coefficient and the disturbance variance factor, and calculate the mutation probability of the trajectory point according to the deviation between the trajectory point and the position predicted by the linear model.

[0094] The system first constructs a linear model of the position of the track point based on the previously calculated speed drift coefficient and disturbance variance factor. Assume that in the two-dimensional plane, the position coordinates of the track point at time t are (xt, yt), the speed drift coefficients are expressed as β0x and β0y in the x and y directions, the actual speed components of the ship in the x and y directions are vx and vy, and the disturbance variance factors are τx^(-1) and τy^(-1) in the x and y directions, respectively. The linear model can be expressed as: xt+1=xt+β0x+vx∆t+εx, yt+1=yt+β0y+vy∆t+εy; where ∆t is the time interval, εx and εy are error terms, which obey a normal distribution with a mean of 0 and variances of τx^(-1) and τy^(-1), respectively.

[0095] The system predicts the position of each trajectory point at the subsequent moment based on this linear model and obtains the predicted position (xpred, ypred). Then, the system calculates the deviation between the actual position (xactual, yactual) of the trajectory point and the predicted position. The deviation can be calculated by the Euclidean distance, that is, d=√((xactual-xpred)²+(yactual-ypred)²).

[0096] Next, the system calculates the mutation probability of the trajectory point based on the deviation. Assume that the deviation d follows the normal distribution N(0, σ²), where σ² is related to the disturbance variance factor. The mutation probability can be obtained by calculating the probability that the deviation d exceeds a certain threshold. For example, if a threshold T is set, the mutation probability P can be expressed as P=1-Φ((Td) / σ), where Φ is the cumulative distribution function of the standard normal distribution.

[0097] The system performs such calculations for each track point in the initial ship track, and determines whether the track point is an abnormal point by comparing the mutation probability with the preset threshold. If the mutation probability is greater than the preset threshold, it is considered that the track point has undergone a mutation, which may be caused by measurement errors, sudden turns of the ship, etc.; if it is less than the preset threshold, it is considered that the track point is in normal motion mode. In this way, the system can identify abnormal points in the track and provide a basis for subsequent track correction and optimization.

[0098] S213, removing the abnormal points whose mutation probability exceeds a preset probability threshold, and obtaining the trajectory segments after removing the abnormal points.

[0099] After calculating the mutation probability of each track point in the initial ship track, the system will compare the mutation probability of each track point with the preset probability threshold. The preset probability threshold is pre-set based on actual conditions and experience, for example, set to 0.7. For each track point in the track, the system accurately obtains its mutation probability value. Assume that in a certain ship track, the mutation probability of a track point A is calculated to be 0.8. Since 0.8 is greater than the preset probability threshold of 0.7, the system determines that the track point A is an abnormal point. The system will perform such a comparison operation on all track points in the entire track. Once a track point is determined to be an abnormal point, the system will remove the abnormal point from the track. For example, in a track containing 100 track points, after comparison, it is found that the mutation probability of 5 track points exceeds the preset probability threshold, and the system will remove these 5 abnormal points from the track. After such a removal operation, the remaining track points constitute the track segment after the abnormal points are removed. This process helps to improve the quality of the track, making the subsequent analysis and processing based on the track more accurate and reliable.

[0100] S214. A moving average method is used to supplement the missing parts in the trajectory segment after the abnormal points are removed. When the time interval between the trajectory segments exceeds a preset interval threshold, a two-way search method is used to supplement the missing parts to obtain the optimized ship trajectory. The similarity energy function includes a dynamics option, a continuity option, a fusion interval option, a color similarity option, and an adjustment option.

[0101] The system first checks the trajectory segments after removing the abnormal points to determine the missing parts. For these missing parts, the system uses the moving average method to supplement. Specifically, the system sets a time window for each missing point, for example, the window size is 5 trajectory points. For a missing point, the system obtains the position information of 2 known trajectory points before and after it (a total of 4). Assuming that the position coordinates of these 4 trajectory points are (x1, y1), (x2, y2), (x3, y3), (x4, y4), the supplementary position coordinates of the missing point are ((x1+x2+x3+x4) / 4, (y1+y2+y3+y4) / 4). The system supplements all missing points in this way. During the supplementation process, the system checks the time interval between trajectory segments. The system pre-sets an interval threshold, such as 3 minutes. When the system detects that the time interval between two trajectory segments exceeds this preset interval threshold, it will use a two-way search method to supplement. The system starts from the endpoints of the two trajectory segments and searches towards the middle. During the search process, the system will combine the similarity energy function (including dynamics option, persistence option, fusion interval option, color similarity option and adjustment option) to evaluate possible supplementary track points. For example, the dynamics option will constrain the range of change of the target's motion speed. The system will find track points with reasonable speed changes according to this option for supplementation. Through bidirectional search, the system finds suitable track points to connect the two track segments. After processing by the moving average method and the bidirectional search method, the system obtains the optimized ship track, which is more complete and accurate and can better reflect the actual navigation conditions of the ship.

[0102] S215, calculating the state estimation difference of the optimized ship trajectory of different sensors, where the state estimation difference includes position, heading and speed information.

[0103] In order to optimize the ship's trajectory, the system processes the state information obtained by different sensors and calculates the state estimation difference between them. For position information, assume that there are two sensors A and B. At a certain time t, the position coordinates of the ship measured by sensor A are (xA, yA), and the position coordinates measured by sensor B are (xB, yB). The system obtains the position state estimation difference by calculating the distance between the two points. The calculation formula is dpos=√((xA-xB)²+(yA-yB)²). The system performs such calculations at each moment in the optimized ship trajectory to obtain a sequence of position state estimation differences on the entire trajectory. For heading information, at the same time t, the ship's heading measured by sensor A is θA, and the heading measured by sensor B is θB. The system calculates their angle difference. In order to avoid the circular problem of angle calculation, the formula dheading=min(|θA-θB|, 360-|θA-θB|) is used to obtain the heading state estimation difference. The system calculates the heading information at each moment to obtain a sequence of heading state estimation differences. For speed information, at time t, the ship speed measured by sensor A is vA, and the speed measured by sensor B is vB. The system directly calculates their difference dspeed=|vA-vB|, and calculates it at each moment on the entire optimized ship trajectory to obtain a speed state estimation difference sequence. The system obtains a complete state estimation difference by performing the above calculations on the position, heading, and speed information of different sensors at each moment on the optimized ship trajectory. These state estimation differences can be used to evaluate the measurement accuracy and reliability of different sensors, providing an important basis for subsequent sensor fusion and data processing.

[0104] S216 , performing a likelihood ratio test based on the state estimation difference through a joint probability density function based on uncertain distribution to determine whether different sensor trajectories belong to the same target, and obtaining an associated multi-source trajectory group.

[0105] After the system obtains the state estimation difference of different sensors for the optimized ship trajectory, a likelihood ratio test is performed using a joint probability density function based on uncertain distribution. Suppose the state estimation difference of the sensor for targets i and j at time k is tijk(k)={tij(l)}; l=1, 2, ⋯, k, where i and j represent the trajectories of different sensors, and tij(l) is the state estimation difference at time l. Suppose H0 and H1 are two events, H0 indicates that Xi1(l|l) and Xj2(l|l) are track data of the same target, and H1 indicates that they do not belong to the track data of the same target. Its joint probability density function is: , where tij0(0)=Xi1(0|0)-Xj2(0|0) is the prior information, obtained by weighted track association. The joint probability density can also be written as ,in , Pi1(l|l) is the estimated error covariance of sensor 1 for target i at time l, and Pj2(l|l) is the estimated error covariance of sensor 2 for target j at time l. Perform a likelihood ratio test and calculate , the corresponding log-likelihood ratio is . Define the test statistic . The system compares λij(k) with the significance level α. If λij(k) is less than α, the null hypothesis H0 is accepted, that is, the trajectories of different sensors belong to the same target; if λij(k) is greater than or equal to α, the alternative hypothesis H1 is accepted, that is, they do not belong to the same target. For example, if the significance level α=0.05 is set, for the trajectories of two sensors, λij(k)=0.03 is calculated, which is less than α, then the system determines that the trajectories corresponding to the two sensors belong to the same target. The system performs such a likelihood ratio test on all pairs of trajectories from different sensors, and combines the trajectories belonging to the same target to obtain the associated multi-source trajectory group, providing an accurate data basis for subsequent trajectory fusion.

[0106] S217: Calculate the standard deviation of the measurement error of each optimized ship trajectory in the multi-source trajectory group relative to the average trajectory of the multi-source trajectory group.

[0107] After obtaining the associated multi-source trajectory group, the system starts to calculate the standard deviation of the measurement error of each optimized ship trajectory relative to the average trajectory. First, the average trajectory of the multi-source trajectory group is calculated. For the position information, assuming that there are M trajectories in the multi-source trajectory group, at a certain time t, the position coordinates of the i-th trajectory are (xi, yi), then the position coordinates of the average trajectory at this time are (x̄, ȳ) = (1 / M∑i = 1Mxi, 1 / M∑i = 1Myi). For the heading information and speed information, the calculation method is similar, and the heading and speed of each trajectory at the same time are averaged. Next, for each optimized ship trajectory in the multi-source trajectory group, the measurement error with the average trajectory is calculated. Taking the position information as an example, at time t, the position coordinates of the j-th optimized ship trajectory are (xj, yj), and the error with the average trajectory position is ∆xj = xj-x̄, ∆yj = yj-ȳ. Then, the standard deviation of the measurement error is calculated. Assuming that the multi-source trajectory group has N moments in a period of time, the calculation formula for the position measurement error standard deviation σpos is σpos=√(1 / N∑t=1N(∆xj²+∆yj²)). The calculation method for the measurement error standard deviations σang and σsp of heading and speed is similar. For example, in a multi-source trajectory group, 3 optimized ship trajectories are included, which are monitored for 10 moments. When calculating the position measurement error standard deviation of a certain trajectory, first calculate the error between the trajectory position and the average trajectory position at each moment, then average the sum of the squares of these errors, and finally take the square root to get σpos. Through such calculations, the system can obtain the measurement error standard deviation of each optimized ship trajectory in the multi-source trajectory group in terms of position, heading and speed relative to the average trajectory, and these standard deviations will be used for subsequent calculation of dynamic fusion weights.

[0108] S218. Calculate the dynamic fusion weight according to the measurement error standard deviation.

[0109] After obtaining the standard deviation of the measurement error of each optimized ship trajectory in the multi-source trajectory group relative to the average trajectory in terms of position, heading and speed, the system calculates the dynamic fusion weight based on these standard deviations. For position information, suppose there are M optimized ship trajectories in the multi-source trajectory group, and the standard deviation of the measurement error of the position of the i-th trajectory is σpos_i, then the calculation formula of its position fusion weight ωpos_i is ωpos_i=(1 / (σpos_i)²) / (∑j=1M1 / (σpos_j)²). For example, there are 3 optimized ship trajectories in a multi-source trajectory group, and their standard deviations of measurement errors in position are σpos_1=0.5, σpos_2=0.8, and σpos_3=1.0 respectively. First calculate the denominator ∑j=131 / (σpos_j)²=1 / (0.5)²+1 / (0.8)²+1 / (1.0)²=4+1.5625+1=6.5625, then calculate ωpos_1=1 / (0.5)² / 6.5625=4 / 6.5625≈0.61, and similarly calculate ωpos_2 and ωpos_3. For heading information, the heading fusion weight ωang_i of the i-th track is calculated as ωang_i=(1 / (σang_i)²) / (∑j=1M1 / (σang_j)²), where σang_i is the standard deviation of the measurement error of the i-th track in terms of heading. For speed information, the speed fusion weight ωsp_i of the ith trajectory is calculated as ωsp_i=(1 / (σsp_i)²) / (∑j=1M1 / (σsp_j)²), where σsp_i is the standard deviation of the measurement error of the ith trajectory in terms of speed. Through this calculation method, the system determines the dynamic fusion weights for each optimized ship trajectory in the multi-source trajectory group in terms of position, heading and speed. In the subsequent weighted fusion process, these weights can reasonably allocate the contribution of each trajectory in generating the final ship target trajectory according to its accuracy and reliability, so that the fused trajectory can more accurately reflect the true motion state of the ship.

[0110] S219: Perform weighted fusion on the multi-source trajectory group according to the dynamic fusion weight to obtain the ship target trajectory.

[0111] After obtaining the dynamic fusion weight of each optimized ship trajectory in the multi-source trajectory group, the system starts weighted fusion to obtain the ship target trajectory. Assuming that there are M optimized ship trajectories in the multi-source trajectory group, at a certain time t, the target track point information of each trajectory includes the position Posi(t), heading Angi(t) and speed Spi(t), and the corresponding dynamic fusion weights are ωpos_i, ωang_i and ωsp_i respectively. For the position information, the position Pos(t) of the ship target trajectory at this moment is calculated by the following formula: Pos(t)=∑i=1Mωpos_iPosi(t). For example, if there are three optimized ship trajectories in the multi-source trajectory group, at time t, their positions are Pos1(t)=(x1, y1), Pos2(t)=(x2, y2), Pos3(t)=(x3, y3), and the corresponding position dynamic fusion weights are ωpos_1=0.4, ωpos_2=0.3, ωpos_3=0.3, respectively. Then the position of the ship target trajectory at this moment is Pos(t)=0.4×(x1, y1)+0.3×(x2, y2)+0.3×(x3, y3)=(0.4x1+0.3x2+0.3x3, 0.4y1+0.3y2+0.3y3). For the heading information, the calculation formula of the heading Ang(t) of the ship target trajectory at this moment is Ang(t)=∑i=1Mωang_iAngi(t). Similarly, for speed information, the speed of the ship target trajectory at that moment is Sp(t)=∑i=1Mωsp_iSpi(t). According to the above method, the system performs weighted fusion calculation on the position, heading and speed information at each moment in the multi-source trajectory group. From the start moment to the end moment of the multi-source trajectory group, such calculation is performed one by one for each moment to obtain a series of complete ship target trajectory points, which are connected to form the ship target trajectory. Through this weighted fusion method, the reliability of different sensor trajectories is comprehensively considered, so that the final ship target trajectory can more accurately reflect the actual motion state of the ship, providing more reliable data support for ship monitoring and tracking.

[0112] The following describes the ship target identification and tracking system in the embodiment of the present invention from the perspective of hardware processing. Figure 3 , which is a schematic diagram of the structure of a physical device of a ship target identification and tracking system in an embodiment of the present application.

[0113] It should be noted that Figure 3 The structure of the ship target identification and tracking system shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0114] like Figure 3As shown, the ship target identification and tracking system includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 302 or the program loaded from the storage part 308 to the random access memory (RAM) 303, such as executing the method described in the above embodiment. In RAM 303, various programs and data required for system operation are also stored. CPU 301, ROM 302 and RAM 303 are connected to each other through bus 304. Input / output (I / O) interface 305 is also connected to bus 304.

[0115] The following components are connected to the I / O interface 305: an input section 306 including an audio input device, a button switch, etc.; an output section 307 including a liquid crystal display (LCD) and an audio output device, an indicator light, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 309 performs communication processing via a network such as the Internet. A drive 310 is also connected to the I / O interface 305 as needed. A removable medium 311, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 310 as needed so that a computer program read therefrom is installed into the storage section 308 as needed.

[0116] In particular, according to an embodiment of the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present invention includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 309, and / or installed from a removable medium 311. When the computer program is executed by the central processing unit (CPU) 301, various functions defined in the present invention are performed.

[0117] It should be noted that specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device.

[0118] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present invention. Each box in the flowchart or block diagram may represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box may also occur in an order different from that marked in the accompanying drawings.

[0119] Specifically, the ship target identification and tracking system of this embodiment includes a processor and a memory. The memory stores a computer program. When the computer program is executed by the processor, the ship target identification and tracking method based on multi-source data fusion provided in the above embodiment is implemented.

[0120] As another aspect, the present invention further provides a computer-readable storage medium, which may be included in the ship target identification and tracking system described in the above embodiment; or may exist independently without being assembled into the ship target identification and tracking system. The above storage medium carries one or more computer programs, and when the above one or more computer programs are executed by a processor of the ship target identification and tracking system, the ship target identification and tracking system implements the ship target identification and tracking method based on multi-source data fusion provided in the above embodiment.

[0121] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0122] As used in the above embodiments, the term "when..." may be interpreted to mean "if..." or "after..." or "in response to determining..." or "in response to detecting...", depending on the context. Similarly, the phrases "upon determining..." or "if (the stated condition or event) is detected" may be interpreted to mean "if determining..." or "in response to determining..." or "upon detecting (the stated condition or event)" or "in response to detecting (the stated condition or event)", depending on the context.

[0123] Those skilled in the art can understand that to implement all or part of the processes in the above-mentioned embodiments, the processes can be completed by computer programs to instruct related hardware, and the programs can be stored in computer-readable storage media. When the programs are executed, they can include the processes of the above-mentioned method embodiments. The aforementioned storage media include: ROM or random access memory RAM, magnetic disk or optical disk and other media that can store program codes.

Claims

1. A ship target recognition and tracking method based on multi-source data fusion, characterized in that: Using a ship target identification and tracking system, the method comprises: Collecting ship monitoring data of the same sea area from multiple source sensors, wherein the ship monitoring data includes ship position coordinates, area characteristics and color characteristics; generating an initial ship track of each sensor according to the ship monitoring data, calculating the detection point similarity of the initial ship track and associating them to obtain an associated track pair, wherein the detection point similarity includes speed similarity, area similarity and color similarity; Performing outlier identification and trajectory optimization on the associated trajectory pair based on a similarity energy function to obtain an optimized ship trajectory, wherein the similarity energy function includes a dynamics option, a persistence option, a fusion interval option, a color similarity option, and an adjustment option; A similarity test is performed by calculating the state difference between the optimized ship trajectories, and the optimized ship trajectories whose similarity exceeds a preset threshold are determined to be the same target and associated to form a multi-source trajectory group; The dynamic fusion weight is calculated according to the standard deviation of the measurement error of each optimized ship trajectory in the multi-source trajectory group relative to its average trajectory, and the ship target trajectory is obtained by performing weighted fusion.

2. The method according to claim 1, characterized in that The step of calculating the similarity of the detection points of the initial ship trajectory and associating them to obtain an associated trajectory pair specifically includes: Calculating the speed similarity of adjacent detection points based on the position information of the detection points in the initial ship trajectory; Calculating the area similarity of adjacent detection points based on the rectangular frame of the detection points in the initial ship trajectory; Calculating the color similarity of adjacent detection points according to the color features of the detection points in the initial ship trajectory; Adjacent detection points are associated based on a weighted combination of the speed similarity, the area similarity, and the color similarity to obtain an associated trajectory pair.

3. The method according to claim 1, characterized in that The step of performing outlier identification and trajectory optimization on the initial ship trajectory based on the similarity energy function to obtain an optimized ship trajectory specifically includes: The similarity of adjacent track points in the initial ship track is calculated using a similarity energy function, wherein the similarity energy function includes a dynamics option, a persistence option, a fusion interval option, a color similarity option, and an adjustment option; Based on the velocity drift coefficient and disturbance variance factor, a linear model is constructed to calculate the mutation probability of the trajectory points, and the trajectory points whose mutation probability exceeds the preset threshold are eliminated. The moving mean method is used to supplement the missing segments after removing the abnormal points to obtain the optimized ship trajectory.

4. The method according to claim 1, characterized in that: The step of performing a similarity test by calculating the state difference between the optimized ship trajectories, determining the optimized ship trajectories with similarity exceeding a preset threshold as the same target and associating them to form a multi-source trajectory group specifically includes: Calculating the state estimation difference of the optimized ship trajectory of different sensors, wherein the state estimation difference includes position, heading and speed information; A likelihood ratio test is performed according to the state estimation difference through a joint probability density function based on uncertain distribution to determine whether different sensor trajectories belong to the same target, thereby obtaining an associated multi-source trajectory group.

5. The method according to claim 4, characterized in that The step of calculating the dynamic fusion weight according to the standard deviation of the measurement error of each optimized ship trajectory in the multi-source trajectory group relative to its average trajectory, and performing weighted fusion to obtain the ship target trajectory specifically includes: Calculating a standard deviation of measurement errors of each optimized ship trajectory in the multi-source trajectory group relative to an average trajectory of the multi-source trajectory group; Calculating a dynamic fusion weight according to the measurement error standard deviation; The multi-source trajectory group is weightedly fused according to the dynamic fusion weight to obtain the ship target trajectory.

6. The method according to claim 1, characterized in that The step of generating an initial ship trajectory of each sensor according to the ship monitoring data specifically includes: When no detection point associated with the target detection point can be found in the previous frame image, a new trajectory segment is generated with the detection point as the trajectory starting point; When no detection point associated with the target detection point can be found in the next frame image, the target detection point is used as the end point of the current trajectory segment; Determine whether the number of frames in which the trajectory appears continuously is less than a preset threshold. If so, it is determined to be a trajectory fragment; The energy function value after the trajectory fragment is fused with other trajectory segments is calculated, and when the energy function value decreases, the trajectory fragment is retained, otherwise it is removed from the trajectory space to obtain the initial ship trajectory.

7. The method according to claim 3, characterized in that The step of eliminating trajectory points whose mutation probability exceeds a preset threshold and using a moving average method to supplement the missing segments after eliminating the abnormal points to obtain an optimized ship trajectory specifically includes: Setting a sliding time window, and calculating the velocity drift coefficient and the disturbance variance factor of the trajectory points within the sliding time window; A linear model of the position of the trajectory point is constructed based on the velocity drift coefficient and the disturbance variance factor, and a mutation probability of the trajectory point is calculated according to the deviation between the trajectory point and the position predicted by the linear model; Eliminate the abnormal points whose mutation probability exceeds a preset probability threshold, and obtain the trajectory segment after eliminating the abnormal points; The moving mean method is used to supplement the missing parts in the trajectory segments after the abnormal points are removed. When the time interval between the trajectory segments exceeds a preset interval threshold, a two-way search method is used to supplement them to obtain the optimized ship trajectory.

8. A ship target identification and tracking system, characterized in that: The ship target identification and tracking system includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to enable the ship target identification and tracking system to execute the method described in any one of claims 1-7.

9. A computer-readable storage medium comprising instructions, characterized in that: When the instruction is executed on a ship target identification and tracking system, the ship target identification and tracking system is caused to execute the method as claimed in any one of claims 1 to 7.

10. A computer program product, characterized in that When the computer program product runs on a ship target identification and tracking system, the ship target identification and tracking system is enabled to execute the method according to any one of claims 1 to 7.

Images