Method, device and medium for tracking small ship target with multispectral visual saliency

Abstract

The application provides a multispectral visual saliency small ship target tracking method, device and medium, the method comprises the following steps: segmenting the candidate ship patch by using the multispectral visual saliency feature fusion strategy, performing sea ship detection on each wave band of a single frame image, and obtaining the centroid position, size and corresponding wave band gray scale feature of the suspected ship target; the target positions detected by the multiple spectrum images are globally nearest neighbor correlated to obtain the final reliable ship target; the ship position of each frame image is accurately geometrically corrected by multi-target correlation matching between the multi-modal data; an improved multi-hypothesis tracking algorithm based on multispectral radiation and size features is designed to track the ship trajectory and estimate the motion state thereof. Through the application, the weak and small ship targets in the GF-4 remote sensing image sequence can be effectively detected and tracked with very high accuracy and recall rate, and the current most advanced performance is generated.

Description

TECHNICAL FIELD

[0001] The present document relates to the technical field of marine information monitoring, and particularly relates to a multi-spectral visual saliency small ship target tracking method, device and medium. BACKGROUND

[0002] Surface target system perception (especially ship detection and tracking) is a very important task of marine information monitoring, and has a wide range of applications in the fields of maritime traffic, marine pollution, illegal fishing, illegal immigration, marine piracy, national defense and maritime security, and border control. With the progress of space-based remote sensing technology, multi-modal satellite data, including satellite automatic identification system (AIS), synthetic aperture radar (SAR), multi-spectral and hyper-spectral optical sensors, global navigation satellite system reflectometry (GNSS-R), have become important means for ship target monitoring. Among them, geostationary earth orbit (GEO) remote sensing satellites have attracted more and more attention due to their advantages such as wide scanning coverage, near real-time continuous observation and high response sensitivity, and therefore have great application potential in marine monitoring. The GF-4 satellite launched in 2015 is equipped with a staring imaging optical sensor, which provides panchromatic multi-spectral (PMS) images with a spatial resolution of 50m and a coverage range of 400km x 400km. This kind of video-like satellite has high revisit observation capability, and the time resolution is as high as 20s. These advantages make it possible to detect and track ships at sea and obtain their motion parameters such as geographical position, speed, heading and moving track, which can assist decision-making and guide low earth orbit (LEO) satellites with higher spatial resolution for further fine identification.

[0003] However, ships in GEO satellite imagery appear as weak targets in vision, with fewer pixels, less features, and lower resolution. In addition, the optical remote sensing imagery of the sea usually exists the influence of sea clutter, reefs, clouds and other interference, which makes it difficult to detect and track ships. Current ship tracking strategies based on GF-4 image sequences mainly include two steps: (1) detecting candidate ships in each frame of image; (2) tracking ships through data association between multi-frame image detection results. Regarding ship detection, the mainstream methods developed in optical remote sensing imagery include methods based on gray statistical feature analysis, methods based on deep learning and methods using human vision system (HVS) mechanism. The first type of method detects ships through the gray statistical data of the hull or wake, but when reefs, broken clouds, clutter and target dense scenes appear, these methods perform poorly. Deep learning-based methods can extract mid-level and high-level features of targets through training and improve the robustness of target detection. However, the limitation of deep learning for weak target detection is that the target lacks information such as texture, structure and shape, which leads to the decline of the performance of the trained detector. In contrast, the HVS-based method has received great attention due to its sensitivity to local anomalies and selective attention mechanism, which fully utilizes intensity, contrast, color, shape, multi-scale representation and other features to generate visual saliency maps and extract regions of interest from the entire scene. Yao et al. designed a local saliency map algorithm based on peak signal-to-noise ratio (PSNR) to detect weak ships in GF-4 images. Similarly, Yu et al. proposed a multi-scale double-neighborhood difference contrast measurement method to determine the location of candidate ships, and then eliminate false alarms through shape feature analysis. The shortcomings of these methods applied to GF-4 imagery are: (1) the multispectral radiation characteristics of ships are not utilized, which can distinguish them from other interference targets; (2) the displacement of moving ships between different bands is ignored, which can lead to higher false alarms.

[0004] Data association is another necessary step for ship tracking, which aims to associate the detection results in multiple images, which helps to eliminate false alarms and reduce the missed alarms in single frame detection. The classical data association methods mainly include nearest neighbor (NN) and its variants, probability hypothesis density (PHD), joint probabilistic data association (JPDA), and multiple hypothesis tracking (MHT). Liu et al. introduced the MHT method to track ships from GF-4 sequence images. By embedding the amplitude information of the near infrared (NIR) band into the ship motion model, Yao et al. proposed an improved MHT method, which can further suppress false alarms and achieve better tracking performance. Xiao et al. used the discriminative associated filtering method with channel and spatial reliability to track ships and associate the detection results of multiple frames. Yu et al. used the JPDA method to associate the detected ships from GF-4 time sequence images, while Wang et al. associated the detected ships by intersection over union (IoU), and then calculated the similarity measure to estimate the apparent stability of the ships in the detected tracks. However, these existing methods do not fully utilize the unique visual information (such as multispectral radiation and size features) of ships, which may lead to incorrect association when dealing with dense ship targets and scenes with severe interference targets (such as reefs and broken clouds). SUMMARY

[0005] The present application provides a multispectral visual saliency-based weak and small ship target tracking method, device and medium, aiming to solve the above problems.

[0006] The present application provides a multispectral visual saliency-based weak and small ship target tracking method, device and medium, aiming to solve the above problems.

[0007] S1, obtaining a gray feature saliency map and a contrast feature saliency map from a geostationary orbit satellite GF-4 image sequence, and combining ship prior size features to segment candidate ship patches from each single-band image;

[0008] S2, globally nearest neighbor association of the positions of the candidate ship patches detected in multiple spectral images to obtain the final reliable ship targets and obtain the attribute features of the ships, and geometric correction of the ship positions of each image through multi-target association matching between multi-modal data;

[0009] S3, the ship is modeled, based on the ship attribute characteristics, the multispectral radiation and size information is embedded into the improved MHT tracker to realize the tracking of small ship.

[0010] The embodiment of the present application provides a computer device, including a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the multispectral visual saliency-based small ship target tracking method when executing the computer program.

[0011] The embodiment of the present application provides a computer readable storage medium, the computer readable storage medium stores a computer program, characterized in that the computer program is executed by the processor to implement the steps of the multispectral visual saliency-based small ship target tracking method.

[0012] The embodiment of the present application adopts the combination of gray scale and contrast feature saliency, which can utilize complementary advantages, weaken sea clutter and background interference, and reduce false alarm and missed alarm. By introducing adaptive parameter selection and protection domain, the features of small ship targets can be well preserved or even improved. The multispectral detection candidate target positions are associated to determine the final ship target, and then the AIS data is used for accurate geometric correction of the ship position of each frame of image. Finally, an improved MHT algorithm fusing multispectral radiation and size features is constructed to track the trajectory of the ship. The small ship targets in the GF-4 remote sensing image sequence are effectively detected and tracked with very high accuracy and recall rate, and the current most advanced performance is generated. BRIEF DESCRIPTION OF DRAWINGS

[0013] In order to more clearly illustrate the technical solutions in the one or more embodiments of the present application or the prior art, the drawings needed to be used in the embodiment or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0014] Figure 1 The flowchart of the multispectral visual saliency-based small ship target tracking method of the embodiment of the present application;

[0015] Figure 2 The working flowchart of the multispectral visual saliency-based small ship target tracking method of the embodiment of the present application;

[0016] Figure 3 The multispectral visual saliency-based small ship target tracking method of the embodiment of the present application;

[0017] Figure 4 GF-4 image contrast significant diagram of embodiment of the present application;

[0018] Figure 4 (a) is the result generated by traditional lateral inhibition network;

[0019] Figure 4 (b) is the result generated by lateral inhibition network with protection domain;

[0020] Figure 5 Workflow diagram of weak small ship target detection method of embodiment of the present application;

[0021] Figure 6 Schematic diagram of multispectral radiation characteristics of different types of targets of embodiment of the present application;

[0022] Figure 6 (a) is GF-4 false color image;

[0023] Figure 6 (b) is corresponding NIR band image

[0024] Figure 6 (c) is red band image

[0025] Figure 6 (d) is green band image;

[0026] Figure 7 Schematic diagram of displacement phenomenon of moving ship in multispectral image of embodiment of the present application;

[0027] Figure 8 Schematic diagram of research area and data set coverage of embodiment of the present application;

[0028] Figure 9 Local magnification effect diagram of near-infrared band ship detection of embodiment of the present application;

[0029] Figure 9 (a) is local image diagram of NIR band;

[0030] Figure 9 (b) is detection result of single NIR band;

[0031] Figure 10 Visual effect diagram of ship detection of two ROIs of embodiment of the present application;

[0032] Figure 10 (a) is ROI1 result;

[0033] Figure 10 (b) is ROI2 result;

[0034] Figure 11Fig. 10 is a schematic diagram of the multispectral gray level feature variation of 10 ships between multiple frames of images according to an embodiment of the present application;

[0035] Figure 12 Fig. 11 is a schematic diagram of the size feature variation of 10 ships between multiple frames of images according to an embodiment of the present application;

[0036] Figure 13 Fig. 12 is a schematic diagram of the point pair matching for geometric correction in a first frame of images according to an embodiment of the present application;

[0037] Figure 14 Fig. 13 is a schematic diagram of the ship detection and tracking results in a GF-4 image sequence according to an embodiment of the present application;

[0038] Figure 14 (a)- Figure 14 (b) is the ship detection results of two ROIs in all 5 frames of images;

[0039] Figure 14 (c)- Figure 14 (d) is the ship tracking results of two ROIs and the AIS data points for verification;

[0040] Figure 14 (e)- Figure 14 (f) is the ship trajectories generated in two ROIs;

[0041] Figure 15 Fig. 15 is a schematic diagram of the approximation of the motion state between the tracking results and the AIS data in five frames of images according to an embodiment of the present application;

[0042] Figure 15 (a) is a schematic diagram of the point position error between the estimated position and the AIS position;

[0043] Figure 15 (b) is a schematic diagram of the correspondence between the estimated speed and the AIS speed;

[0044] Figure 15 (c) is a schematic diagram of the correspondence between the estimated heading and the AIS heading. DETAILED DESCRIPTION

[0045] In order to enable persons skilled in the art to better understand the technical solutions in one or more embodiments of the present application, the technical solutions in one or more embodiments of the present application will be described clearly and completely below with reference to the drawings in one or more embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on one or more embodiments of the present application, all other embodiments obtained by persons skilled in the art without creative labor should fall within the protection scope of the present document.

[0046] Method embodiments

[0047] The embodiment of the present application provides a multispectral visual saliency small ship target tracking method, Figure 1 For the multispectral visual saliency small ship target tracking method of the embodiment of the present application, according to Figure 1 As shown in the figure, the multispectral visual saliency small ship target tracking method of the embodiment of the present application comprises:

[0048] S1, by means of the GF-4 image sequence of the geostationary satellite, the gray feature saliency map and the contrast feature saliency map are obtained, and the candidate ship patch is segmented from each single-band image combined with the ship prior size feature;

[0049] The ship in the GF-4 geostationary satellite image has several pixels and low distinguishability, and can be regarded as a small target. As shown in the figure, Figure 3 The visual saliency of the ship is usually dependent on its multispectral radiation (i.e. brightness and color), contrast, size and other features. It is worth noting that the target detection method based on the HVS mechanism has attracted more and more attention in recent years, and has proved to have high robust performance in detecting small targets in complex scenes. In the present application, a multispectral visual saliency feature fusion strategy is designed to segment the candidate ship target patch from each single-band image, which mainly includes:

[0050] S11, the adaptive mean shift based on the space-gray joint domain is used to generate the gray feature saliency map, and the sea clutter and noise are filtered out;

[0051] The basic idea of the mean shift algorithm is that: given an initial sample space, the density function can be calculated according to the sample points in the feature space, and the position with the maximum function value is the optimal solution. Let Denote the set of feature vectors in the d-dimensional feature space. The density function at point The density function at point

[0052]

[0053] Then the mean shift vector is

[0054]

[0055] Where g(*) = -K(*).

[0056] The embodiment of the present application adopts the Gaussian kernel function of the space-gray joint domain to process the GF-4 image, and adopts the space and gray feature vectors to estimate the corresponding feature bandwidth h s And h g Based on this kernel function, the point The distribution of the feature vector can be expressed as,

[0057]

[0058] wherein and denote the spatial and intensity part of the feature vector, respectively, C is a normalization constant, and k(*) is the profile function of the kernel K(*). Accordingly, the mean shift vector can be expressed as

[0059]

[0060] When the intensity saliency feature is obtained by using the mean shift algorithm, the spatial bandwidth determines the range of the current point density gradient estimation. If this value is too small, it may lead to the ship target splitting or the poor clutter / noise removal performance, while too large bandwidth may lead to over-smoothing and loss of target features. In addition, the selection of the spatial bandwidth will affect the iteration speed of the algorithm. Therefore, an adaptive spatial bandwidth is adopted, that is, the spatial bandwidth is initialized to 2 according to the statistics of the ship size features, and the incremental step is set to 1. Experiments show that when the number of sample points with similar gray values to the gray value of the current point to be smoothed is less than half of the number of all sample points in the spatial bandwidth, the corresponding spatial bandwidth is the optimal solution. When the gray difference between the sample point and the current point to be smoothed is not greater than 4, it is considered that the gray values are similar. When processing scenes with dense ship targets, a smaller spatial bandwidth can retain the details and gray features. Otherwise, a larger bandwidth will obtain better smoothing effect.

[0061] Similarly, a larger gray bandwidth may lead to over-smoothing, while a smaller bandwidth cannot better filter noise and sea clutter. Generally, the globally optimal fixed bandwidth and adaptive bandwidth are two common selection methods. Here, we introduce the asymptotic average integrated variance criterion commonly used in Plug-in rules to estimate the adaptive gray bandwidth,

[0062]

[0063] wherein d is the dimension of the feature space, n is the number of pixels, σ j is the standard deviation, g i , respectively denote the gray value and the average gray value of each pixel within the bandwidth range.

[0064] The adaptive mean shift algorithm with spatial-intensity joint domain considers both spatial and intensity domain information, and can well retain the brightness features of the ship target and filter out sea clutter and noise without changing the target size, thereby generating a good intensity saliency map.

[0065] S12, generating a contrast feature saliency map by using the side suppression network with a protection domain, and step S12 specifically comprises:

[0066] Due to its advantages of highlighting target edges and enhancing contrast, a side-inhibition network is used to improve the contrast of small ship targets. The model used in this invention is...

[0067]

[0068] Where e i,j ,r i,j Let k be the input and output of the receptor unit (i,j), respectively. ij,pq Let R represent the inhibition coefficient of receptor unit (i,j) with respect to unit (p,q), and R be the radius of the inhibition region. For a grayscale image, if this radius is 2, the model can be expressed as...

[0069]

[0070] Where g(i,j) is the gray value of pixel (i,j), and k i (i = 1, 2) is the inhibition coefficient.

[0071] Considering the grayscale and size characteristics of small ship targets, this embodiment of the invention establishes a protection region centered on the current pixel to prevent the target pixel from affecting the output. In this embodiment, the radii of the protection region and the suppression region are set to 1 and 2, respectively, as shown in the following formula:

[0072]

[0073] Regarding the inhibition coefficient, let k i =α / R i R1 and R2 are set to 2 and 3 respectively, and α is a constant that satisfies the condition 1 + 16k1 + 24k2 = 0. The suppression coefficient matrix forms a 7×7 template, in which 24 coefficients with a value of k2 are distributed in the outermost layer, 16 coefficients with a value of k1 are distributed in the next outermost layer, and the coefficient in the center of the template is 1.

[0074] By convolving the image with a coefficient matrix, a contrast saliency map can be obtained. Then, thresholding can be performed to obtain the potential ship position distribution. Based on this coefficient template, the contrast features of small ships can be further enhanced while preserving the integrity of the target body. Figure 4 As shown, Figure 4 (a) shows the result generated by a traditional side-inhibition network; Figure 4 (b) shows the result of generating a side suppression network with a guard domain. Compared with a conventional side suppression network, the side suppression network with a guard domain enhances the contrast features of most ships and suppresses interference information from almost the entire ocean background.

[0075] In summary, the combination of the gray level and contrast feature saliency can take advantage of the complementary strengths to weaken the sea clutter and background interference, while reducing false alarms and missed alarms. By introducing adaptive parameter selection and protection domain, the features of weak and small ship targets can be well preserved or even improved. Therefore, the present application performs image segmentation on the gray level saliency map under the guidance of the candidate ship position distribution map. With the help of the size statistical characteristics of ships in GF-4 satellite images, we can determine the candidate ship patches in each single waveband of each frame of image. Finally, the centroid position, average gray level and size characteristics of the ship patches in each waveband can be obtained. Here, due to the potential influence of ship wakes, we retain the image blocks with a size between 2 and 75 pixels. Figure 5 The entire workflow of the proposed weak and small ship target detection method is given.

[0076] S13, performing region growing-based image segmentation on the gray level saliency map by using the seed points obtained by thresholding the contrast saliency map;

[0077] S14, extracting candidate ship patches by using the prior size characteristics of ship targets, so as to obtain the centroid position, average gray level and size characteristics of the candidate ship patches in each waveband of each frame of image.

[0078] S2, performing global nearest neighbor correlation on the positions of the candidate ship patches detected in multiple spectral images to obtain the final reliable ship targets and the attribute characteristics of the ships, and performing geometric correction on the ship positions of each frame of image by multi-target correlation matching between multi-modal data;

[0079] Almost all existing ship detection methods based on GF-4 images only use single spectral information, such as the NIR waveband with the maximum contrast between ships and the sea background, but this will generate a large number of false alarms due to the existence of interference targets such as reef islands, broken clouds and sea clutter. In addition, the accuracy and efficiency of the subsequent ship tracking algorithm will be greatly reduced. In fact, multi-spectral radiation characteristics are inherent properties of the target itself, and are significant features that can be distinguished from other targets. Figure 6 Fig. 1 is a schematic diagram of the multi-spectral radiation characteristics of different types of targets in the embodiment of the present application, Figure 6 (a) is a GF-4 false color image, Figure 6 (b) is the corresponding NIR waveband image, Figure 6 (c) is the red waveband image, Figure 6(d) is a green band image; it can be seen that the reefs, which usually have a large contrast feature in the near infrared band, will greatly affect the ship detection, and some static reefs are similar to ships in appearance. In contrast, the reefs will disappear in the red and green bands, and the contrast of some relatively small ships begins to weaken in the red band until it disappears in the green band. Therefore, ship detection in a single NIR band will bring many false alarms, and detection in other bands may result in missed alarms. For these reasons, we should make full use of multispectral radiation information.

[0080] At the same time, there is a time lag between the collection of different spectral images. For GF-4 images, the time interval between multispectral data bands is up to several seconds, and the time interval between the panchromatic band and the NIR band is about 40s, which causes the position of the target previously detected on the multispectral image to shift, as shown in Figure 7 The greater the speed of the ship, the greater the displacement. Therefore, in order to determine the final detection result of the ship target, it is still necessary to perform data association between the previous detections in multiple bands. Specifically, the association criteria of the present application are as follows:

[0081] (1) Establish association based on the position of the candidate ship detected in each band of a single frame image;

[0082] (2) The detection in all other bands should be associated with the detection in the NIR band;

[0083] (3) Fix a contrast threshold, such as setting the contrast threshold to 45 in the embodiment of the present application, and define the corresponding candidate ship in the NIR band whose contrast value in the saliency map is greater than the threshold as a high-contrast target;

[0084] (4) Set a distance threshold, such as setting the distance threshold to 15 pixels in the embodiment of the present application, and if the distance between two detections in two bands exceeds the threshold, it is considered that no association can occur;

[0085] (5) If a candidate ship is a high-contrast target and is not associated with any detection in other bands, discard the candidate ship;

[0086] (6) If the detection in a band is associated with a certain detection in the NIR band, keep the average gray scale of the band, and if no association occurs, mark the corresponding gray scale of the band as NULL;

[0087] (7) The final determined ship attribute features include its centroid position in the NIR band, the average gray scale vector of multiple bands, and the size of the NIR band.

[0088] As one of the most popular data association methods, GNN algorithm is used in the present invention for the association between multispectral detections and between multi-modal data. GNN assigns the best association with the lowest global cost, which allows each element in the set Θ to be assigned to only one element in the set Ψ The optimization model can be formulated as

[0089]

[0090] where n and m are the number of elements in the sets Θ and Ψ, respectively, δ is the distance threshold, and M ij is the association variable. M ij = 1 indicates one-to-one association, while M ij = 0 indicates no association. The association relationship in GNN assignment can be obtained by the Hungarian algorithm, which is a simple and effective method to solve the linear assignment problem.

[0091] After the multispectral detections are associated, the final reliable ship targets in each frame and their corresponding static attribute features can be determined. The combination of motion characteristics and static attribute features will greatly improve the reliability and accuracy of ship tracking. However, due to the systematic error of RPC-based orthorectification, the precision of ship position is very low, which will affect the performance of ship tracker. In order to accurately correct the geometric position, we introduce the widely used affine transformation model, i.e.,

[0092]

[0093] where (l', s') and (l, s) are the image pixel coordinates obtained from the projection of GCPs based on the RPC model and the corresponding GF-4 image matching points, respectively, and (e i , f i ) (i = 0, 1, 2) are the transformation coefficients. Since there are almost no or even no GCPs on the sea surface, the AIS information of the ship will be a choice. AIS is a cooperative automatic reporting system that records the static information (such as identification number, type, length, and width) and motion information (such as longitude, latitude, heading, and speed) of the ship. In the present invention, we use part of the AIS data in the experimental area for geometric correction, and another part for performance verification of the proposed method.

[0094] Since the ship positions detected in NIR band are reserved for subsequent ship tracking, the time lag of about 40s between the acquisition time of NIR image and the time given in image file should be considered when performing linear interpolation of AIS data. After the spatio-temporal unification of GF-4 image and AIS data, GNN data association is still needed to seek the best matching association of ship positions between multi-modal data. In order to robustly correct systematic errors, we further use random sample consensus (RANSAC) algorithm to eliminate gross errors in point-to-point association. Finally, the accurate geographic coordinates of ships in each frame of image are converted by using RPC correction model and equation (10).

[0095] S3, modeling the motion of the ship, embedding multispectral radiation and size information into the improved MHT tracker based on ship attribute features to track the small and weak ship.

[0096] Single-frame detection can only obtain the system distribution and static attribute features of the ship target on the sea, and for marine surveillance, it is more important to track the motion trajectory of the ship to predict the overall situation. Moreover, multi-frame data association can further reduce false alarms and occasional missed alarms caused by interference targets in single-frame detection. Based on the previously extracted ship attribute features, the multispectral radiation and size information are embedded into the tracking framework.

[0097] The motion state of the ship target in the kth frame can be represented as where λ k , and v represent the geographic position and velocity component along the longitude and latitude directions, respectively, represents the multispectral gray value vector, and s k is the target size. The speed and heading (relative to true north) relative to the ground can be calculated by the velocity component in the plane rectangular coordinate system. The measurement in the kth frame can be described as where lon k and lat k represent the longitude and latitude coordinates of the measurement, is the vector of average gray values of multiple bands, and size k is the size of the measurement. The units of longitude and latitude and heading are degrees (°), and the units of distance, time and speed are nautical miles (nm), hours (h) and knots (kn) respectively.

[0098] The target state transition model formula used in the embodiment of the present application is

[0099]

[0100] where f k-1G is the state transition function k-1 V is the process noise matrix k-1 and v is the process noise. Let and v k-1 follow a multivariate Gaussian distribution with mean zero and covariance matrix where σ v is the standard deviation of the process noise. For constant heading motion trajectory, the track prediction from the geographic position to at time T can be expressed as

[0101]

[0102] where T is the time interval, a ~ 60 nm / ° is a constant that converts the distance from nautical miles (nm) to degrees (°), and φ is the middle latitude. Therefore, the state transition equation can be expressed as

[0103]

[0104] Since the above equation is nonlinear, the first order extended Kalman filter (EKF) is used to perform the ship state prediction and update.

[0105] The measurement equation can be modeled as

[0106]

[0107] where

[0108]

[0109] where H k is the measurement matrix, w k represents the measurement noise and follows a Gaussian distribution with mean zero and covariance matrix R k , σ p , σ g and σ s are the standard deviations of the position error, gray level and size bias, respectively.

[0110] MHT is known as the theoretically optimal data association algorithm. When the measurement-track association conflicts, MHT generates alternative logical hypotheses to delay the decision so that the subsequent measurements can resolve the uncertainty. For this purpose, we propose an improved MHT tracker that embeds multispectral gray level and size features for weak and small ship tracking in multiple frames. The cumulative log-likelihood ratio is often used as the track score to evaluate the probability of the track hypothesis. The score of track j at the kth frame can be expressed in the recursive form as

[0111] Sj (k)=S j (k-1)+ΔS j (k) (16)

[0112] Where ΔS j (k) is the score increment, which can be calculated by the following formula:

[0113]

[0114] Where P D For the detection probability, λ n With λ f These represent the spatial densities of newly formed targets and clutter, respectively. Let d represent the residual covariance matrix. ij The distance is Mahalanobis distance. i = 0 indicates no correlation, while i > 0 indicates that trajectory j is correlated with measurement i in the kth frame.

[0115] Typically, a target's static attributes are independent of its motion information and contribute significantly to associated tracking. For static attributes such as multispectral grayscale and size, we use a normal distribution function to measure the similarity between the measured and predicted trajectories. Therefore, the score increment generated by the target features can be expressed as...

[0116]

[0117] in and These represent the multispectral and size score increments, respectively, where m represents the spectral band. With size i To measure the characteristics of i, With s j Let be the predicted feature of trajectory j, and c be the constant background hypothesis posterior probability. Finally, the total score increment is the joint increment generated by motion, multispectral radiation and size features, i.e., the sum of equations (17) and (18). Figure 2 This is a flowchart illustrating the overall workflow of the multispectral visual saliency method for tracking weak ship targets according to an embodiment of the present invention.

[0118] During data association, since it cannot be guaranteed that grayscale features of all bands can be extracted during the ship detection phase, only spectral features shared by measurements and trajectories are used for association. Empirically, most ships detected in the NIR band can also be detected in the red band; therefore, utilizing grayscale information from these two bands for association tracking can achieve a balance between performance and efficiency. Furthermore, the maximum speed threshold for moving ships is crucial to avoiding infeasible data association, while minimum speed constraints can further remove interfering targets such as reefs and islands.

[0119] To verify the performance of our method in weak and small ship target detection and tracking, we illustrate our method with a specific example. The specific experimental process is as follows:

[0120] The GF-4 time-series data composed of 5 frames of Level 1A PMS imagery are selected for the experiment. The imaging area is located in the East China Sea, as shown in Figure 8 , the time range is from 03:47:24 to 03:59:47 (UTC) on March 9, 2017, and the time interval between adjacent frames is about 186 seconds. Each frame has five spectral bands (i.e., NIR, red, green, blue, and panchromatic bands), and the size of each band image is 10,240 x 10,240 pixels, with a spatial resolution of 50 m. Two ROIs bounded by red boxes are selected for the experiment, and each ROI covers an area of more than 100 km x 100 km. In addition, the ship AIS dataset is used for geometric accuracy correction and performance verification, which covers the entire study area in space and time.

[0121] In the ship detection stage, we set the region growing threshold for image segmentation to 2, and most other parameters are adaptive or automatically calculated. Figure 9 The local magnified view of ship detection in the near-infrared band is shown. Figure 9 There are many dense ships in (a), some of which appear smaller and weaker. Figure 9 In (b), we can find that all the ships can be well detected. Figure 10 The visual effect of ship detection in the first frame image for the two ROIs is presented, Figure 10 (a) is the ROI1 result, Figure 10 (b) is the ROI2 result, in which the positions of the detected ships are marked with red circles. The size of the circle represents the size of the ship, and the green-filled gray value in the circle means the average gray value of the ship in the NIR band. We can find that the ships with stronger radiation characteristics usually have larger size characteristics, and as the size of the ship increases, its distribution will become more sparse. ROI1 is a non-uniform area, and the overall intensity of the area is brighter, which makes the contrast of the ship very low. The upper left corner of the ROI2 area contains some reefs and islands, and the bottom of the image contains some broken clouds.

[0122] The multispectral gray and size features extracted after ship detection are the static properties of the ship, which change weakly between multiple frames, as shown in Figure 11 and Figure 12 . Figure 11The gray level feature changes of the 10 ships in the NIR and red bands between multiple frames are shown, where different markers represent different ship targets, and the red and green curves represent the gray level values in the NIR and red bands, respectively. It is worth noting that the gray level change range in the red band is usually much smaller than that in the NIR band, and the change value does not exceed 8. In addition, the gray level value of some ships in the red band is larger than that in the NIR band. Similarly, the change value of the ship size feature in the NIR band usually does not exceed 4 pixels. More deeply, we can find that the multispectral gray level and size features of each ship are generally different from those of other ships, which is the reason why we embed these features into the ship tracking model. The point pair matching situation in the first frame after the multi-modal data association and RANSAC processing is shown in Figure 13 , which are uniformly distributed in the study area and will be used for geometric correction of the ship detection position. The positioning error is different in different sea areas.

[0123] In the ship tracking stage, the related parameter settings are as follows: σ p = 0.002°, σ v = 0.01 nm / h 2 , σ g = 10, σ s = 4, c = 0.1, P D = 0.95, λ f = 10 -11 and λ n = 0. Figure 14 The ship detection and tracking results based on the GF-4 image sequence are given. In Figure 14 (a) Figure 14 (b), we show the ship detection results of two ROIs in all five frames of images, where the ship positions have been geometrically corrected. We notice that there are still some false alarms and a few missed alarms in the detection results. However, through subsequent multi-frame association tracking, most of the false alarms have been eliminated, and some missed alarms have been well estimated and appropriately filled. In Figure 14 (c) Figure 14 (d), a set of circles of the same color represent a ship track point generated by our tracker, from which we can see that the tracking results can be highly consistent with the interpolated AIS track points. Finally, Figure 14 (e) and Figure 14 (f) show the ship trajectories in two ROIs, where the red line represents a trajectory, and the star and diamond markers represent the start and end of the trajectory, respectively.

[0124] The present application adopts indicators such as precision and recall to quantitatively evaluate the performance of the ship detection and tracking algorithm, and some related methods in recent years are also used for comparison. The precision and recall can be calculated by the following formula

[0125]

[0126] where N TP , N FP and N FN are the number of true samples, false positive samples and false negative samples, respectively. Table 1 gives the values of the metrics for ship detection and tracking in the two ROIs in the five frames. N TP , N FP and N FN are determined by cross-referencing with AIS data and manual annotation. There are 86 and 39 ships in ROI1 and ROI2, respectively, whose life cycles span the whole image sequence, among which 49 tracked ships can be verified by ship AIS data. From Table 1, we can see again that false alarms can be effectively reduced by MHT tracking. In contrast, the overall precision and recall of our method are superior to those of other methods, and the advantage in ship detection seems to be more prominent, which further demonstrates the effectiveness of incorporating multispectral visual saliency into the framework of ship detection and tracking.

[0127] Table 1. Comparison of quantitative indicators for ship detection and tracking

[0128]

[0129] The present application uses the ships whose trajectories are verified by AIS data in the two ROIs to evaluate the accuracy of motion state estimation. Figure 15 shows the closeness of the motion states between the trajectory tracking results in the five frames and the AIS data, Figure 15 (a) is the point error between the estimated position and the AIS position; Figure 15 (b) is the correspondence between the estimated speed and the AIS speed; Figure 15 (c) is the correspondence between the estimated heading and the AIS heading. Table 2 gives the comparison of motion state estimation errors between our method and other methods, which demonstrates the high accuracy and reliability of our method. The average estimation errors of the proposed method for position, speed and heading are 83.2 m, 0.26 kn and 2.24°, respectively.

[0130] Table 2. Comparison of motion state estimation errors

[0131]

[0132] GEO remote sensing satellites have the ability of wide coverage, high revisit and continuous observation, which show good application prospects in marine monitoring and situation awareness. Most of the existing ship detection and tracking methods based on such satellite images will produce high false alarm and miss alarm, which is mainly because the ship appears as a small target and is often affected by interference targets such as reefs and broken clouds. By introducing the visual saliency features of multi-spectral bands, we propose a new visual detection and association tracking method to solve the current difficulties. First, the candidate ship detection is performed on each band of each frame image by a multi-visual saliency feature fusion strategy to obtain the position, size and corresponding spectral features of suspicious ship patches. Then, since the moving ship has displacement between multiple band images of the same frame, we associate the candidate target positions of multi-spectral detection to determine the final ship target, and then use the AIS data to accurately geometrically correct the ship position of each frame image. Finally, an improved MHT algorithm is constructed to track the ship trajectory by fusing the multi-spectral radiation and size features. Experiments and comparisons prove the high performance and high reliability of our method.

[0133] Considering the higher sea conditions and more complex environment of the marine scene, it is still a challenging work to improve the performance of ship detection and tracking. The weak target detection method that can eliminate interference and enhance target contrast at the same time will be beneficial to the accuracy and efficiency of the whole process. More robust and reliable motion modeling can improve the accuracy of target state estimation. At the same time, deep learning algorithms can also mine more advanced semantic features for detection and tracking. In addition, the fusion of multi-modal remote sensing data such as satellite video, hyperspectral and SAR images can better serve marine monitoring by taking advantage of complementary advantages. These are potential research areas that we will focus on in the future.

[0134] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for tracking small and weak ship targets with multispectral visual saliency, characterized in that, The method comprises the following steps: S1, obtaining a gray feature saliency map and a contrast feature saliency map through a GF-4 image sequence of a geostationary satellite, and combining a ship prior size feature to segment a candidate ship patch from each single-band image; S2, performing global nearest neighbor association on the positions of the candidate ship patches detected in multiple spectral images to obtain final reliable ship targets and attribute features of the ships, and performing geometric correction on the positions of the ships in each image through multi-target association matching between multi-modal data; S3, modeling the motion of the ships, embedding multispectral radiation and size information into an improved MHT tracker based on the attribute features of the ships to track small and weak ships; The embedding of the multispectral radiation and size information into the improved MHT tracker in S3 to track the small and weak ships specifically comprises: In a first Frame tracks The score is expressed in a recursive form as follows, Equation 12; wherein, is the score increment, Equation 13; wherein, is the detection probability, is the detection probability, are the spatial densities of new-born targets and clutter, respectively, denotes the residual covariance matrix, is the Mahalanobis distance, denotes the uncorrelated but denotes the track and the measurement In the first frame association; The score increment generated by the target features is represented as: Equation 14; wherein, with being the multispectral and size score increments, respectively, denotes the spectral band, with being the feature of the measurement , with being the predicted feature of the trajectory , being the constant background assumed posterior probability, being the standard deviation of the gray scale deviation, being the standard deviation of the size deviation, and finally, the total score increment is the joint increment resulting from the motion, multispectral radiation and size features, i.e. the sum of equation 13 and equation 14.

2. The method of claim 1, wherein, S1 specifically comprises: S11, generating a gray feature saliency map by using adaptive mean shift based on a space-gray joint domain, and filtering sea clutter and noise; S12, generating a contrast feature saliency map by using a side suppression network with a protection domain; S13, performing region growing-based image segmentation on the gray feature saliency map by using seed points obtained through thresholding of the contrast feature saliency map; S14, extracting a candidate ship patch by means of a prior size feature of a ship target, so as to obtain the centroid position, average gray value and size feature of the candidate ship patch in each band of a single image.

3. The method of claim 2, wherein, S11 specifically comprises: GF-4 images are processed using a Gaussian kernel function in the spatial-grayscale joint domain, and the corresponding feature bandwidth is estimated using spatial and grayscale feature vectors. and Based on this kernel function, points on the GF-4 image The distribution is expressed as: Formula 1: wherein with denote the space of eigenvectors and the gray part, respectively, is a normalization constant, is the kernel profile function, respectively, the mean shift vector is expressed as: Formula 2: An adaptive spatial bandwidth is adopted, that is, the spatial bandwidth is initialized according to the statistics of the size feature of the ship, and an incremental step is set, when the number of sample points with similar gray values to the gray value of the current point to be smoothed is less than half of the number of all sample points in the spatial bandwidth, the corresponding spatial bandwidth is the optimal solution, and an asymptotic average integral variance criterion commonly used in Plug-in rules is introduced to estimate the adaptive gray bandwidth. Formula 3: wherein, is the dimension of the feature space, is the number of pixels, is the standard deviation, respectively represent the gray scale and the average gray scale value of each pixel in the bandwidth range.

4. The method of claim 2, wherein, S12 specifically comprises: The model of the side suppression network with a protection domain is obtained through formula 4: Formula 4: wherein are the input and output of the susceptor unit respectively, denotes the susceptor unit for the unit the damping coefficient of the unit is the damping domain radius; When the radius of the protection domain and the suppression domain is set to 1 and 2 respectively, the formula is: Formula 5: wherein is a gray value of a pixel; with respect to the suppression coefficient, let , be respectively 2 and 3, be a constant and satisfy the condition ; The contrast saliency map is obtained by using a coefficient matrix to perform convolution operation on the image.

5. The method of claim 1, wherein, The global nearest neighbor association on the positions of the candidate ship patches detected in multiple spectral images in S2 to obtain final reliable ship targets and attribute features of the ships specifically comprises: S21, establishing association based on the positions of the candidate ships detected in each band of a single image obtained in step S1, and correlating the detection in all other bands with the detection in the NIR band; S22, setting a contrast threshold value in advance, and defining the corresponding candidate ship in the NIR band with a contrast value greater than the contrast threshold value in the saliency map as a high-contrast target; S23, setting a distance threshold value in advance, if the distance between two detections in two bands exceeds the distance threshold value, association cannot occur, and if a candidate ship is a high-contrast target and is not associated with any detection in other bands, the candidate ship is discarded. S24, if a detection in one band is associated with a certain detection in the NIR band, the average gray of the band is reserved, if no association occurs, the corresponding gray of the band is marked as NULL; S25, obtaining the finally determined attribute features of the ship, the finally determined attribute features of the ship including: the centroid position in the NIR band, the average gray vector of multiple bands and the size of the NIR band.

6. The method of claim 1, wherein, The geometric correction of the ship position of each frame of image in S2 is specifically as follows: An affine transformation model is introduced, Equation 6; wherein, with are image pixel coordinates obtained from the projection of the GCPs based on the RPC model and the corresponding GF-4 image matching points, respectively, is a conversion factor; A random sample consistency algorithm is used to eliminate gross errors in point pair association, and an RPC correction model and an affine transformation model are used to convert the accurate geographic coordinates of the ship in each frame of image.

7. The method of claim 1, wherein, The motion modeling of the ship in S3 specifically includes: A target state transition model formula is obtained: Equation 7; wherein is the state transition function, and are the process noise matrix and process noise, respectively, such that and follows a multivariate Gaussian distribution with mean zero and covariance matrix where is the standard deviation of the process noise. For a great circle trajectory, the track prediction from a geographic position to is represented as: Formula 8: wherein, is a time interval, denotes a constant converting distance from nautical miles (nm) to degrees (°), is the middle latitude; The state transition equation is expressed as: Formula 9; A first-order extended Kalman filter is used for ship state prediction and update, and a measurement equation model is modeled as: Formula 10; wherein Equation 11; wherein, is a measurement matrix, denotes the measurement noise and is subject to a Gaussian distribution with zero mean and covariance matrix , and are the standard deviations of the position error, the gray scale and the size deviation, respectively.​ 8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to realize the steps of the weak small ship target tracking method of multi-spectral visual saliency according to any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 8. The computer program is executed by the processor to realize the steps of the weak small ship target tracking method of multi-spectral visual saliency according to any one of claims 1 to 7.

Images