A ship image track extraction method for non-continuous interference of visual features

Abstract

The present invention provides a ship image track extraction method for non-continuous visual feature interference, comprising: acquiring ship images, including images of scenes where the ship is occluded and images of scenes where the ship is not occluded; detecting the ship images based on the DeepSnake model and the YOLOX model to obtain the ship's position in the image sequence; inputting the ship's position and the detection results of the DeepSnake model into the YOLOX model to obtain the ship's external contour; and tracking the ship's trajectory based on the external contour and the ship's position to obtain the ship's motion trajectory in the image sequence. The present invention introduces occlusion state determination and further utilizes the Kalman algorithm to obtain a more accurate ship trajectory, thereby improving the accuracy of extracting ship navigation tracks from shipborne surveillance video data and enhancing robustness in complex sea conditions.

Description

Technical Field

[0001] The present invention belongs to the technical field of target detection, pixel-level tracking and trajectory prediction, and in particular relates to a ship image track extraction method oriented to discontinuous interference of visual features. Background Art

[0002] Since the beginning of the 21st century, the shipping industry has flourished, and research on surface ship detection began early. Methods for surface ship detection can be divided into the following categories: Surface ship detection methods based on satellite remote sensing images. These images cover a wide range of ocean areas and contain numerous targets, but the images have low resolution and poor stereoscopic perception. The relative size of ships in the images is too small, making it difficult to obtain precise positioning coordinates of ships. Surface ship detection methods based on aperture synthetic radar images. Aperture synthetic radar can obtain high-resolution radar images in extremely low-visibility weather, but it lacks rich spectral information and is less effective for ship detection. Surface ship detection methods based on infrared images are not affected by ambient brightness. Infrared image detection can also obtain target positions at night or in foggy weather. However, infrared image resolution is low, and uneven airflow on the ocean surface can cause temperature fluctuations, interfering with infrared imaging and affecting detection. Surface ship detection methods based on visible light images offer rich spectral information, making it easy to distinguish between targets and inexpensive. Compared with other images, visible light images have a more obvious advantage in port monitoring or offshore surface ship detection. However, the ship trajectories obtained by methods based on visible light imagery often fail to recognize the spatiotemporal correlations between ships in adjacent image frames due to occlusion or changes in the ship image representation. Ship detection under occlusion conditions is unsatisfactory, especially when detecting multiple moving ships, with significant false and missed detections.

[0003] In recent years, autonomous navigation has received increasing attention, and automatic collision avoidance technology is an integral part of autonomous navigation safety. In real-world scenarios, obstacle detection has a significant impact on the safety of autonomous navigation. Surface ship segmentation helps autonomous navigation successfully avoid obstacles, so it is essential to incorporate surface ship segmentation into ship trajectory extraction.

[0004] The multi-target tracking model based on the Bytetrack algorithm is a classic multi-target tracking algorithm. The Bytetrack algorithm can be easily and quickly applied to ship trajectory extraction. However, the complex background imaging factors in shipborne surveillance videos, especially in scenes with discontinuous visual features, can lead to significant deviations in ship target detection, reducing tracking effectiveness and increasing the difficulty of tracking and matching. This results in certain limitations in trajectory extraction methods based on the Bytetrack ship tracking algorithm.

[0005] Therefore, it is urgent to propose a ship image track extraction method oriented to the discontinuous interference of visual features to meet the challenge of multi-target ship trajectory extraction in scenarios with discontinuous ship visual features. Summary of the Invention

[0006] To solve the above technical problems, the present invention proposes a ship image track extraction method for discontinuous interference of visual features. DeepSnake (Part 1) and YOLOX are used as detection algorithms, DeepSnake (Part 2) is used as the segmentation algorithm, and the improved Bytetrack algorithm is used as the tracking algorithm. An instance segmentation and tracking trajectory extraction framework is established. The Bytetrack algorithm structure is improved in combination with the ship motion characteristics. The data fusion method is used to improve the ship detection effect of the detection algorithm. The improved Bytetrack algorithm can obtain better trajectory extraction effect in scenarios with discontinuous ship visual features, providing protection and technical support for port and shipping traffic safety.

[0007] To achieve the above object, the present invention provides a ship image track extraction method oriented to discontinuous interference of visual features, comprising:

[0008] Collecting ship images, the ship images including: images of scenes where the ship is obstructed and images of scenes where the ship is not obstructed;

[0009] Detecting the ship image according to the DeepSnake model and the YOLOX model to obtain the ship position in the image sequence;

[0010] Inputting the position of the ship and the detection result of the DeepSnake model into the DeepSnake model to obtain the outer contour of the ship;

[0011] The trajectory of the ship is tracked based on the outer contour and the position of the ship to obtain the motion trajectory of the ship in the image sequence.

[0012] Optionally, before detecting the ship image according to the DeepSnake model, the method includes:

[0013] Performing linear interpolation processing on the ship image, and performing pixel normalization operation on the ship image after the linear interpolation processing.

[0014] Optionally, obtaining the position of the ship in the image sequence includes:

[0015] Performing feature extraction on the ship image using the DeepSnake model to obtain a ship feature map, and obtaining a first preliminary ship position group of the ship based on the ship feature map;

[0016] Detecting the ship image according to the YOLOX model to obtain a second preliminary ship position group of the ship;

[0017] The ship position is acquired based on the first preliminary ship position group and the second preliminary ship position group.

[0018] Optionally, fusing the first preliminary ship position group and the second preliminary ship position group includes:

[0019] performing maximum value suppression processing on the second preliminary ship position group to obtain a third preliminary ship position group;

[0020] Taking the intersection of the first preliminary ship position group and the third preliminary ship position group to obtain a first determined ship position group;

[0021] comparing the confidences of the detection frames of the third preliminary ship position group that are not in the intersection with a preset minimum confidence, retaining the detection frames with confidences greater than the preset minimum confidence, and obtaining a second determined ship position group;

[0022] performing an intersection-and-combination process on the first determined ship position group and the second determined ship position group using the third preliminary ship position group to obtain a third determined ship position group;

[0023] The first determined ship position group, the second determined ship position group and the third determined ship position group are fused.

[0024] Optionally, obtaining the outer contour of the ship includes:

[0025] The ship position is mapped onto the ship feature map and combined, and the DeepSnake model is used to extract the contour of the mapping combination result to obtain the outer contour.

[0026] Optionally, obtaining the motion trajectory of the ship in the image sequence includes:

[0027] The improved ByteTrack multi-target tracking algorithm based on the outer contour is introduced to track the position of the ship and obtain a preliminary motion trajectory. The preliminary motion trajectory includes: successfully tracked trajectories and unsuccessfully tracked trajectories. The expression of the ship's motion state in the preliminary motion trajectory is:

[0028]

[0029] Where x and y represent the center coordinates of the ship's position, a is the ratio of the height to the width of the ship image, and h is the height of the ship image. They represent the first-order derivatives of the corresponding parameters respectively;

[0030] Determining whether the outer contour of the ship in the unsuccessfully tracked trajectory is blocked, and if blocked, extracting the unsuccessfully tracked trajectory using a Kalman filter;

[0031] The extracted trajectory is combined with the successfully tracked trajectory to obtain the complete motion trajectory.

[0032] Optionally, obtaining a preliminary motion trajectory includes:

[0033] Initializing the Kalman filter using the first frame position detection value in the image sequence where the ship position is located;

[0034] Using the Kalman filter to predict and update the position of the ship, obtaining a position prediction value and a corresponding outer contour position of the ship in each frame of the image sequence;

[0035] ID matching is performed on the position prediction value of each frame in the image sequence and the corresponding outer contour position to obtain the preliminary motion trajectory.

[0036] Optionally, predicting and updating the ship position using the Kalman filter includes:

[0037] Based on the position detection value of the previous frame of the ship position, the Kalman filter is used to predict the position prediction value of the next frame of the ship position; the expression of the position prediction value of the next frame is:

[0038]

[0039] Among them, P i is the ship’s previous position detection value, is the next frame position prediction value output by the Kalman filter, A is the state transfer matrix;

[0040] Obtain the error covariance matrix of the next frame position prediction value; the expression of the error covariance matrix is:

[0041]

[0042] in, is the error covariance matrix of the ship's next frame position prediction value, L i is the error covariance matrix of the position detection value of the ship in one frame, Q is the covariance matrix of the process excitation noise, and T is the transpose symbol in matrix theory;

[0043] Get the Kalman gain of the next frame position detection value; the expression of the Kalman gain is:

[0044]

[0045] Where K i+1 is the Kalman gain, R is the noise matrix, and H is the measurement matrix;

[0046] The next frame position prediction value is updated according to the error covariance matrix and the Kalman gain; the expression of the updated next frame position prediction value is:

[0047]

[0048] Among them, P i+1 is the updated predicted value of the next frame position of the ship, L i+1 is the error covariance matrix of the updated ship in the next frame, x i+1 is the next frame position detection value of the ship, and I is the unit matrix.

[0049] Optionally, performing ID matching on the position prediction value of each frame in the image sequence and the corresponding outer contour position includes:

[0050] All detection values ​​of the ship position in the image sequence are divided into high-score detection frames and low-score detection frames according to a preset threshold;

[0051] Perform a first ID match on the high-scoring detection frame to obtain the predicted value and detection value of the successful match. a 、Unmatched prediction value Utrack a , Unmatched detection value Udet a ; The expression matching the first ID is:

[0052]

[0053] Where α is the detected value of the ship's outer contour position in each frame of the image sequence, β is the predicted value of the ship's outer contour position in each frame of the image sequence, and IOU(α,β) is the intersection-over-union distance between the detected value and the predicted value of the ship's outer contour position.

[0054] Perform a second ID match on the low-scoring detection frame to obtain the predicted value and detection value of the successful match at this stage. b 、Unmatched prediction value Utrack b , Unmatched detection value Udet b ; The expression matching the second ID is:

[0055]

[0056] Among them, θ is the ship appearance position detection value of the low-score detection frame in the image sequence, γ is the ship appearance position prediction value in the image sequence, and IOU (θ, γ) is the ship appearance position detection value of the low-score detection frame and Utrack a The intersection-over-union (IoU) distance between the predicted values ​​of the ship's appearance position;

[0057] Udet a Perform appearance position intersection and distance matching with the uncertain tracker to obtain the predicted value and detection value of the successful match at this stage. c 、Unmatched prediction value Utrack c , Unmatched detection value Udet c ;

[0058] Matched a 、Matched b 、Matched c The matched position information is used as the observation value of the corresponding tracker to update the Kalman algorithm; for Utrack b Recorded as Lost state tracker; Utrack c and Udet b Delete; c Determine it as a new ID, regard the new ID as the ship that appears for the first time in the image sequence, create a new ship motion trajectory, reinitialize the Kalman filter based on the position detection value of the ship that appears for the first time, and reuse the Kalman filter to predict and update the ship position.

[0059] Optionally, determining whether the outer contour of the ship in the unsuccessfully tracked trajectory is blocked includes:

[0060] For the Lost state tracker, according to its historical appearance profile information and Matched a 、Matched b 、Matched c The Hungarian algorithm of intersection and union and distance is used to determine whether there is occlusion. If it is occluded, it is marked as a Block state tracker, otherwise it is still the Lost state tracker;

[0061] The extraction of the unsuccessfully tracked trajectory using the Kalman algorithm includes:

[0062] For the Block state tracker, the confidence level of Kalman is adjusted according to the number of frames it is tracked. For a Block state tracker whose tracking frame number m is less than n frames, the Kalman prediction of the corresponding tracker will be obtained. and Multiply separately Get new Perform Kalman update, where n is the manually set frame value and m is the number of frames that the program believes the ship target is tracked. Each time the tracker successfully matches the ship detection frame, m will increase by one. When n ≥ m, Will be assigned a value of 1.

[0063] Compared with the prior art, the present invention has the following advantages and technical effects:

[0064] (1) The DeepSnake, YOLOX and Bytetrack algorithms were integrated and improved to obtain better ship detection, ship segmentation and ship tracking effects. A ship image track extraction method oriented to the discontinuous interference of visual features was proposed, and high-precision ship trajectory extraction and analysis were performed in complex scenarios with discontinuous ship visual features.

[0065] (2) This invention fuses the DeepSnake and YOLOX algorithms. First, the ship position obtained by the YOLOX detection algorithm is fused with the ship position obtained by the DeepSnake target detection algorithm. Second, the fusion result is mapped back to the feature map obtained by the DeepSnake algorithm to perform ship contour regression. Similarly, this invention also improves the Bytetrack multi-target tracking algorithm by introducing the judgment of the occlusion state and further utilizing the Kalman algorithm to obtain more accurate ship trajectories. This not only improves the accuracy of extracting ship navigation trajectories from shipborne surveillance video data, but also improves its robustness in complex sea conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0067] Figure 1 This is a flow chart of a method for extracting ship tracks from images according to an embodiment of the present invention;

[0068] Figure 2 This is a diagram of a ship detection framework of a ship image track extraction method according to an embodiment of the present invention;

[0069] Figure 3 This is a schematic diagram of a network structure for extracting a ship's exterior contour according to an embodiment of the present invention;

[0070] Figure 4 This is a ship trajectory extraction framework diagram according to an embodiment of the present invention;

[0071] Figure 5 This is a schematic diagram of ship detection results according to an embodiment of the present invention;

[0072] Figure 6 Schematic diagram of a ship trajectory according to an embodiment of the present invention. DETAILED DESCRIPTION

[0073] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0074] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0075] Example

[0076] This embodiment provides a ship image track extraction method for non-continuous interference of visual features, including:

[0077] Acquiring ship images, wherein the ship images include scene images where the ship is obstructed and scene images where the ship is not obstructed;

[0078] The ship images are detected based on the DeepSnake model and the YOLOX model to obtain the ship positions in the image sequence. The ship images used are obtained from the ship video. The images are acquired in sequence according to the order (number of frames) of the images appearing in the video to obtain the image sequence.

[0079] Inputting the ship position and the detection result of the DeepSnake model into the YOLOX model to obtain the outer contour of the ship;

[0080] The trajectory of the ship is tracked based on the outer contour and the position of the ship to obtain the motion trajectory of the ship in the image sequence.

[0081] Furthermore, obtaining the position of the ship in the image sequence includes:

[0082] The DeepSnake model is used to extract features from the ship images to obtain a ship feature map of the image sequence, and a first preliminary ship position group of the ship is obtained based on the ship feature map of the image sequence; the ship images are detected according to the YOLOX model to obtain a second preliminary ship position group of the ship; the first preliminary ship position group and the second preliminary ship position group are fused to obtain the ship position.

[0083] Furthermore, obtaining the outer contour of the ship includes:

[0084] The ship position is mapped onto the ship feature map of the image sequence and combined, and the contour of the mapping and combination result is extracted using the DeepSnake model to obtain the outer contour.

[0085] Furthermore, obtaining the motion trajectory of the ship in the image sequence includes:

[0086] The improved ByteTrack multi-target tracking algorithm based on the outer contour is introduced to track the position of the ship and obtain a preliminary motion trajectory. The preliminary motion trajectory includes: successfully tracked trajectories and unsuccessfully tracked trajectories. The expression of the ship's motion state in the preliminary motion trajectory is:

[0087]

[0088] Where x and y represent the center coordinates of the ship's position, a is the ratio of the height to the width of the ship image, and h is the height of the ship image. They represent the first-order derivatives of the corresponding parameters respectively;

[0089] determining whether the outer contour of the ship in the unsuccessfully tracked trajectory is blocked, and if blocked, extracting the unsuccessfully tracked trajectory using a Kalman algorithm;

[0090] The extracted trajectory is combined with the successfully tracked trajectory to obtain the complete motion trajectory.

[0091] Furthermore, obtaining the preliminary motion trajectory includes:

[0092] Initializing the Kalman filter using the first frame position detection value in the image sequence where the ship position is located;

[0093] Using the Kalman filter to predict and update the position of the ship, obtaining a position prediction value and a corresponding outer contour position of the ship in each frame of the image sequence;

[0094] ID matching is performed on the position prediction value of each frame in the image sequence and the corresponding outer contour position to obtain the preliminary motion trajectory.

[0095] Specifically, this embodiment provides a ship image track extraction method oriented to the discontinuous interference of visual features, such as Figure 1 The following steps are shown:

[0096] (1) Collecting maritime-related ship-occluded scene image data on-site by shooting ship images with cameras in offshore waters;

[0097] Images of maritime-related ship obstruction scenes are collected by using ship-borne cameras in offshore waters, cooperating with shipping companies to obtain ship navigation records, and obtaining ship-borne surveillance videos provided by maritime regulatory authorities.

[0098] (2) Ship detection is performed on the input maritime surveillance video based on the DeepSnake instance segmentation algorithm and the YOLOX target detection algorithm;

[0099] (2.1) Before using DeepSnake to detect ships, the image data must be preprocessed to normalize the image data:

[0100] First, based on linear interpolation, the image is resized to a suitable size to ensure that the length and width are divisible by 32 (so that the detection frame can be mapped back to the feature map later):

[0101] input w =(img w |31)+1

[0102] input h =(img h |31)+1

[0103] Where img w and img h Indicates the width and height of the original image, input w and input h Indicates the size of the image input to the network.

[0104] Secondly, the pixels of the ship image are normalized to ensure that the data in all dimensions are within a range of variation.

[0105] (2.2) Use the CenterNet network from the first part of DeepSnake to extract ship features and predict the preliminary ship position group 1 in the image sequence;

[0106] (2.3) Predicting preliminary ship position cluster 2 in the image sequence using the YOLOX detection algorithm;

[0107] (3) The ship detection results obtained by the two algorithms are fused to remove false detections and missed detections as much as possible, and the final output of the ship detection frame in the image sequence is obtained, that is, the current position of the ship in the image sequence;

[0108] (3.1) Perform non-maximum suppression on the ship position cluster 2 obtained by the YOLOX detection algorithm to remove most of the false detections and obtain a better ship position cluster 3;

[0109] (3.2) Take the intersection of ship position cluster 1 and ship position cluster 3 obtained by the CenterNet network to ensure that the low-scoring but effective ship detection frame is retained, and record it as the determined ship position cluster 1;

[0110] (3.3) For the remaining ship position groups in ship position group 3, retain those with a confidence level greater than the minimum confidence level Min. score The detection frame is recorded as the determined ship position group 2; the lowest confidence level is a manually set value, which can be obtained based on experience. If you fully believe in the target detection results, you can set it lower.

[0111] (3.4) First, the score of the detection frame of ship position group 3 to the determined ship position group 1 and the determined ship position group 2 is lower than Min score Ship score score = score + Min score , which is convenient for post-processing. Ship position group 3 is then used to complement the determined ship position group 1 and the determined ship position group 2. Specifically, an intersection-over-union (IOU) comparison is performed first, and the detection frames that do not overlap with the determined ships are used as the determined ship position group 3 to reduce missed detections. Finally, the three determined ship position groups are combined to obtain the final determined ship positions in the image sequence.

[0112] (4) The fused ship detection frame result and the ship features obtained by the DeepSnake instance segmentation algorithm are input into the contour regression network of the DeepSnake instance segmentation algorithm to obtain the outer contour of the ship;

[0113] The final determined image sequence ship position is mapped to the position on the image sequence ship feature map obtained in (2.2), and then the image sequence ship features and the corresponding ship positions are passed to the second part of the DeepSnake contour regression network to obtain the appearance contour of the ship.

[0114] (5) Using the outer contour of the ship and the position detection value of the ship, the trajectory of the ship is tracked based on the improved ByteTrack multi-target tracking algorithm, and the motion trajectory of the ship in the image sequence is preliminarily obtained; wherein, the improved ByteTrack multi-target tracking algorithm introduces the outer contour information; it is determined whether the outer contour of the ship in the unsuccessful track is blocked. If it is blocked, the Kalman filter is used to extract the unsuccessful track;

[0115] (5.1) The ship's motion state is modeled based on the Kalman filter. The Kalman filter is initialized based on the detected value of the ship's position in the first frame of the image sequence. The expression describing the ship's motion state is as follows:

[0116]

[0117] Where x and y represent the center coordinates of the ship's position, a is the ratio of the height to the width of the ship image, and h is the height of the ship image. They represent the first-order derivatives of the corresponding parameters respectively;

[0118] (5.2) Based on the ship position detection value of the previous frame in the image sequence, the Kalman filter is used to generate the ship position prediction value of the next frame in the image sequence, and the ship position detection value in the next frame is combined to update the ship position prediction value;

[0119] The process of predicting and updating the ship's position based on Kalman filtering is as follows;

[0120] First, based on the position detection value of the ship in the previous frame in the image sequence, the position prediction value of the ship in the next frame is predicted. The expression is as follows:

[0121]

[0122] Where, P i Indicates the position detection value of the ship in the previous frame, It means that the Kalman filter outputs the position prediction value of the next frame, and A is the state transfer matrix;

[0123] Secondly, the error covariance matrix of the predicted value of the ship's position in the next frame in the image sequence is calculated, and its expression is as follows:

[0124]

[0125] Where, The error covariance matrix of the predicted value of the ship's position in the next frame, L i represents the error covariance matrix of the ship's position detection value in the previous frame, and Q is the covariance matrix of the process excitation noise;

[0126] Again, the Kalman gain is calculated based on the position detection value of the ship in the next frame in the image sequence, and its expression is as follows:

[0127] K i+1 =L i+1 H T (HL i+1 H T +R) -1

[0128] Where K i+1 represents the Kalman gain, R represents the noise matrix, and H represents the measurement matrix;

[0129] Finally, the predicted value of the ship's position in the next frame in the image sequence is updated, and its expression is as follows:

[0130]

[0131] Where, P i+1 Indicates the updated predicted position of the ship in the next frame, L i+1 represents the error covariance matrix of the ship in the next frame after the update, x i+1 is the position detection value of the ship in the next frame, and I is the unit matrix;

[0132] (5.3) Based on the Kalman filter, the predicted ship position and the predicted ship outline position are obtained for each frame in the image sequence, and the outline position estimation process (appearance outline + the difference between the center of the historical position and the predicted center);

[0133] (5.4) Based on the predicted ship position and appearance contour position of each frame in the image sequence obtained by the Kalman filter, ID matching is performed:

[0134] (5.4.1) The image sequence ship position detection values ​​are divided into high-score detection frames H socre and low-score detection box L score , processed separately; divided according to the threshold set in advance;

[0135] (5.4.2) For H score For the first ID matching, the Hungarian algorithm based on the intersection-over-union (IOU) distance of the appearance contour is used for the first matching. Its expression is as follows:

[0136]

[0137] In the formula, α represents the detected value of the ship's appearance contour position in each frame of the image sequence, β represents the predicted value of the ship's appearance contour position in each frame of the image sequence, and IOU (α, β) represents the intersection-over-union (IOU) distance between the detected value and the predicted value of the ship's appearance contour position. Then, the Hungarian algorithm is used for matching, and this step will obtain Matched a 、Utrack a and Udet a They represent the prediction value and detection value of successful matching, the prediction value of unsuccessful matching, and the detection value of unsuccessful matching, respectively.

[0138] (5.4.3) For L score For the second ID matching, the Hungarian algorithm based on the intersection-over-union (IOU) distance of position information is used for the second matching. The expression is as follows:

[0139]

[0140] Where, θ represents the L scoreThe ship's appearance position detection value, γ represents the Utrack obtained by (5.4.2) in the image sequence a The predicted value of the ship's appearance position, IOU (θ, γ) represents L score Ship appearance position detection value and Utrack a The Intersection-over-Union (IOU) distance between the predicted values ​​of the ship's appearance position is then used to obtain the Matched b 、Utrack b and Udet b , which respectively represent the prediction value and detection value of successful matching, the prediction value of unsuccessful matching and the detection value of unsuccessful matching at this stage.

[0141] (5.4.4) For Udet obtained from (5.4.3) a The appearance position intersection-over-union (IOU) distance matching is performed with the uncertain tracker (the high-scoring detection frame that has only been detected once up to the current frame) to obtain the Matched c 、Utrack c and Udet c , which respectively represent the prediction value and detection value of successful matching, the prediction value of unsuccessful matching and the detection value of unsuccessful matching at this stage.

[0142] (5.4.5) Integrate the previous matching results, Matched a、b、c The matched position information is used as the observation value of the corresponding tracker to update the Kalman algorithm; for Utrack b Recorded as Lost state tracker; for Utrack c and Udet b Will be deleted; for Udet c The ID is determined as a new one, and it is considered as the first ship to appear in the image sequence. A new ship motion trajectory is created, and the Kalman filter is reinitialized based on the ship position detection value, and the process returns to the above step (5.1).

[0143] (6) For ships that are not tracked successfully, determine whether they are blocked by the ship based on the intersection-over-union (IOU) distance and the outline obtained by (5);

[0144] For the Lost state tracker obtained by (5.4.5), according to its historical appearance profile information and the Matched a、b、c The Hungarian algorithm is used to determine whether an object is occluded using the Intersection-over-Union (IOU) distance of the appearance contour information. If occluded, it is recorded as a Blocked state; otherwise, it remains in a Lost state.

[0145] (7) For the obscured ship, the Kalman algorithm is used to further extract the complete ship motion trajectory.

[0146] The Block state tracker obtained by (6) adjusts the Kalman confidence level according to the number of frames it is tracking. For a Block state tracker whose tracking frame number m is less than n frames, the Kalman prediction of the corresponding tracker obtained by (5.3) is and Multiply separately Get new Perform Kalman update, where n is the manually set frame value, and m is the number of frames that the program believes the ship target is tracked. Each time the tracker successfully matches the ship detection frame, m will increase by one. When n ≥ m, Will be assigned a value of 1.

[0147] A specific application example of the present invention is described in detail below:

[0148] In this embodiment, a ship image track extraction method for discontinuous interference of visual features is implemented on a server with an operating system of Ubuntu 20.04OS, a CPU of Intel(R) Xeon(R) Gold 6230R, and a GPU of NVIDIA Quadro RTX A5000. The experimental platform uses Python 3.7 to deploy the experimental program.

[0149] This embodiment includes three discontinuous scenes of ship visual features, namely (1) the scene of transition from no occlusion to partial occlusion and then to no occlusion; (2) the scene of transition from no occlusion to full occlusion; (3) the scene of transition from no occlusion to full occlusion and then to no occlusion; and they are marked as scene 1, scene 2, and scene 3 respectively. The detailed information of the shipborne surveillance video is shown in Table 1, which shows the shipborne surveillance video information taken under different sea conditions.

[0150] Table 1

[0151]

[0152]

[0153] In order to quantitatively evaluate the ship trajectory extraction method for complex sea conditions proposed in this invention, experiments were conducted under the above three different sea conditions, and the Detectors instance segmentation algorithm was combined with different multi-target tracking algorithms (ByteTrack, DeepSort) for comparison, using evaluation indicators such as MOTA, IDF1, IDR, and IDP. In the experiment, a tracking ID was assigned to each target ship, which was used to finally extract the motion trajectory of the ship's navigation. Among them, MOTA represents the accuracy of multi-target ship tracking; IDP represents the recognition accuracy of the ship tracking ID; IDR represents the recognition recall rate of the ship tracking ID; IDF1 represents the average ratio of the ship tracking ID, and its calculation formula is as follows:

[0154]

[0155] Where FN represents the number of times the ship tracking ID is lost, FP represents the number of times the ship tracking ID is wrong, φ represents the number of times the ship tracking ID jumps, T represents the ideal number of times the ship tracking ID is correct, and TP represents the actual number of times the ship tracking ID is correct.

[0156] Table 2

[0157]

[0158]

[0159] Scenario 1 transitions from unobstructed to partially obstructed, and then back to unobstructed. In this scenario, the ship's motion varies significantly, and there's also wave interference. In addition, a large number of target ships are located near the sea-sky line in the surveillance footage. The experimental results, as shown in Table 2, show the performance of different algorithms for extracting ship tracks in Scenario 1. The proposed ship track extraction method achieves superior ship tracking results, significantly improving on the MOTA metric compared to other methods and outperforming other algorithms in other metrics.

[0160] Table 3

[0161]

[0162] Scenario 2 transitions from unobstructed to fully obstructed conditions. In this scenario, a small vessel and a large vessel are traveling towards each other, encountering each other. The experimental results, as shown in Table 3, show that the ship trajectory extraction method proposed in this paper achieves excellent ship tracking results, accurately extracting the motion trajectory of obstructed vessels. It significantly improves on the MOTA metric compared to other methods and also outperforms other algorithms in other metrics.

[0163] Table 4

[0164]

[0165] Scenario 3 transitions from unobstructed to fully obstructed conditions. In this scenario, a small vessel and a large vessel are traveling towards each other, encountering each other. Experimental results, as shown in Table 4, show that the ship trajectory extraction method proposed in this paper achieves excellent ship tracking results. It can maintain ID changes when a ship returns from obstruction to unobstructed status. It also outperforms other algorithms in other metrics.

[0166] The present invention proposes a ship image track extraction method oriented to the discontinuous interference of visual features. The overall framework can be divided into three small frameworks.

[0167] The first is the ship detection part based on DeepSnake and YOLOX, such as Figure 2 As shown in the figure, maritime image data is collected on-site by shooting ship images with cameras in offshore waters; ship detection is performed on the input maritime surveillance video based on the target detection part of the first stage of the DeepSnake instance segmentation algorithm and the YOLOX target detection algorithm. In this step, the target detection part of the first stage of the DeepSnake instance segmentation algorithm obtains ship detection result 1 and ship image features, and the YOLOX target detection algorithm obtains ship detection result 2; the detection results obtained by the two algorithms are fused and processed, and the following is obtained: Figure 5 Final vessel inspection results shown;

[0168] The second is based on the DeepSnake contour extraction network part, such as Figure 3 As shown in the figure, this part will take the final ship detection results and ship image features obtained by the ship detection part based on DeepSnake and YOLOX as input, and will obtain the appearance contour information of the ship through the backbone network part, fusion part and prediction part in sequence;

[0169] Finally, the ship tracking part is based on the improved Bytetrack, such as Figure 4As shown, the ship detection target obtained by the detection part and the ship appearance contour information of the contour extraction network part are taken as input; they are split into high-score detection frames and low-score detection frames; first, the high-score detection frame is associated with the ship target obtained by the trajectory prediction obtained previously, and this step is described in (5.4.2); secondly, the association 2 operation is performed, and this step is described in (5.4.3); thirdly, the association 3 operation is performed, and this step is described in (5.4.4); for the ship detection result that fails to match in association 3, a new trajectory is created for it, and for the ship position prediction value that fails to match in association 3, its motion trajectory is terminated; the successful matching results obtained by the three association steps are merged with the newly generated trajectory to obtain a preliminary ship motion trajectory; for the ship position prediction value that fails to match in association 2, the occlusion state will be judged with the preliminary ship motion trajectory. If it is an occluded ship, the Kalman prediction value will be used as the ship detection value, and the Kalman update will be performed to further obtain the final ship navigation trajectory as shown in Figure 6 The method provided by the present invention has good tracking performance and can obtain good pixel-level ship trajectory extraction effect in complex sea conditions where the ship's visual features are lost, providing safety assurance and technical support for the ship's autonomous navigation decision-making.

[0170] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A ship image track extraction method for non-continuous interference of visual features, characterized by: include: Collecting ship images, the ship images including: images of scenes where the ship is obstructed and images of scenes where the ship is not obstructed; Detecting the ship image using the object detection network portion of the first stage of the DeepSnake model to obtain a ship feature map and a first preliminary ship position group of the ship; Detecting the ship image according to the YOLOX model to obtain a second preliminary ship position group of the ship; fusing the first preliminary ship position group and the second preliminary ship position group to obtain ship positions; Mapping the ship position onto the ship feature map and combining them, and performing contour extraction on the mapping and combining result using the contour extraction network portion of the second stage of the DeepSnake model to obtain the outer contour; The trajectory of the ship is tracked based on the outer contour and the position of the ship to obtain the motion trajectory of the ship in the image sequence.

2. The ship image track extraction method oriented to discontinuous interference of visual features according to claim 1 is characterized in that: Before detecting the ship image according to the DeepSnake model, the following steps are included: Performing linear interpolation processing on the ship image, and performing pixel normalization operation on the ship image after the linear interpolation processing.

3. The ship image track extraction method oriented to discontinuous interference of visual features according to claim 1 is characterized in that: The fusing of the first preliminary ship position group and the second preliminary ship position group includes: performing maximum value suppression processing on the second preliminary ship position group to obtain a third preliminary ship position group; Taking the intersection of the first preliminary ship position group and the third preliminary ship position group to obtain a first determined ship position group; comparing the confidences of the detection frames of the third preliminary ship position group that are not in the intersection with a preset minimum confidence, retaining the detection frames with confidences greater than the preset minimum confidence, and obtaining a second determined ship position group; performing an intersection-and-combination process on the first determined ship position group and the second determined ship position group using the third preliminary ship position group to obtain a third determined ship position group; The first determined ship position group, the second determined ship position group and the third determined ship position group are fused.

4. The ship image track extraction method oriented to discontinuous interference of visual features according to claim 1 is characterized in that: Obtaining the motion trajectory of the ship in the image sequence includes: The improved ByteTrack multi-target tracking algorithm based on the outer contour is introduced to track the position of the ship and obtain a preliminary motion trajectory. The preliminary motion trajectory includes: successfully tracked trajectories and unsuccessfully tracked trajectories. The expression of the ship's motion state in the preliminary motion trajectory is: Where x and y represent the center coordinates of the ship's position, a is the ratio of the height to the width of the ship image, and h is the height of the ship image. They represent the first-order derivatives of the corresponding parameters respectively; Determining whether the outer contour of the ship in the unsuccessfully tracked trajectory is blocked, and if blocked, extracting the unsuccessfully tracked trajectory using a Kalman filter; The extracted trajectory is combined with the successfully tracked trajectory to obtain the complete motion trajectory.

5. The ship image track extraction method oriented to discontinuous interference of visual features according to claim 4 is characterized in that: Obtaining a preliminary motion trajectory includes: Initializing the Kalman filter using the first frame position detection value in the image sequence where the ship position is located; Using the Kalman filter to predict and update the position of the ship, obtaining a position prediction value and a corresponding outer contour position of the ship in each frame of the image sequence; ID matching is performed on the position prediction value of each frame in the image sequence and the corresponding outer contour position to obtain the preliminary motion trajectory.

6. The ship image track extraction method oriented to discontinuous interference of visual features according to claim 5 is characterized in that: Predicting and updating the ship position using the Kalman filter includes: Based on the position detection value of the previous frame of the ship position, the Kalman filter is used to predict the position prediction value of the next frame of the ship position; the expression of the position prediction value of the next frame is: Among them, P i is the ship’s previous position detection value, is the next frame position prediction value output by the Kalman filter, A is the state transfer matrix; Obtain the error covariance matrix of the next frame position prediction value; the expression of the error covariance matrix is: in, is the error covariance matrix of the ship's next frame position prediction value, L i is the error covariance matrix of the position detection value of the ship in one frame, Q is the covariance matrix of the process excitation noise, and T is the transpose symbol in matrix theory; Get the Kalman gain of the next frame position detection value; the expression of the Kalman gain is: Where K i+1 is the Kalman gain, R is the noise matrix, and H is the measurement matrix; The next frame position prediction value is updated according to the error covariance matrix and the Kalman gain; the expression of the updated next frame position prediction value is: Among them, P i+1 is the updated predicted value of the next frame position of the ship, L i+1 is the error covariance matrix of the updated ship in the next frame, x i+1 is the next frame position detection value of the ship, and I is the unit matrix.

7. The ship image track extraction method oriented to discontinuous interference of visual features according to claim 6 is characterized in that: Performing ID matching on the position prediction value of each frame in the image sequence and the corresponding outer contour position includes: All detection values ​​of the ship position in the image sequence are divided into high-score detection frames and low-score detection frames according to a preset threshold; Perform a first ID match on the high-scoring detection frame to obtain the predicted value and detection value of the successful match. a 、Unmatched prediction value Utrack a , Unmatched detection value Udet a ; The expression matching the first ID is: Where α is the detected value of the ship's outer contour position in each frame of the image sequence, β is the predicted value of the ship's outer contour position in each frame of the image sequence, and IOU(α,β) is the intersection-over-union distance between the detected value and the predicted value of the ship's outer contour position. Perform a second ID match on the low-scoring detection frame to obtain the predicted value and detection value of the successful match at this stage. b 、Unmatched prediction value Utrack b , Unmatched detection value Udet b ; The expression matching the second ID is: Among them, θ is the ship appearance position detection value of the low-score detection frame in the image sequence, γ is the ship appearance position prediction value in the image sequence, and IOU (θ, γ) is the ship appearance position detection value of the low-score detection frame and Utrack a The intersection-over-union (IoU) distance between the predicted values ​​of the ship's appearance position; Udet a Perform appearance position intersection and distance matching with the uncertain tracker to obtain the predicted value and detection value of the successful match at this stage. c 、Unmatched prediction value Utrack c , Unmatched detection value Udet c ; Matched a 、Matched b 、Matched c The matched position information is used as the observation value of the corresponding tracker to update the Kalman algorithm; for Utrack b Recorded as Lost state tracker; Utrack c and Udet b Delete; c Determine it as a new ID, regard the new ID as the ship that appears for the first time in the image sequence, create a new ship motion trajectory, reinitialize the Kalman filter based on the position detection value of the ship that appears for the first time, and reuse the Kalman filter to predict and update the ship position.

8. The ship image track extraction method oriented to discontinuous interference of visual features according to claim 7 is characterized in that: Determining whether the outer contour of the vessel in the unsuccessfully tracked trajectory is blocked includes: For the Lost state tracker, according to its historical appearance profile information and Matched a 、Matched b 、Matched c The Hungarian algorithm of intersection and union and distance is used to determine whether there is occlusion. If it is occluded, it is marked as a Block state tracker, otherwise it is still the Lost state tracker; The extraction of the unsuccessfully tracked trajectory using the Kalman algorithm includes: For the Block state tracker, the confidence level of Kalman is adjusted according to the number of frames it is tracked. For a Block state tracker whose tracking frame number m is less than n frames, the Kalman prediction of the corresponding tracker will be obtained. and Multiply separately Get new Perform Kalman update, where n is the manually set frame value and m is the number of frames that the program believes the ship target is tracked.

Images