A Ship Target Tracking Method, System and Medium Based on Visual Twin Coding

Abstract

The present invention discloses a ship target tracking method, system and medium based on visual twin coding, including: acquiring a ship video sequence, performing target detection on the ship video sequence to obtain detection frames corresponding to each frame of image; updating and predicting the detection frames to obtain target prediction frames corresponding to each ship target in the current frame of image and target detection frames corresponding to each ship target in the next frame of image; constructing a cost matrix according to each target prediction frame and each target detection frame through a visual twin coding model and an intersection over union calculation formula, and then performing target association according to the cost matrix through the Hungarian algorithm to obtain an initial ship target tracking result; classifying the detection frames to obtain ship target types, comparing the similarity between disappeared targets and newly emerged targets, and performing target re-identification according to the similarity comparison result to obtain a secondary ship target tracking result. The present invention can achieve precise tracking of ship targets and can be widely applied to the field of target tracking technology.

Description

Technical Field

[0001] The present application relates to the technical field of target tracking, and in particular, to a ship target tracking method, system and medium based on visual twin coding. Background Art

[0002] With the continuous growth of the transportation demand in ports and inland waterways, ship monitoring technology plays an increasingly important role in traffic management and navigation safety guarantee. However, these areas usually have the characteristics of narrow waterways, heavy traffic and complex environment. Ships are easily affected by other ships, port facilities and terrain occlusion during navigation. The occlusion problem not only poses challenges to the continuous tracking and effective detection of ship targets, but also increases the risk of target misjudgment and loss, bringing potential threats to navigation safety.

[0003] Traditional target tracking algorithms such as SORT and DeepSORT are mainly based on Kalman filtering and data association technology, and have obvious deficiencies in complex environments with target occlusion. When the target is partially occluded, it is difficult to capture complete feature information, resulting in a significant decrease in target matching accuracy; when the target reappears after being completely occluded, there is a lack of an effective re-identification mechanism, and it is easy to misjudge the original target as a new target. These methods also have poor adaptability to dynamic environments, and their robustness and stability cannot be guaranteed in the face of complex scenarios such as multi-target occlusion and illumination changes in ports and inland waterways. Summary of the Invention

[0004] To solve the above technical problems, the purpose of the present invention is to provide a ship target tracking method, system and medium based on visual twin coding, which can improve the accuracy of ship target tracking and re-identification.

[0005] To achieve the above purpose, one aspect of the embodiments of the present application proposes a ship target tracking method based on visual twin coding, including the following steps:

[0006] Obtain a ship video sequence to be recognized, perform target detection on each frame image in the ship video sequence, and obtain detection frames corresponding to each frame of the image;

[0007] Update and predict the detection frames to obtain target prediction frames corresponding to each ship target in the current frame of the image and target detection frames corresponding to each ship target in the next frame of the image;

[0008] Construct a cost matrix according to each target prediction frame and each target detection frame through a pre-trained visual twin coding model and an intersection over union calculation formula, and then perform target association according to the cost matrix through the Hungarian algorithm to obtain the initial ship target tracking result;

[0009] Classify the detection boxes to obtain the ship target types, compare the similarity between the disappearing targets and the newly emerging targets in the ship target types, and then perform target re-identification according to the similarity comparison result to obtain the secondary tracking result of the ship target.

[0010] In some embodiments, constructing a cost matrix according to each of the target prediction boxes and each of the target detection boxes through the pre-trained visual twin coding model and the intersection over union calculation formula specifically includes:

[0011] Calculate the similarity between each of the target prediction boxes and each of the target detection boxes through the visual twin coding model to obtain a first similarity;

[0012] Perform a matching calculation between each of the target prediction boxes and each of the target detection boxes through the intersection over union calculation formula to obtain an intersection over union value;

[0013] Taking each ship target in the current frame of the image as a row and each ship target in the next frame of the image as a column, construct the cost matrix according to the first similarity and the intersection over union value.

[0014] In some embodiments, calculating the similarity between each of the target prediction boxes and each of the target detection boxes through the visual twin coding model to obtain a first similarity specifically includes:

[0015] Train the visual twin coding model based on a deep convolutional neural network;

[0016] Input each of the target prediction boxes and each of the target detection boxes into the visual twin coding model;

[0017] Extract features from the target prediction boxes to obtain first deep features, and extract features from the target detection boxes to obtain second deep features;

[0018] Calculate the similarity between the first deep features and the second deep features through the Mahalanobis distance calculation formula to obtain the first similarity.

[0019] In some embodiments, performing target association according to the cost matrix through the Hungarian algorithm to obtain the initial tracking result of the ship target specifically includes:

[0020] Determine the row minimum values of each row and the column minimum values of each column in the cost matrix;

[0021] Perform a subtraction operation on each row of the cost matrix and the row minimum value in each row, and perform a subtraction operation on each column in the cost matrix and the column minimum value in each column to obtain a first cost matrix;

[0022] Determine the number of independent zero elements and the matrix dimension of the first cost matrix;

[0023] When the number of independent zero elements is equal to the matrix dimension, obtain the initial tracking result of the ship target;

[0024] When the number of independent zero elements is greater than or less than the matrix dimension, adjust the first cost matrix until the number of independent zero elements is equal to the matrix dimension, and obtain the initial tracking result of the ship target.

[0025] In some embodiments, the ship target types include associated targets, disappearing targets, and newly emerging targets. The classification of the detection boxes to obtain the ship target types specifically includes:

[0026] Divide the detection boxes that are detected in both the current frame image and the previous frame image and successfully associated into the associated targets;

[0027] Divide the detection boxes that are detected in the current frame image and not detected in the next frame image into the disappearing targets;

[0028] Divide the detection boxes that are detected in the current frame image and not detected in the previous frame image into the newly emerging targets.

[0029] In some embodiments, the similarity comparison between the disappearing targets and the newly emerging targets in the ship target types, and then the target re-identification according to the similarity comparison result to obtain the secondary tracking result of the ship target specifically includes:

[0030] Store the detection boxes corresponding to the last frame images of the disappearing targets to obtain a disappearing target set;

[0031] Calculate the similarity between the newly emerging targets and each detection box in the disappearing target set through the visual twin coding model to obtain a second similarity;

[0032] Determine a similarity threshold, compare the second similarity with the similarity threshold, and then perform target re-identification according to the similarity comparison result to obtain the secondary tracking result of the ship target.

[0033] In some embodiments, the comparison of the second similarity with the similarity threshold, and then the target re-identification according to the similarity comparison result to obtain the secondary tracking result of the ship target specifically includes:

[0034] When the second similarity is greater than or equal to the similarity threshold, associate the newly emerging target with the corresponding detection box to obtain the secondary tracking result of the ship target;

[0035] When the second similarity is less than the similarity threshold, configure an identification number and generate a trajectory for the new target to obtain the secondary tracking result of the ship target.

[0036] To achieve the above object, on the other hand, an embodiment of the present application proposes a ship target tracking system based on visual twin coding, including:

[0037] A target detection module, configured to obtain a ship video sequence to be recognized, perform target detection on each frame image in the ship video sequence, and obtain a detection box corresponding to each frame of the image;

[0038] An update prediction module, configured to update and predict the detection box to obtain a target prediction box corresponding to each ship target in the current frame image and a target detection box corresponding to each ship target in the next frame image;

[0039] A target association module, configured to construct a cost matrix according to each target prediction box and each target detection box through a pre-trained visual twin coding model and an intersection over union calculation formula, and then perform target association according to the cost matrix through the Hungarian algorithm to obtain a primary tracking result of the ship target;

[0040] A target re-identification module, configured to classify the detection box to obtain a ship target type, compare the similarity between the disappeared target and the new target in the ship target type, and then perform target re-identification according to the similarity comparison result to obtain a secondary tracking result of the ship target.

[0041] To achieve the above object, on the other hand, an embodiment of the present application proposes an electronic device, where the electronic device includes a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for realizing connection communication between the processor and the memory. When the program is executed by the processor, it implements the ship target tracking method based on visual twin coding as described above.

[0042] To achieve the above object, on the other hand, an embodiment of the present application proposes a storage medium, where the storage medium is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the ship target tracking method based on visual twin coding as described above.

[0043] The beneficial effects of the present invention are as follows: The ship target tracking method, system and medium based on visual twin coding of the present invention construct a cost matrix by combining the intersection over union with the similarity calculated by the visual twin coding model, and then use the Hungarian algorithm to achieve multi-target association according to the cost matrix, which can effectively improve the accuracy of target matching, solve the technical problem that traditional methods cannot continuously track in the case of partial occlusion of ship targets, and achieve precise tracking of ship targets. Moreover, after the ship target is completely occluded or disappears, the ship target is classified, and the re-emerging target is re-identified with high precision, which can significantly improve the robustness and accuracy of target tracking, solve the target tracking problems in complex scenarios such as multi-ship intersection, occlusion and light change, and enhance the adaptability of the system to the dynamic environment. Description of the Drawings

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following introduces the drawings required to be used in the embodiments of the present invention. It should be understood that the drawings introduced below only conveniently and clearly show some embodiments of the technical solutions in the present invention. For those skilled in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0045] Figure 1 It is a flowchart of the steps of a ship target tracking method based on visual twin coding provided by an embodiment of the present invention;

[0046] Figure 2 It is a schematic diagram of the process of target detection provided by an embodiment of the present invention;

[0047] Figure 3 It is a schematic diagram of the encoder-decoder structure based on Transformer provided by an embodiment of the present invention;

[0048] Figure 4 It is an example diagram of update and prediction based on Kalman filter provided by an embodiment of the present invention;

[0049] Figure 5 It is a schematic diagram of the training result of the visual twin coding model provided by an embodiment of the present invention;

[0050] Figure 6 It is a schematic diagram of the process of similarity calculation provided by an embodiment of the present invention;

[0051] Figure 7 It is a schematic diagram of association based on the Hungarian algorithm provided by an embodiment of the present invention;

[0052] Figure 8 It is an example diagram of partial occlusion and complete occlusion provided by an embodiment of the present invention;

[0053] Figure 9An example diagram for re-identifying ship targets provided by an embodiment of the present invention;

[0054] Figure 10 A schematic flowchart of target re-identification provided by an embodiment of the present invention;

[0055] Figure 11 A schematic structural diagram of a ship target tracking system based on visual twin coding provided by an embodiment of the present invention;

[0056] Figure 12 A schematic hardware structure diagram of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0057] In order to make the objectives, technical solutions and advantages of the present application more clear and understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the appended claims.

[0058] It can be understood that the terms "first", "second", etc. used in the present application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if", "when" as used herein may be interpreted as "when...", "while...", or "in response to determining".

[0059] The terms "at least one", "multiple", "each", "any one", etc. used in the present application, at least one includes one, two or more than two, multiple includes two or more than two, each refers to each of the corresponding multiple, and any one refers to any one of the multiple.

[0060] With the continuous growth of transportation demands in ports and inland waterways, ship monitoring technology plays an increasingly important role in traffic management and navigation safety assurance. However, these areas usually have the characteristics of narrow waterways, heavy traffic, and complex environments. Ships are easily affected by other ships, port facilities, and terrain occlusion during navigation. The occlusion problem not only poses challenges to the continuous tracking and effective detection of ship targets, but also increases the risk of target misjudgment and loss, posing a potential threat to navigation safety.

[0061] Traditional object tracking algorithms such as SORT and DeepSORT are mainly based on Kalman filtering and data association techniques, and have obvious deficiencies in complex environments with object occlusion. When the object is partially occluded, it is difficult to capture complete feature information, resulting in a significant decrease in the accuracy of object matching; when the object reappears after being completely occluded, there is a lack of an effective re-identification mechanism, and the original object is easily misjudged as a new object. These methods also have poor adaptability to dynamic environments. In the face of complex scenarios such as multi-object occlusion and light changes in ports and inland waterways, the robustness and stability cannot be guaranteed.

[0062] Therefore, the embodiments of the present invention propose a ship target tracking method based on visual twin coding. By combining the intersection over union with the similarity calculated by the visual twin coding model to construct a cost matrix, and then using the Hungarian algorithm to achieve multi-object association according to the cost matrix, it can effectively improve the accuracy of object matching, solve the technical problem that traditional methods cannot continuously track in the case of partial occlusion of ship targets, and achieve precise tracking of ship targets. Moreover, after the ship target is completely occluded or disappears, the ship target is classified, and the re-emerging target is re-identified with high precision, which can significantly improve the robustness and accuracy of target tracking, solve the target tracking problem in complex scenarios such as multi-ship intersection, occlusion, and light changes, and enhance the adaptability of the system to dynamic environments.

[0063] Refer to Figure 1 , Figure 1 FIG. is a step flow chart of a ship target tracking method based on visual twin coding provided by the embodiments of the present invention. The embodiments of the present invention propose a ship target tracking method based on visual twin coding, and the method includes steps S101 to S104:

[0064] S101. Obtain a ship video sequence to be recognized, perform object detection on each frame image in the ship video sequence, and obtain detection frames corresponding to each frame image;

[0065] In some optional embodiments, first process the ship video sequence, use the DETR algorithm to detect each frame image target in the ship video sequence, obtain the position category of the ship target and the corresponding detection frame, and assign a corresponding identification number (ID number) to the ship target. The DETR algorithm adopts the encoder-decoder structure of Transformer, realizes unique prediction through bipartite matching loss, and improves efficiency by using parallel decoding. It regards object detection as a direct set prediction problem, simplifies the training process, and avoids non-maximum suppression and anchor box design.

[0066] Specifically, as Figure 2 shown is a schematic diagram of the target detection process, as Figure 3The figure shows a schematic diagram of the Transformer-based encoder-decoder structure. First, each frame image in the ship video sequence is input into the backbone network of ResNet50 for feature extraction to obtain the feature sequence of each frame of the ship driving image. After extracting the feature sequence information of each frame of the ship driving image, it is combined with the position encoding information and input into the Transformer encoder-decoder structure to solve the problem of the loss of the spatial distribution between pixels in the image. In the Transformer encoder-decoder structure, a three-layer perceptron and a linear projection layer in the feed-forward network (FFN) are used for calculation. Among them, the feed-forward network (FFN) is used as the detection head to directly predict and output the classification category and the detection box. The prediction output includes the class label, the detection box information, and a specific "no object" category; the perceptron is used to predict the normalized center coordinates, height, and width of the target box; the linear layer is used to predict the class label using the softmax function.

[0067] S102. Update and predict the detection box to obtain the target prediction box corresponding to each ship target in the current frame image and the target detection box corresponding to each ship target in the next frame image;

[0068] In some alternative embodiments, the essence of the target tracking problem is the problem of associating the positions of the target in the front and rear frames. To address the problem of tracking failure due to target occlusion, the embodiments of the present invention propose a target tracking algorithm based on Kalman filtering. As Figure 4 shown in the update prediction example diagram, first, use Kalman filtering to predict the position of the ship target at time (i.e., the current frame) to obtain the time (i.e., the next frame), and obtain the target prediction box for the frame. Then, use the obtained target prediction box for the frame and the target detection box for the frame to perform target association to complete the tracking process of the target at time.

[0069] Specifically, first predict the current ship position to obtain the position where the ship will be at the next moment. Its system state equation and observation equation are as follows:

[0070] ;

[0071] ;

[0072] Among them, is the system state vector, which represents the motion state of the detection target in the embodiments of the present invention and can be represented by the state of the visual target detection bounding box; is the system noise, is the observation noise, and the variables in the two are independent of each other.

[0073] Next, each ship target is modeled with the following parameters:

[0074] ;

[0075] ;

[0076] where and represent the horizontal and vertical pixel positions of the center of the target bounding box, represents the area of the target bounding box, represents the aspect ratio of the target bounding box, , and respectively represent , and 's rate of change, is the state transition matrix, is the time step, that is, the time between two frames.

[0077] The obtained observation matrix and the observed quantity are shown as follows:

[0078] ;

[0079] ;

[0080] When there is no ship operation observation data (i.e., the actual measurement values of the system state obtained through sensors or other measurement means), the following prediction steps are continuously executed:

[0081] ;

[0082] When the ship operation observation data arrives, the following update process is executed:

[0083] ;

[0084] It should be noted that after successfully associating the target prediction box of the frame and the target detection box of the frame using the Hungarian algorithm based on the intersection over union and visual twin coding, the target state tracked at is updated using the bounding box obtained from the target detection process. If there is no observation data related to the target, no correction is performed and only its state is predicted, thus completing the prediction update of the position of the ship prediction box.

[0085] S103. Based on the pre-trained visual twin encoding model and the intersection over union (IoU) calculation formula, construct a cost matrix according to each target prediction box and each target detection box, and then perform target association according to the cost matrix through the Hungarian algorithm to obtain the initial tracking result of the ship target;

[0086] In some alternative embodiments, when performing target tracking on a video sequence of a ship's travel, after determining that the detection target of the current frame is not empty, first use the visual twin encoding model to construct a target prediction box for the next frame of this ship target based on the ship target detection box of the current frame, then calculate the first similarity between this target prediction box and the target detection box of the next frame, and then combine the ratio of the overlapping area between the target prediction box and the target detection box to the union of areas to construct a cost matrix to determine the matching degree of the target boxes in the previous and next frames. Before performing the matching, when the IoU or similarity of the matching is lower than the threshold, the pair of association results is considered inappropriate.

[0087] Further as an alternative implementation, the step of constructing a cost matrix according to each target prediction box and each target detection box based on the pre-trained visual twin encoding model and the IoU calculation formula can be specifically divided into the following steps S1031 to S1033:

[0088] S1031. Calculate the similarity between each target prediction box and each target detection box through the visual twin encoding model to obtain the first similarity;

[0089] Specifically, when the ship target is covered by the surrounding environment or a part of itself during its movement, it may cause fundamental changes in the shape and appearance of the target, making traditional feature extraction and matching methods ineffective. Therefore, the embodiments of the present invention use a visual twin encoding model to extract image features and calculate the similarity of the same ship target in the previous and next frames, so as to achieve efficient ship target matching subsequently. The visual twin encoding model uses the same neural network to encode the correlation information between the pictures of the previous and next frames, obtains features in the same distribution domain to solve the problem of feature space difference, calculates the similarity of the same target in the previous and next frames, and realizes efficient extraction and comparison of image features through shared weights.

[0090] Further as an alternative implementation, step S1031 can be specifically divided into the following steps S10311 to S10314:

[0091] S10311. Train the visual twin encoding model based on a deep convolutional neural network;

[0092] In some alternative embodiments, a large number of pictures are used to train the deep convolutional neural network. The deep convolutional neural network in the embodiments of the present invention selects the GG16 network, and sets the learning rate to , set the number of iterations to 100, and set the ratio of the training set to the test set to 7:3. The Loss function uses the contrastive loss function, and its definition is as follows:

[0093] ;

[0094] Among them, represents the Mahalanobis distance, is a boundary value used to define how far apart dissimilar samples should be. The more similar the samples are, the value of Figure 5 is closer to 1, otherwise it is closer to 0. The final result of the training is as Figure 5 shown. Within 1 - 20 epochs, the loss functions of the training set and the test set decrease rapidly. From 21 - 60 epochs, the rate of decrease starts to slow down, but still shows a stable downward trend. From 61 - 100 epochs, it tends to be stable and the rate of decrease further slows down, and finally converges. Moreover, the difference between the loss functions of the training set and the test set is not large, indicating that the visually similar encoding model obtained by training can provide a basis for solving the target occlusion problem in the next step.

[0095] S10312. Input each target prediction box and each target detection box into the visually similar encoding model;

[0096] S10313. Extract features from the target prediction box to obtain the first depth feature, and extract features from the target detection box to obtain the second depth feature;

[0097] Specifically, as Figure 6 shown is the schematic diagram of the similarity calculation process. First, use the VGG16 network to extract features from the pictures of the front and back two frames. By constructing five convolutional layers composed of multiple 3*3 convolutional kernels in each layer, and setting a max convolutional layer with a size of 2*2 and a stride of 2 between every two convolutional layers to reduce the image dimension and increase invariance to ensure that the features obtained from the two input images are in the same distribution domain, the first depth feature and the second depth feature are obtained.

[0098] S10314. Calculate the similarity between the first depth feature and the second depth feature through the Mahalanobis distance calculation formula to obtain the first similarity.

[0099] Specifically, after extracting features from the input target prediction box and target detection box, perform similarity measurement on the obtained first depth feature and the second depth feature, and calculate the first similarity of the two pictures through the following formula:

[0100] ;

[0101] ;

[0102] Among them, represents the first similarity degree, and respectively represent the first depth feature and the second depth feature, represents the spatial distance between the first depth feature and the second depth feature.

[0103] S1032. Calculate the intersection over union (IoU) between each target prediction box and each target detection box to obtain the IoU value;

[0104] Specifically, the larger the IoU value, the smaller the distance between the two detection boxes, and the greater the possibility that the two detection boxes are of the same target. The IoU calculation formula is as follows:

[0105] ;

[0106] S1033. Take each ship target in the current frame image as a row, and each ship target in the next frame image as a column, and construct a cost matrix according to the first similarity degree and the IoU value.

[0107] In some alternative embodiments, when a ship target is navigating in water, it is likely to be partially occluded. At this time, its appearance and features may change, which may cause errors in the tracker when identifying the target. Therefore, the embodiments of the present invention use visual twin coding to extract features from the front and rear frame pictures in the ship video sequence to obtain the similarity degree of the front and rear frame ship targets, and combine the ratio of the overlapping area between the target prediction box and the target detection box to the union of their areas to construct the cost matrix in the Hungarian algorithm, so as to achieve efficient matching and association of multiple ship targets.

[0108] Specifically, the first column of the cost matrix is the ship targets detected in the current frame image, the first row is the ship target prediction boxes predicted based on the previous frame image, and then the cost values between these targets are calculated in the cost matrix through the following formula , and the cost value the smaller it is, the more likely the two targets are the same target:

[0109] ;

[0110] wherein, is the cost matrix element value between the th detection target in the current frame and the th prediction target based on the previous frame.

[0111] It should be noted that in the existing ship target tracking methods, a cost matrix composed of label cost, depth feature cost, position cost, and intersection over union is used. The depth features extracted by it are insensitive to occlusion, resulting in the situation that the target is easily lost during the ship target tracking process or the target is wrongly associated with other targets. At the same time, the position cost in the existing cost matrix has insufficient adaptability to the change of the target position and may not be effectively processed in adjacent frames with large target displacements, easily causing tracking interruption and resulting in a low recall rate. In the cost matrix constructed in the embodiments of the present invention, the visual twin coding model has powerful feature extraction and similarity calculation capabilities, can effectively solve the problem of re-identifying occluded targets, and can effectively distinguish whether it is a new target or a previously occluded target, so as to better maintain the accuracy of tracking. Combining the intersection over union value and the first similarity extracted by the visual twin coding model to construct a cost matrix, so that between adjacent frames, the change of the spatial position and the appearance feature of the target can be considered simultaneously, reducing the probability of false matching between different targets and improving the continuity of detection and tracking.

[0112] During the ship target tracking process, the F1 score of the embodiments of the present invention reaches 0.932, which is higher than 0.885 of the existing methods, indicating that the cost matrix makes the embodiments of the present invention better at capturing all possible ship targets and making the detection results highly credible. At the same time, the multiple object tracking accuracy (MOTA) of the embodiments of the present invention reaches 0.869, which is higher than 0.862 of the existing methods, indicating that the cost matrix enables the embodiments of the present invention to better maintain the continuity of the target trajectory and is not easily lost due to short-term occlusion. It can be seen that the first similarity calculated by the visual twin coding model used in the embodiments of the present invention and the intersection over union value The constructed cost matrix has significant performance advantages in ship target tracking and target re-identification.

[0113] Further as an optional implementation manner, the step of obtaining the initial tracking result of the ship target according to the cost matrix through the Hungarian algorithm can be specifically divided into the following steps S1034 to S1038:

[0114] S1034. Determine the row minimum value of each row and the column minimum value of each column in the cost matrix;

[0115] S1035. Subtract each row of the cost matrix from the row minimum value in each row, and subtract each column of the cost matrix from the column minimum value in each column to obtain a first cost matrix;

[0116] S1036. Determine the number of independent zero elements and the matrix dimension of the first cost matrix;

[0117] S1037. When the number of independent zero elements is equal to the matrix dimension, obtain the initial tracking result of the ship target;

[0118] S1038. When the number of independent zero elements is greater than or less than the matrix dimension, adjust the first cost matrix until the number of independent zero elements is equal to the matrix dimension, and obtain the initial tracking result of the ship target.

[0119] Specifically, the cost matrix mobilization Hungarian algorithm is used to solve the assignment problem and find the optimal matching scheme in the cost matrix to minimize the overall matching cost. The specific process is as follows:

[0120] First, subtract the minimum value of each row in the obtained cost matrix from each row, and then subtract the minimum value of each column from each column of the matrix. Subsequently, find the independent zero elements in the matrix (that is, at most one zero can be selected in each row and each column) to form a preliminary match. If the number of independent zero elements is equal to the dimension of the matrix, the match is completed, and the initial tracking result of the ship target is output. If the number of independent zero elements is not sufficient to achieve a complete match, locate the minimum value that is not covered in the matrix, then subtract this minimum value from the uncovered elements and add it to the elements that are covered by both the row and the column at the same time to obtain the first cost matrix. For the adjusted first cost matrix, repeat the process of finding independent zero elements. If still not enough matches are found, continue to adjust the first cost matrix until all targets are matched. Return the final matching relationship, that is, the optimal assignment correspondence between each tracking target and the detection target, to achieve continuous tracking of the ship target. Even in the case of partial occlusion of the target, it is still possible to effectively associate the tracking box and the detection box to achieve accurate association and matching of the ship target.

[0121] As Figure 7 shown in the association schematic diagram of the Hungarian algorithm, at frame time, ship targets 1 and 2 are detected, and their target prediction boxes at frame time are obtained, and the cost values are calculated with ship targets a and b detected at frame time respectively to construct a cost matrix. Subsequently, the Hungarian algorithm is used for optimal matching, and the obtained association result is that the target 1 at frame time is associated with the ship target a detected at frame time, and the target 2 at frame time is associated with the ship target b detected at frame time.

[0122] S104. Classify the detection boxes to obtain the ship target types, compare the similarity between the disappearing targets and the newly emerging targets in the ship target types, and then perform target re-identification according to the similarity comparison result to obtain the secondary tracking result of the ship target.

[0123] Specifically, in the field of ship target tracking, the occlusion problem has always been a key challenge affecting the performance and reliability of tracking algorithms. Ship occlusion problems usually occur in complex water environments such as ports or inland waterways, and can be divided into two categories: partial occlusion and complete occlusion. Figure 8 The following are examples of partial occlusion and full occlusion. Partial occlusion means that part of the ship target is blocked by other ships, port facilities or terrain; while full occlusion means that the ship target is completely blocked by other objects. Figure 9 The figure shows an example of ship target re-identification. When a ship target is completely blocked, it may disappear in the visual monitoring system, which means the loss of the target for the tracker. When it re-enters the field of view, the tracker needs to be able to accurately identify the re-appeared ship target and resume correct tracking. Therefore, an embodiment of the present invention proposes a ship target re-identification algorithm based on visual twin coding. By using visual twin coding technology, the problem of re-tracking of ship targets after being blocked or re-entering the field of view of the image is effectively solved, avoiding misidentification of blocked targets as new targets.

[0124] As an optional implementation, the ship target type includes an associated target, a disappearing target, and a new target. The step of classifying the detection frame to obtain the ship target type can be specifically divided into the following steps S1041 to S1043:

[0125] S1041, classifying the detection frames detected and successfully associated in both the current frame image and the previous frame image as associated targets;

[0126] S1042, classifying the detection frame detected in the current frame image and not detected in the next frame image as a disappearing target;

[0127] S1043: Classify the detection frame detected in the current frame image and not detected in the previous frame image as a new target.

[0128] Specifically, a target that is not detected due to being occluded or driving out of the image field of view, and does not appear in the current frame image but appears in the previous frame image is defined as a disappeared target; for a target that was previously occluded and reappears, or appears in the image field of view for the first time, a target that appears in the current frame image but not in the previous frame image is defined as a new target; a target that appears in both the previous frame image and the current frame image and is successfully associated, that is, a target that is successfully tracked, is defined as an associated target.

[0129] As an optional implementation, a similarity comparison is performed between the disappeared target and the newly generated target in the ship target type, and then the target is re-identified according to the similarity comparison result to obtain the secondary tracking result of the ship target. This step can be specifically divided into the following steps S1044 to S1046:

[0130] S1044. Store the detection boxes corresponding to the last frames of each disappearing target to obtain a disappearing target set;

[0131] S1045. Calculate the similarity between the newly emerged target and each detection box in the disappearing target set through a visual twin coding model to obtain a second similarity;

[0132] S1046. Determine a similarity threshold, compare the second similarity with the similarity threshold, and then perform target re-identification based on the similarity comparison result to obtain a secondary tracking result of the ship target.

[0133] Further, as an optional implementation manner, the step of comparing the second similarity with the similarity threshold and then performing target re-identification based on the similarity comparison result to obtain a secondary tracking result of the ship target can be specifically divided into the following steps S10461 and S10462:

[0134] S10461. When the second similarity is greater than or equal to the similarity threshold, associate the newly emerged target with the corresponding detection box to obtain a secondary tracking result of the ship target;

[0135] S10462. When the second similarity is less than the similarity threshold, configure an identification number and generate a trajectory for the newly emerged target to obtain a secondary tracking result of the ship target.

[0136] Specifically, such as Figure 10The following is a schematic diagram of the process of object re-identification. First, the DETR algorithm is used to detect ship objects in each frame of the image, obtaining the object category and detection bounding box and assigning an identification number (ID number). Then, the Kalman filter is used to obtain the predicted bounding box of the object in the current frame image, and the visual twin coding model is used to calculate the similarity between the predicted bounding box of the object in the current frame image and the detected bounding box of the object in the next frame image. Furthermore, a cost matrix is constructed based on the intersection over union (IoU) of the overlapping area and the union area of the current predicted bounding box and the detected bounding box, and the similarity calculated by the visual twin coding model. Then, the Hungarian algorithm is used to achieve multi-object association of ships in the image. Next, the ship objects in the current frame image are classified into three categories: disappearing objects, newly emerging objects, and associated objects. After classifying the detection bounding boxes, the detection bounding boxes in the last frame image of the disappearing objects are stored in the disappearing object set. Then, for each disappearing object stored in the disappearing object set and the newly emerging object that appears, the visual twin coding model is used to encode the association information between them and then perform a similarity comparison. If the second similarity is greater than or equal to the similarity threshold, the newly emerging object is regarded as the re-identified object; if the second similarity is less than the similarity threshold, it is regarded as a newly emerging object, and a new identification number (ID number) is assigned and a new trajectory is generated to achieve accurate discrimination and processing of the ship object re-identification situation. Among them, the similarity threshold can be defined according to different usage scenarios and is not limited here.

[0137] Starting from the initial frame of the ship video sequence, object tracking is continuously performed on subsequent frame images until the end of the ship video sequence, so as to achieve full tracking of the entire ship video sequence based on visual twin coding.

[0138] The above describes the ship object tracking method based on visual twin coding in the embodiments of the present invention. It can be recognized that the embodiments of the present invention have the following advantages:

[0139] First, by utilizing the feature extraction ability of the VGG16 network, through a five-layer convolutional structure and max-pooling operations, the stability and consistency of feature extraction can be ensured. Furthermore, the Mahalanobis distance is used for similarity calculation, which can further improve the efficiency of feature matching. Then, combined with the DETR algorithm and the Kalman filter, the IoU value and the first similarity calculated by visual twin coding are combined to construct a cost matrix, and then the Hungarian algorithm is used to achieve multi-object association, which can effectively improve the accuracy of object matching, solve the technical problem that traditional methods cannot continuously track in the case of partial occlusion of objects, and achieve precise tracking of ship objects.

[0140] Second, introduce a ship target re-identification method based on visual twin coding. After the target is occluded or disappears, the target is classified (disappeared target, newly emerged target, and associated target), and then the newly emerged target that reappears is re-identified with high precision, which can significantly improve the robustness and accuracy of target tracking, solve the target tracking problem in complex scenarios such as multi-ship intersection, occlusion, and light change, and enhance the adaptability of the system to the dynamic environment.

[0141] Referring to Figure 11 , the embodiment of the present invention also provides a ship target tracking system based on visual twin coding, including:

[0142] A target detection module, configured to obtain a ship video sequence to be recognized, perform target detection on each frame image in the ship video sequence, and obtain a detection box corresponding to each frame image;

[0143] An update prediction module, configured to update and predict the detection box to obtain a target prediction box corresponding to each ship target in the current frame image and a target detection box corresponding to each ship target in the next frame image;

[0144] A target association module, configured to construct a cost matrix according to each target prediction box and each target detection box through a pre-trained visual twin coding model and an intersection over union calculation formula, and then perform target association according to the cost matrix through the Hungarian algorithm to obtain the initial ship target tracking result;

[0145] A target re-identification module, configured to classify the detection box to obtain the ship target type, compare the similarity between the disappeared target and the newly emerged target in the ship target type, and then perform target re-identification according to the similarity comparison result to obtain the secondary ship target tracking result.

[0146] The content in the above-mentioned embodiment of the ship target tracking method based on visual twin coding is applicable to the embodiment of this ship target tracking system based on visual twin coding. The functions specifically implemented by the embodiment of this ship target tracking system based on visual twin coding are the same as those of the above-mentioned embodiment of the ship target tracking method based on visual twin coding, and the beneficial effects achieved are also the same as those of the above-mentioned embodiment of the ship target tracking method based on visual twin coding.

[0147] The embodiment of the present invention also provides an electronic device, which includes: a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for realizing the connection and communication between the processor and the memory. When the program is executed by the processor, it implements the above-mentioned ship target tracking method based on visual twin coding. This electronic device can be any intelligent terminal including a tablet computer, an in-vehicle computer, etc.

[0148] Such as Figure 12The following is a schematic diagram of the hardware structure of the electronic device provided by the embodiment of the present invention. Referring to Figure 12 An embodiment of the present invention provides an electronic device, including:

[0149] The processor 1001 can be implemented in ways such as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided by the embodiments of the present invention;

[0150] The memory 1002 can be implemented in forms such as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When implementing the technical solutions provided by the embodiments of this specification through software or firmware, the relevant program codes are stored in the memory 1002 and are called by the processor 1001 to execute the ship target tracking method based on visual twin coding of the embodiments of the present invention;

[0151] The input / output interface 1003 is used to implement information input and output;

[0152] The communication interface 1004 is used to implement communication and interaction between this device and other devices, and can implement communication through wired means (such as USB, network cable, etc.) or through wireless means (such as mobile network, WIFI, Bluetooth, etc.);

[0153] The bus 1005 transmits information between various components of the device (such as the processor 1001, the memory 1002, the input / output interface 1003, and the communication interface 1004);

[0154] Among them, the processor 1001, the memory 1002, the input / output interface 1003, and the communication interface 1004 are communicatively connected to each other inside the device through the bus 1005.

[0155] An embodiment of the present invention also provides a storage medium, which is a computer-readable storage medium for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned ship target tracking method based on visual twin coding.

[0156] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs and non-transitory computer-executable programs. In addition, the memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory may optionally include memories that are remotely located relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0157] Embodiments of the present invention also disclose a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes Figure 1 the method shown.

[0158] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order mentioned in the operation diagrams. For example, depending on the functions / operations involved, two consecutive blocks shown may actually be executed substantially simultaneously, or the above blocks can sometimes be executed in the reverse order. In addition, the embodiments presented and described in the flowcharts of the present invention are provided by way of example for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operations and logical flows presented herein. Alternative embodiments are foreseeable, in which the order of various operations is changed and the sub-operations described as part of a larger operation are executed independently.

[0159] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the above functions and / or features may be integrated in a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It can also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More precisely, considering the attributes, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skills of an engineer. Therefore, those skilled in the art can implement the present invention as set forth in the claims without undue experimentation. It can also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, and the scope of the present invention is determined by the full scope of the appended claims and their equivalents.

[0160] When the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the above methods in various embodiments of the present invention. The foregoing storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs.

[0161] The logic and / or steps represented in the flowchart or described in other ways herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in combination with these instruction execution systems, apparatus, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0162] It should be understood that various parts of the present invention can be implemented by hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, any one or a combination of the following well-known technologies in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGA), field-programmable gate arrays (FPGA), etc.

[0163] Although the embodiments of the present invention have been shown and described, those of ordinary skill in the art can understand that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the claims and their equivalents.

[0164] The above is a specific description of the preferred embodiment of the present invention. However, the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent deformations or substitutions without departing from the spirit of the present invention, and these equivalent deformations or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A ship target tracking method based on visual twin coding, characterized in that Including the following steps: Obtain the ship video sequence to be recognized, perform object detection on each frame image in the ship video sequence through the DETR algorithm, and obtain the detection boxes corresponding to each frame of the image; Update and predict the detection boxes to obtain the target prediction boxes corresponding to each ship target in the current frame image and the target detection boxes corresponding to each ship target in the next frame image; Construct a cost matrix according to each target prediction box and each target detection box through a pre-trained visual twin coding model and an intersection over union calculation formula, and then perform target association according to the cost matrix through the Hungarian algorithm to obtain the initial tracking result of the ship target; Classify the detection boxes to obtain the ship target types, compare the similarity between the disappearing targets and the new targets in the ship target types, and then perform target re-identification according to the similarity comparison result to obtain the secondary tracking result of the ship target; The constructing of the cost matrix according to each target prediction box and each target detection box through a pre-trained visual twin coding model and an intersection over union calculation formula specifically includes: Calculate the similarity between each target prediction box and each target detection box through the visual twin coding model to obtain the first similarity; Perform matching calculation between each target prediction box and each target detection box through the intersection over union calculation formula to obtain the intersection over union value; Taking each ship target in the image of the current frame as a row and each ship target in the image of the next frame as a column, construct the cost matrix according to the first similarity and the intersection-over-union ratio, and calculate the cost values between these targets in the cost matrix through the following formula , where the cost value is smaller, indicating that the two targets are more likely to be the same target: .

2. The ship target tracking method based on visual twin coding according to claim 1, characterized in that, The calculating of the similarity between each target prediction box and each target detection box through the visual twin coding model to obtain the first similarity specifically includes: The visual twin coding model is trained based on a deep convolutional neural network; Input each target prediction box and each target detection box into the visual twin coding model; Extract features from the target prediction box to obtain the first deep feature, and extract features from the target detection box to obtain the second deep feature; Calculate the similarity between the first deep feature and the second deep feature through the Mahalanobis distance calculation formula to obtain the first similarity.

3. A method for tracking ship targets based on visual twin coding according to claim 1, characterized in that, The performing of target association according to the cost matrix through the Hungarian algorithm to obtain the initial tracking result of the ship target specifically includes: Determine the row minimum values of each row and the column minimum values of each column in the cost matrix; Perform subtraction operations on each row of the cost matrix and the row minimum value in each row, and perform subtraction operations on each column of the cost matrix and the column minimum value in each column to obtain the first cost matrix; Determine the number of independent zero elements and the matrix dimension of the first cost matrix; When the number of independent zero elements is equal to the matrix dimension, obtain the initial tracking result of the ship target; When the number of independent zero elements is greater than or less than the matrix dimension, adjust the first cost matrix until the number of independent zero elements is equal to the matrix dimension to obtain the initial tracking result of the ship target.

4. A ship target tracking method based on visual twin coding according to claim 1, characterized in that, The ship target types include associated targets, the disappearing targets, and the new targets. The classifying of the detection boxes to obtain the ship target types specifically includes: Divide the detection boxes that are detected in both the current-frame image and the previous-frame image and are successfully associated into the associated targets; Divide the detection boxes that are detected in the current-frame image but not detected in the next-frame image into the disappearing targets; Divide the detection boxes that are detected in the current-frame image but not detected in the previous-frame image into the newly emerging targets.

5. A method for ship target tracking based on visual twin coding according to claim 1, characterized in that, Performing a similarity comparison between the disappearing targets and the newly emerging targets in the ship target type, and then performing target re-identification based on the similarity comparison result to obtain the ship target secondary tracking result, specifically including: Store the detection boxes corresponding to the last-frame images of each of the disappearing targets to obtain a disappearing target set; Calculate the similarity between the newly emerging targets and each of the detection boxes in the disappearing target set through the visual twin coding model to obtain a second similarity; Determine a similarity threshold, compare the second similarity with the similarity threshold, and then perform target re-identification based on the similarity comparison result to obtain the ship target secondary tracking result.

6. The ship target tracking method based on visual twin coding according to claim 5, characterized in that, The comparing the second similarity with the similarity threshold, and then performing target re-identification based on the similarity comparison result to obtain the ship target secondary tracking result, specifically including: When the second similarity is greater than or equal to the similarity threshold, associate the newly emerging target with the corresponding detection box to obtain the ship target secondary tracking result; When the second similarity is less than the similarity threshold, configure an identification number and generate a trajectory for the newly emerging target to obtain the ship target secondary tracking result.

7. A ship target tracking system based on visual twin coding, characterized in that, Including: A target detection module, configured to obtain a ship video sequence to be recognized, perform target detection on each frame image in the ship video sequence through the DETR algorithm, and obtain detection boxes corresponding to each frame image; An update prediction module, configured to update and predict the detection boxes to obtain target prediction boxes corresponding to each ship target in the current-frame image and target detection boxes corresponding to each ship target in the next-frame image; A target association module, configured to construct a cost matrix according to each of the target prediction boxes and each of the target detection boxes through a pre-trained visual twin coding model and an intersection over union calculation formula, and then perform target association according to the cost matrix through the Hungarian algorithm to obtain a ship target primary tracking result; A target re-identification module, configured to classify the detection boxes to obtain a ship target type, perform a similarity comparison between the disappearing targets and the newly emerging targets in the ship target type, and then perform target re-identification based on the similarity comparison result to obtain the ship target secondary tracking result; The constructing a cost matrix according to each of the target prediction boxes and each of the target detection boxes through a pre-trained visual twin coding model and an intersection over union calculation formula, specifically including: Calculate the similarity between each of the target prediction boxes and each of the target detection boxes through the visual twin coding model to obtain a first similarity; Calculating the intersection over union ratio by using the intersection over union calculation formula to match and calculate each of the target prediction boxes and each of the target detection boxes, and obtaining the intersection over union ratio value; Taking each ship target in the image of the current frame as a row and each ship target in the image of the next frame as a column, a cost matrix is constructed according to the first similarity and the intersection-over-union ratio, and the cost values between these targets are calculated in the cost matrix by the following formula , where the cost value The smaller it is, the more likely the two targets are the same target: .

8. An electronic device, characterized in that, The electronic device includes a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the ship target tracking method based on visual twin coding according to any one of claims 1 to 6 are realized.

9. A storage medium, which is a computer-readable storage medium for computer-readable storage, characterized in that The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to realize the steps of the ship target tracking method based on visual twin coding according to any one of claims 1 to 6.

Images