Ship target tracking method and system based on visual twin coding, and medium

Abstract

The invention discloses a ship target tracking method and system based on visual twin coding and a medium, and the method comprises the steps: obtaining a ship video sequence, carrying out the target detection of the ship video sequence, and obtaining a detection frame corresponding to each frame of image; performing update prediction on the detection frame to obtain a target prediction frame corresponding to each ship target in the current frame image and a target detection frame corresponding to each ship target in the next frame image; constructing a cost matrix according to each target prediction frame and each target detection frame through a visual twin coding model and an intersection-to-parallel ratio calculation formula, and then carrying out target association according to the cost matrix through a Hungary algorithm to obtain a ship target initial tracking result; and classifying the detection frame to obtain a ship target type, performing similarity comparison on an evanescent target and a new target, and performing target re-identification according to a similarity comparison result to obtain a ship target secondary tracking result. The ship target tracking method can realize accurate tracking of the ship target, and can be widely applied to the technical field of target tracking.

Description

Technical Field

[0001] The present application relates to the field of target tracking technology, and in particular to a ship target tracking method, system and medium based on visual twin coding. Background Art

[0002] With the growing demand for port and inland waterway transportation, ship monitoring technology plays an increasingly important role in traffic management and navigation safety. However, these areas are usually characterized by narrow waterways, busy traffic, and complex environments. Ships are easily affected by other ships, port facilities, and terrain during navigation. The occlusion problem not only poses a challenge to the continuous tracking and effective detection of ship targets, but also increases the risk of misjudgment and loss of targets, posing a potential threat to navigation safety.

[0003] Traditional target tracking algorithms such as SORT and DeepSORT are mainly based on Kalman filtering and data association technology, which have obvious shortcomings in complex environments with target occlusion. When the target is partially occluded, it is difficult to capture complete feature information, resulting in a significant decrease in target matching accuracy; and when the target reappears after being completely occluded, there is a lack of effective re-identification mechanism, and it is easy to misjudge the original target as a new target. These methods are also poorly adaptable to dynamic environments. When faced with complex scenes such as multi-target occlusion and lighting changes in ports and inland waterways, robustness and stability cannot be guaranteed. Summary of the invention

[0004] In order to solve the above technical problems, the purpose of the present invention is to provide a ship target tracking method, system and medium based on visual twin coding, which can improve the accuracy of ship target tracking and re-identification.

[0005] To achieve the above purpose, one aspect of an embodiment of the present application proposes a ship target tracking method based on visual twin coding, comprising the following steps: Acquire a video sequence of a ship to be identified, perform target detection on each frame image in the video sequence of the ship, and obtain a detection frame corresponding to each frame image; The detection frame is updated and predicted to obtain a target prediction frame corresponding to each ship target in the image of the current frame and a target detection frame corresponding to each ship target in the image of the next frame; A cost matrix is ​​constructed according to each target prediction frame and each target detection frame through a pre-trained visual twin encoding model and an intersection-over-union calculation formula, and then the target is associated according to the cost matrix through the Hungarian algorithm to obtain the initial tracking result of the ship target; The detection frame is classified to obtain the ship target type, and a similarity comparison is performed between the disappeared target and the new target in the ship target type, and then the target is re-identified according to the similarity comparison result to obtain the secondary tracking result of the ship target.

[0006] In some embodiments, the cost matrix is ​​constructed according to each of the target prediction boxes and each of the target detection boxes using a pre-trained visual twin encoding model and an intersection-over-union calculation formula, specifically including: Calculating the similarity between each of the target prediction frames and each of the target detection frames through the visual twin encoding model to obtain a first similarity; Matching and calculating each target prediction frame with each target detection frame using the intersection-and-union ratio calculation formula to obtain an intersection-and-union ratio; The cost matrix is ​​constructed by taking each ship target in the image of the current frame as a row and each ship target in the image of the next frame as a column, and according to the first similarity and the intersection-union ratio.

[0007] In some embodiments, the calculating the similarity between each of the target prediction boxes and each of the target detection boxes through the visual twin encoding model to obtain a first similarity specifically includes: The visual twin encoding model is obtained based on deep convolutional neural network training; Inputting each of the target prediction boxes and each of the target detection boxes into the visual twin encoding model; Performing feature extraction on the target prediction frame to obtain a first depth feature, and performing feature extraction on the target detection frame to obtain a second depth feature; The first similarity is obtained by calculating the similarity between the first depth feature and the second depth feature using a Mahalanobis distance calculation formula.

[0008] In some embodiments, the target association is performed according to the cost matrix by using the Hungarian algorithm to obtain the initial tracking result of the ship target, specifically including: Determine the row minimum value of each row and the column minimum value of each column in the cost matrix; Subtracting each row of the cost matrix from the row minimum value in each row, and subtracting each column of the cost matrix from the column minimum value in each column, to obtain a first cost matrix; Determining the number of independent zero elements and the matrix dimension of the first cost matrix; When the number of independent zero elements is equal to the matrix dimension, the initial tracking result of the ship target is obtained; When the number of independent zero elements is greater than or less than the matrix dimension, the first cost matrix is ​​adjusted until the number of independent zero elements is equal to the matrix dimension, and the initial tracking result of the ship target is obtained.

[0009] In some embodiments, the ship target type includes the associated target, the disappeared target and the new target, and the classifying the detection frame to obtain the ship target type specifically includes: Classifying the detection frames detected and successfully associated in both the image of the current frame and the image of the previous frame as the associated targets; Classifying the detection frame detected in the image of the current frame and not detected in the image of the next frame as the disappearing target; The detection frame detected in the image of the current frame and not detected in the image of the previous frame is divided into the new target.

[0010] In some embodiments, the similarity comparison between the disappeared target and the newly generated target in the ship target type is performed, and then the target is re-identified according to the similarity comparison result to obtain the secondary tracking result of the ship target, which specifically includes: The detection frame corresponding to the last frame of the image of each disappeared target is stored to obtain a disappeared target set; Calculating the similarity between the new target and each of the detection frames in the disappeared target set by using the visual twin encoding model to obtain a second similarity; A similarity threshold is determined, and a similarity comparison is performed between the second similarity and the similarity threshold, and then the target is re-identified according to the similarity comparison result to obtain the secondary tracking result of the ship target.

[0011] In some embodiments, the similarity comparison between the second similarity and the similarity threshold is performed, and then the target is re-identified according to the similarity comparison result to obtain the secondary tracking result of the ship target, which specifically includes: When the second similarity is greater than or equal to the similarity threshold, the new target is associated with the corresponding detection frame to obtain a secondary tracking result of the ship target; When the second similarity is less than the similarity threshold, an identification number is configured and a trajectory is generated for the new target to obtain a secondary tracking result of the ship target.

[0012] To achieve the above purpose, another aspect of the embodiment of the present application proposes a ship target tracking system based on visual twin coding, including: The target detection module is used to obtain a video sequence of a ship to be identified, perform target detection on each frame image in the video sequence of the ship, and obtain a detection frame corresponding to each frame image; An update prediction module is used to update the detection frame to obtain a target prediction frame corresponding to each ship target in the image of the current frame and a target detection frame corresponding to each ship target in the image of the next frame; A target association module is used to construct a cost matrix according to each target prediction frame and each target detection frame through a pre-trained visual twin encoding model and an intersection-over-union calculation formula, and then perform target association according to the cost matrix through the Hungarian algorithm to obtain the initial tracking result of the ship target; The target re-identification module is used to classify the detection frame to obtain the ship target type, perform similarity comparison between the disappeared target and the new target in the ship target type, and then perform target re-identification based on the similarity comparison result to obtain the secondary tracking result of the ship target.

[0013] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes an electronic device, which includes a memory, a processor, a program stored on the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the ship target tracking method based on visual twin coding as described above is realized.

[0014] To achieve the above-mentioned purpose, another aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium for computer-readable storage, and the storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the ship target tracking method based on visual twin coding as described above.

[0015] The beneficial effects of the present invention are as follows: the ship target tracking method, system and medium based on visual twin coding of the present invention, by combining the intersection-and-union ratio with the similarity calculated by the visual twin coding model to construct a cost matrix, and then using the Hungarian algorithm to achieve multi-target association according to the cost matrix, can effectively improve the accuracy of target matching, solve the technical problem that the traditional method cannot continuously track the ship target when the ship target is partially occluded, and achieve accurate tracking of the ship target. And after the ship target is completely occluded or disappears, the ship target is classified, and the reappearing target is re-identified with high precision, which can significantly improve the robustness and accuracy of target tracking, solve the target tracking problem in complex scenes such as multi-ship intersection, occlusion and light changes, and enhance the system's adaptability to dynamic environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solution in the embodiments of the present invention, the following introduction is made to the drawings required for use in the embodiments of the present invention. It should be understood that the drawings introduced below are only for the convenience of clearly describing some embodiments of the technical solution of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A flowchart of a method for tracking a ship target based on visual twin coding provided by an embodiment of the present invention; Figure 2 A schematic diagram of a target detection process provided by an embodiment of the present invention; Figure 3 A schematic diagram of a Transformer-based encoder-decoder structure provided in an embodiment of the present invention; Figure 4 An example diagram of update prediction based on Kalman filtering provided by an embodiment of the present invention; Figure 5 A schematic diagram of the training results of the visual twin coding model provided in an embodiment of the present invention; Figure 6 A schematic diagram of a similarity calculation process provided by an embodiment of the present invention; Figure 7 A schematic diagram of an association based on the Hungarian algorithm provided in an embodiment of the present invention; Figure 8 An example diagram of partial occlusion and complete occlusion provided by an embodiment of the present invention; Fig. 9 An example diagram of ship target re-identification provided by an embodiment of the present invention; Fig.10 A schematic diagram of a target re-identification process provided by an embodiment of the present invention; Fig.11 A schematic diagram of the structure of a ship target tracking system based on visual twin coding provided by an embodiment of the present invention; Fig.12 A schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present application. They are only examples of devices and methods consistent with some aspects of the embodiments of the present application as detailed in the attached claims.

[0019] It is understood that the terms "first", "second", etc. used in this application can be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another concept. For example, without departing from the scope of the embodiment of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein can be interpreted as "at the time of" or "when" or "in response to determination".

[0020] The terms "at least one", "multiple", "each", "any", etc. used in this application, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0021] With the growing demand for port and inland waterway transportation, ship monitoring technology plays an increasingly important role in traffic management and navigation safety. However, these areas are usually characterized by narrow waterways, busy traffic, and complex environments. Ships are easily affected by other ships, port facilities, and terrain during navigation. The occlusion problem not only poses a challenge to the continuous tracking and effective detection of ship targets, but also increases the risk of misjudgment and loss of targets, posing a potential threat to navigation safety.

[0022] Traditional target tracking algorithms such as SORT and DeepSORT are mainly based on Kalman filtering and data association technology, which have obvious shortcomings in complex environments with target occlusion. When the target is partially occluded, it is difficult to capture complete feature information, resulting in a significant decrease in target matching accuracy; and when the target reappears after being completely occluded, there is a lack of effective re-identification mechanism, and it is easy to misjudge the original target as a new target. These methods are also poorly adaptable to dynamic environments. When faced with complex scenes such as multi-target occlusion and lighting changes in ports and inland waterways, robustness and stability cannot be guaranteed.

[0023] To this end, the embodiment of the present invention proposes a ship target tracking method based on visual twin coding, which constructs a cost matrix by combining the intersection-and-union ratio with the similarity calculated by the visual twin coding model, and then uses the Hungarian algorithm to achieve multi-target association according to the cost matrix, which can effectively improve the accuracy of target matching, solve the technical problem that traditional methods cannot continuously track when the ship target is partially occluded, and achieve accurate tracking of the ship target. And after the ship target is completely occluded or disappears, the ship target is classified, and the reappearing target is re-identified with high precision, which can significantly improve the robustness and accuracy of target tracking, solve the target tracking problems in complex scenes such as multi-ship intersection, occlusion and light changes, and enhance the system's adaptability to dynamic environments.

[0024] Reference Figure 1 , Figure 1 A flowchart of a method for tracking a ship target based on visual twin coding is provided in an embodiment of the present invention. An embodiment of the present invention proposes a method for tracking a ship target based on visual twin coding, and the method includes steps S101 to S104: S101, obtaining a video sequence of a ship to be identified, performing target detection on each frame image in the video sequence of the ship, and obtaining a detection frame corresponding to each frame image; In some optional embodiments, the ship video sequence is first processed, and the DETR algorithm is used to detect the target of each frame image in the ship video sequence, obtain the location category and corresponding detection frame of the ship target, and assign the corresponding identification number (ID number) to the ship target. The DETR algorithm adopts the encoder-decoder structure of Transformer, realizes unique prediction through binary matching loss, and uses parallel decoding to improve efficiency. It regards target detection as a direct set prediction problem, simplifies the training process, and avoids non-maximum suppression and anchor frame design.

[0025] Specifically, Figure 2 The following is a schematic diagram of the target detection process. Figure 3The figure shows a schematic diagram of the encoder-decoder structure based on Transformer. First, each frame image in the ship video sequence is passed into the backbone network of resnet50 for feature extraction to obtain the feature sequence of each frame of the ship driving image. After extracting the feature sequence information of each frame of the ship driving image, it is combined with the position encoding information and input into the Transformer encoder-decoder structure to solve the problem of losing the spatial distribution between pixels in the image. In the Transformer encoder-decoder structure, a 3-layer perceptron and a linear projection layer in the feedforward network (FFN) are used for calculation. Among them, the feedforward network (FFN) is used as the detection head to directly predict the classification category and detection box. The prediction output includes the category label, detection box information, and a specific "no object" category; the perceptron is used to predict the standardized center coordinates, height, and width of the target box; the linear layer is used to predict the class label using the softmax function.

[0026] S102, updating and predicting the detection frame to obtain a target prediction frame corresponding to each ship target in the current frame image and a target detection frame corresponding to each ship target in the next frame image; In some optional embodiments, the essence of the target tracking problem is the correlation problem between the positions of the target in the previous and next frames. To solve the problem of tracking failure due to target occlusion, the embodiment of the present invention proposes a target tracking algorithm based on Kalman filtering. Figure 4 The following is an example of updating the prediction. First, the Kalman filter is used based on The ship target detection frame pair obtained at the moment (i.e. the current frame) The target position of the ship at the moment (i.e. the next frame) is predicted and obtained The target prediction box of the frame is then used to obtain The target prediction box of the frame and The target detection frame of the frame is associated with the target to complete the The process of tracking the target at all times.

[0027] Specifically, the current ship position is first predicted to obtain the ship's position at the next moment. The system state equation and observation equation are as follows: ; ; in, is a system state vector, which represents the motion state of the detection target in the embodiment of the present invention, and can be represented by the state of the visual target detection bounding box; is the system noise, To observe noise, the variables in the two are independent of each other.

[0028] Next, each ship target is modeled using the following parameters: ; ; in, and represents the horizontal and vertical pixel position of the center of the object bounding box, represents the area of ​​the target bounding box, represents the aspect ratio of the target bounding box, , as well as Respectively represent , as well as The rate of change, is the state transfer matrix, is the time step, i.e. the time between two frames.

[0029] The resulting observation matrix and the observed quantity As shown below: ; ; When there is no ship operation observation data (i.e. actual measurements of the system status obtained by sensors or other measurement means), the following prediction steps are always performed: ; When the ship operation observation data arrives, the update process is performed as follows: ; It should be noted that when using the Hungarian algorithm based on intersection-over-union and visual twin coding The target prediction box of the frame and After the object detection box of the frame is successfully associated, the bounding box obtained during the object detection process is used to update The target state is tracked at all times. If there is no observation data related to the target, no correction is performed but only its state is predicted, thereby completing the prediction update of the ship prediction box position.

[0030] S103, constructing a cost matrix according to each target prediction frame and each target detection frame through a pre-trained visual twin encoding model and an intersection-over-union calculation formula, and then performing target association according to the cost matrix through the Hungarian algorithm to obtain the initial tracking result of the ship target; In some optional embodiments, when tracking a target in a video sequence of a ship, first, after determining that the detection target of the current frame is not empty, the visual twin coding model is used to construct the target prediction frame of the next frame of the ship target based on the current frame ship target detection frame, and then the first similarity between the target prediction frame and the next frame target detection frame is calculated, and then the ratio of the overlapping area to the area union between the target prediction frame and the target detection frame is combined to construct a cost matrix to determine the matching degree of the target frames of the previous and next frames. Before matching, when the intersection-over-union ratio or similarity of the matching is lower than the threshold, the association result is considered inappropriate.

[0031] As an optional implementation, the step of constructing a cost matrix according to each target prediction frame and each target detection frame by using a pre-trained visual twin encoding model and an intersection-over-union calculation formula can be specifically divided into the following steps S1031 to S1033: S1031. Calculate the similarity between each target prediction frame and each target detection frame through the visual twin encoding model to obtain a first similarity; Specifically, when a ship target is obscured by the surrounding environment or part of itself during its movement, the shape and appearance of the target may change fundamentally, making traditional feature extraction and matching methods no longer effective. Therefore, an embodiment of the present invention uses a visual twin coding model to extract image features and calculates the similarity of the same ship target in the previous and next frames, so as to achieve efficient ship target matching later. The visual twin coding model uses the same neural network to encode the correlation information between the previous and next frame images, obtains features of the same distribution domain to solve the problem of feature space differences, and calculates the similarity of the same target in the previous and next frames, and realizes efficient extraction and comparison of image features by sharing weights.

[0032] As an optional implementation, step S1031 may be specifically divided into the following steps S10311 to S10314: S10311. Visual twin coding model obtained based on deep convolutional neural network training; In some optional embodiments, a large number of images are used to train the deep convolutional neural network. The deep convolutional neural network in the embodiment of the present invention uses the GG16 network, and the learning rate is set to , set the number of iterations to 100, and set the ratio of training set to test set to 7:3. The Loss function uses the contrast loss function, which is defined as follows: ; in, represents the Mahalanobis distance, is a boundary value that defines how far apart dissimilar samples should be. The more similar the samples are, The value of is closer to 1, otherwise it is close to 0. The final result of the training is as follows Figure 5 As shown in the figure, the loss function of the training set and the test set decreases rapidly within 1-20 epochs, and the rate of decrease begins to slow down within 21-60 epochs, but still shows a stable downward trend. It tends to be stable within 61-100 epochs, and the rate of decrease slows down further and finally converges. The difference between the loss function of the training set and the test set is not large, indicating that the visual twin encoding model obtained by training can provide a basis for solving the target occlusion problem in the future.

[0033] S10312. Input each target prediction frame and each target detection frame into the visual twin encoding model; S10313, extracting features from the target prediction frame to obtain a first depth feature, and extracting features from the target detection frame to obtain a second depth feature; Specifically, Figure 6 The figure shows a flow chart of similarity calculation. First, the VGG16 network is used to extract features from the two frames before and after. Five convolutional layers are constructed, each consisting of multiple 3*3 convolutional kernels, and a maximum convolutional layer with a size of 2*2 and a stride of 2 is set between every two convolutional layers to reduce the image dimension, increase invariance and ensure that the features obtained from the two input images are in the same distribution domain, and obtain the first deep feature and the second deep feature.

[0034] S10314. Calculate the similarity between the first depth feature and the second depth feature using a Mahalanobis distance calculation formula to obtain a first similarity.

[0035] Specifically, after extracting features from the input target prediction frame and target detection frame, the obtained first depth feature is similarly measured with the second depth feature, and the first similarity of the two images is calculated by the following formula: ; ; in, represents the first similarity, and represent the first depth feature and the second depth feature respectively, Represents the spatial distance between the first depth feature and the second depth feature.

[0036] S1032, matching and calculating each target prediction frame with each target detection frame using an intersection-and-union ratio calculation formula to obtain an intersection-and-union ratio; Specifically, the larger the intersection-union ratio is, the smaller the distance between the two detection frames is, and the greater the possibility that the two detection frames are the same target. The calculation formula is as follows: ; S1033 , using each ship target in the current frame image as a row and each ship target in the next frame image as a column, and constructing a cost matrix according to the first similarity and the intersection-union ratio.

[0037] In some optional embodiments, when a ship target is sailing in the water, it is easy to be partially blocked, and its appearance and features may change, which may cause errors in the tracker's target identification. Therefore, the embodiment of the present invention uses visual twin coding to extract features from the front and back frame images in the ship video sequence to obtain the similarity of the ship targets in the front and back frames, and combines the ratio of the overlapping area between the target prediction frame and the target detection frame to the union of their areas to construct the cost matrix in the Hungarian algorithm, so as to achieve efficient matching and association of multiple ship targets.

[0038] Specifically, the first column of the cost matrix is ​​the ship target detected in the current frame image, and the first row is the ship target prediction box obtained based on the previous frame image. Then, the cost value between these targets is calculated in the cost matrix by the following formula: , cost value The smaller the value, the more likely it is that the two targets are the same target: ; in, The current frame detected targets and based on the previous frame The cost matrix element value between the predicted targets.

[0039] It should be noted that in the existing ship target tracking method, a cost matrix composed of label cost, depth feature cost, position cost and intersection-over-union ratio is used. The extracted depth features are insensitive to occlusion, which leads to the situation that the target is easily lost or the target is mistakenly associated with other targets during the ship target tracking process. At the same time, the position cost in the existing cost matrix is ​​not adaptable enough to the change of the target position, and may not be able to effectively handle adjacent frames with large target displacement, which can easily cause tracking interruption and low recall rate. In the cost matrix constructed in the embodiment of the present invention, the visual twin coding model has powerful feature extraction and similarity calculation capabilities, which can effectively solve the problem of re-identification of occluded targets and effectively distinguish whether it is a new target or a previously occluded target, so as to better maintain the accuracy of tracking. Combined with the intersection-over-union ratio The first similarity extracted by the visual twin encoding model To construct the cost matrix, the spatial position change and appearance feature change of the target can be considered simultaneously between adjacent frames, reducing the probability of mismatching between different targets and improving the continuity of detection and tracking.

[0040] In the process of ship target tracking, the F1 score of the embodiment of the present invention reached 0.932, which is higher than 0.885 of the existing method, indicating that the cost matrix enables the embodiment of the present invention to be better at capturing all possible ship targets, making the detection results more reliable. At the same time, the multi-target tracking accuracy (MOTA) of the embodiment of the present invention reached 0.869, which is higher than 0.862 of the existing method, indicating that the cost matrix enables the embodiment of the present invention to better maintain the continuity of the target trajectory and is not prone to losing the target due to short-term occlusion. It can be seen that the first similarity calculated by the visual twin coding model used in the embodiment of the present invention Intersection and Union Ratio The constructed cost matrix has significant performance advantages in ship target tracking and target re-identification.

[0041] As an optional implementation, the step of obtaining the initial tracking result of the ship target by performing target association according to the cost matrix through the Hungarian algorithm can be specifically divided into the following steps S1034 to S1038: S1034, determining the row minimum value of each row and the column minimum value of each column in the cost matrix; S1035, performing a subtraction operation on each row of the cost matrix and the row minimum value in each row, and performing a subtraction operation on each column of the cost matrix and the column minimum value in each column, to obtain a first cost matrix; S1036, determining the number of independent zero elements and the matrix dimension of the first cost matrix; S1037. When the number of independent zero elements is equal to the matrix dimension, the initial tracking result of the ship target is obtained; S1038. When the number of independent zero elements is greater than or less than the matrix dimension, the first cost matrix is ​​adjusted until the number of independent zero elements is equal to the matrix dimension, and the initial tracking result of the ship target is obtained.

[0042] Specifically, the cost matrix is ​​used to mobilize the Hungarian algorithm to solve the assignment problem and find the optimal matching solution in the cost matrix to minimize the overall matching cost. The specific process is as follows: First, subtract the minimum value of each row in the obtained cost matrix, and then subtract the minimum value of each column in the matrix. Then find independent zero elements in the matrix (that is, at most one zero can be selected in each row and each column) to form a preliminary match. If the number of independent zero elements is equal to the dimension of the matrix, the match is completed and the initial tracking result of the ship target is output. If the number of independent zero elements is not enough to achieve a complete match, locate the uncovered minimum value in the matrix, then subtract the minimum value from the uncovered element, and add it to the elements covered by both rows and columns to obtain the first cost matrix. Repeat the process of finding independent zero elements for the adjusted first cost matrix. If enough matches are still not found, continue to adjust the first cost matrix until all targets are matched. Return the final matching relationship, that is, the optimal allocation correspondence between each tracking target and the detection target, to achieve continuous tracking of the ship target. Even if the target is partially occluded, the tracking box and the detection box can still be effectively associated to achieve accurate association and matching of the ship target.

[0043] like Figure 7 The following is a schematic diagram of the Hungarian algorithm. At the frame time, ship targets 1 and 2 are detected and their The target prediction boxes at the frame time and compare them with The cost matrix is ​​constructed by calculating the cost values ​​of the ship targets a and b detected at the frame time, and then the Hungarian algorithm is used for optimal matching. The obtained association result is Target 1 at the frame time is associated with The ship target a detected at the frame time, Target 2 at the frame time is associated with Ship target b detected at frame time.

[0044] S104, classify the detection frame to obtain the ship target type, perform similarity comparison between the disappeared target and the new target in the ship target type, and then re-identify the target according to the similarity comparison result to obtain the secondary tracking result of the ship target.

[0045] Specifically, in the field of ship target tracking, the occlusion problem has always been a key challenge affecting the performance and reliability of tracking algorithms. Ship occlusion problems usually occur in complex water environments such as ports or inland waterways, and can be divided into two categories: partial occlusion and complete occlusion. Figure 8 The following are examples of partial occlusion and full occlusion. Partial occlusion means that part of the ship target is blocked by other ships, port facilities or terrain; while full occlusion means that the ship target is completely blocked by other objects. Fig. 9The figure shows an example of ship target re-identification. When a ship target is completely blocked, it may disappear in the visual monitoring system, which means the loss of the target for the tracker. When it re-enters the field of view, the tracker needs to be able to accurately identify the re-appeared ship target and resume correct tracking. Therefore, an embodiment of the present invention proposes a ship target re-identification algorithm based on visual twin coding. By using visual twin coding technology, the problem of re-tracking of ship targets after being blocked or re-entering the field of view of the image is effectively solved, avoiding misidentification of blocked targets as new targets.

[0046] As an optional implementation, the ship target type includes an associated target, a disappearing target, and a new target. The step of classifying the detection frame to obtain the ship target type can be specifically divided into the following steps S1041 to S1043: S1041, classifying the detection frames detected and successfully associated in both the current frame image and the previous frame image as associated targets; S1042, classifying the detection frame detected in the current frame image and not detected in the next frame image as a disappearing target; S1043: Classify the detection frame detected in the current frame image and not detected in the previous frame image as a new target.

[0047] Specifically, a target that is not detected due to being occluded or driving out of the image field of view, and does not appear in the current frame image but appears in the previous frame image is defined as a disappeared target; for a target that was previously occluded and reappears, or appears in the image field of view for the first time, a target that appears in the current frame image but not in the previous frame image is defined as a new target; a target that appears in both the previous frame image and the current frame image and is successfully associated, that is, a target that is successfully tracked, is defined as an associated target.

[0048] As an optional implementation, a similarity comparison is performed between the disappeared target and the newly generated target in the ship target type, and then the target is re-identified according to the similarity comparison result to obtain the secondary tracking result of the ship target. This step can be specifically divided into the following steps S1044 to S1046: S1044, storing the detection frame corresponding to the last frame image of each disappeared target to obtain a disappeared target set; S1045, calculating the similarity between the new target and each detection frame in the disappeared target set through the visual twin coding model to obtain a second similarity; S1046, determining a similarity threshold, performing a similarity comparison between the second similarity and the similarity threshold, and then performing target re-identification according to the similarity comparison result to obtain a secondary tracking result of the ship target.

[0049] As an optional implementation, the second similarity is compared with the similarity threshold, and then the target is re-identified according to the similarity comparison result to obtain the secondary tracking result of the ship target. This step can be specifically divided into the following steps S10461 and S10462: S10461. When the second similarity is greater than or equal to the similarity threshold, the new target is associated with the corresponding detection frame to obtain a secondary tracking result of the ship target; S10462: When the second similarity is less than the similarity threshold, an identification number is configured for the new target and a trajectory is generated to obtain a secondary tracking result of the ship target.

[0050] Specifically, Fig.10 The figure shows a flow chart of target re-identification. First, the DETR algorithm is used to detect the ship targets in each frame image, obtain the target category and detection frame and give an identification number (ID number). Then, the Kalman filter is used to obtain the target prediction frame of the current frame image, and the visual twin coding model is used to calculate the similarity between the target prediction frame of the current frame image and the target detection frame of the next frame image. Then, the cost matrix is ​​constructed based on the intersection ratio of the overlapping area and area union of the current target prediction frame and the detection frame and the similarity calculated by the visual twin coding model. Then, the Hungarian algorithm is used to realize the multi-target association of ships in the image, and then the ship targets in the current frame image are classified. Classify them into three categories: disappearing targets, new targets and associated targets. After classifying the detection frames, store the detection frames in the last frame of the disappearing targets in the disappearing target set, and then use the visual twin coding model to encode the associated information between the new targets and each disappearing target stored in the disappearing target set, and then compare the similarity. If the second similarity is greater than or equal to the similarity threshold, the new target is used as a re-identified target; if the second similarity is less than the similarity threshold, it is regarded as a new target, and a new identification number (ID number) is assigned, and a new trajectory is generated to achieve accurate judgment and processing of ship target re-identification. Among them, the similarity threshold can be defined according to different usage scenarios and is not limited here.

[0051] Starting from the initial frame of the ship video sequence, target tracking is continuously performed in subsequent frame images until the end of the ship video sequence, thereby achieving full tracking of the entire ship video sequence based on visual twin coding.

[0052] The above describes the ship target tracking method based on visual twin coding according to the embodiment of the present invention. It can be recognized that the embodiment of the present invention has the following advantages: First, by utilizing the feature extraction capability of the VGG16 network, through a five-layer convolution structure and maximum pooling operations, the stability and consistency of feature extraction can be ensured. The Mahalanobis distance is then used for similarity calculation to further improve the efficiency of feature matching. Combined with the DETR algorithm and Kalman filtering, the intersection-over-union ratio is combined with the first similarity calculated by the visual twin encoding to construct a cost matrix. Then, the Hungarian algorithm is used to achieve multi-target association, which can effectively improve the accuracy of target matching, solve the technical problem that traditional methods cannot continuously track when the target is partially occluded, and achieve accurate tracking of ship targets.

[0053] Second, a ship target re-identification method based on visual twin coding is introduced. After the target is occluded or disappears, the target is classified (disappeared target, new target and associated target), and then the re-appearing new target is re-identified with high precision. This can significantly improve the robustness and accuracy of target tracking, solve the target tracking problem in complex scenes such as multiple ship intersections, occlusions and light changes, and enhance the system's adaptability to dynamic environments.

[0054] Reference Fig.11 , an embodiment of the present invention further provides a ship target tracking system based on visual twin coding, comprising: The target detection module is used to obtain the ship video sequence to be identified, perform target detection on each frame image in the ship video sequence, and obtain the detection frame corresponding to each frame image; An update prediction module is used to update the prediction of the detection frame to obtain a target prediction frame corresponding to each ship target in the current frame image and a target detection frame corresponding to each ship target in the next frame image; The target association module is used to construct a cost matrix based on each target prediction frame and each target detection frame through the pre-trained visual twin encoding model and the intersection-over-union calculation formula, and then use the Hungarian algorithm to associate the target according to the cost matrix to obtain the initial tracking result of the ship target; The target re-identification module is used to classify the detection frame, obtain the ship target type, compare the similarity between the disappeared target and the new target in the ship target type, and then re-identify the target based on the similarity comparison result to obtain the secondary tracking result of the ship target.

[0055] The contents of the above-mentioned embodiments of the ship target tracking method based on visual twin coding are all applicable to the present embodiment of the ship target tracking system based on visual twin coding. The functions specifically implemented by the present embodiment of the ship target tracking system based on visual twin coding are the same as those of the above-mentioned embodiments of the ship target tracking method based on visual twin coding, and the beneficial effects achieved are also the same as those achieved by the above-mentioned embodiments of the ship target tracking method based on visual twin coding.

[0056] The embodiment of the present invention further provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and when the program is executed by the processor, the above-mentioned ship target tracking method based on visual twin coding is realized. The electronic device can be any intelligent terminal including a tablet computer, a vehicle-mounted computer, etc.

[0057] like Fig.12 FIG. 1 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention, referring to FIG. Fig.12 , an embodiment of the present invention provides an electronic device, including: The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention; The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other applications. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the ship target tracking method based on visual twin coding in the embodiment of the present invention; Input / output interface 1003, used to implement information input and output; Communication interface 1004, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.); A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 ); The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0058] An embodiment of the present invention also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned ship target tracking method based on visual twin coding.

[0059] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0060] The embodiment of the present invention also discloses a computer program product or a computer program, wherein the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes Figure 1 The method shown.

[0061] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the above-mentioned boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0062] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified to the contrary, one or more of the above-mentioned functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0063] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the above methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0064] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0065] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0066] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0067] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of this application.

Claims

1. A ship target tracking method based on visual twin coding, characterized in that: The following steps are involved: Acquire a video sequence of a ship to be identified, perform target detection on each frame image in the video sequence of the ship, and obtain a detection frame corresponding to each frame image; The detection frame is updated and predicted to obtain a target prediction frame corresponding to each ship target in the image of the current frame and a target detection frame corresponding to each ship target in the image of the next frame; A cost matrix is ​​constructed according to each target prediction frame and each target detection frame through a pre-trained visual twin encoding model and an intersection-over-union calculation formula, and then the target is associated according to the cost matrix through the Hungarian algorithm to obtain the initial tracking result of the ship target; The detection frame is classified to obtain the ship target type, and a similarity comparison is performed between the disappeared target and the new target in the ship target type, and then the target is re-identified according to the similarity comparison result to obtain the secondary tracking result of the ship target.

2. According to claim 1, a ship target tracking method based on visual twin coding is characterized in that: The cost matrix is ​​constructed according to each target prediction frame and each target detection frame through the pre-trained visual twin encoding model and the intersection-over-union calculation formula, specifically including: Calculating the similarity between each of the target prediction frames and each of the target detection frames through the visual twin encoding model to obtain a first similarity; Matching and calculating each target prediction frame with each target detection frame using the intersection-and-union ratio calculation formula to obtain an intersection-and-union ratio; The cost matrix is ​​constructed by taking each ship target in the image of the current frame as a row and each ship target in the image of the next frame as a column, and according to the first similarity and the intersection-union ratio.

3. According to claim 2, a ship target tracking method based on visual twin coding is characterized in that: The calculating the similarity between each target prediction frame and each target detection frame through the visual twin coding model to obtain a first similarity specifically includes: The visual twin encoding model is obtained based on deep convolutional neural network training; Inputting each of the target prediction boxes and each of the target detection boxes into the visual twin encoding model; Performing feature extraction on the target prediction frame to obtain a first depth feature, and performing feature extraction on the target detection frame to obtain a second depth feature; The first similarity is obtained by calculating the similarity between the first depth feature and the second depth feature using a Mahalanobis distance calculation formula.

4. According to claim 1, a ship target tracking method based on visual twin coding is characterized in that: The target association is performed according to the cost matrix by the Hungarian algorithm to obtain the initial tracking result of the ship target, which specifically includes: Determine the row minimum value of each row and the column minimum value of each column in the cost matrix; Subtracting each row of the cost matrix from the row minimum value in each row, and subtracting each column of the cost matrix from the column minimum value in each column, to obtain a first cost matrix; Determining the number of independent zero elements and the matrix dimension of the first cost matrix; When the number of independent zero elements is equal to the matrix dimension, the initial tracking result of the ship target is obtained; When the number of independent zero elements is greater than or less than the matrix dimension, the first cost matrix is ​​adjusted until the number of independent zero elements is equal to the matrix dimension, and the initial tracking result of the ship target is obtained.

5. According to a ship target tracking method based on visual twin coding according to claim 1, it is characterized in that: The ship target type includes the associated target, the disappeared target and the new target, and the classification of the detection frame to obtain the ship target type specifically includes: Classifying the detection frames detected and successfully associated in both the image of the current frame and the image of the previous frame as the associated targets; Classifying the detection frame detected in the image of the current frame and not detected in the image of the next frame as the disappearing target; The detection frame detected in the image of the current frame and not detected in the image of the previous frame is divided into the new target.

6. According to a ship target tracking method based on visual twin coding according to claim 1, it is characterized in that: The similarity comparison between the disappeared target and the new target in the ship target type is performed, and then the target is re-identified according to the similarity comparison result to obtain the secondary tracking result of the ship target, which specifically includes: The detection frame corresponding to the last frame of the image of each disappeared target is stored to obtain a disappeared target set; Calculating the similarity between the new target and each of the detection frames in the disappeared target set by using the visual twin encoding model to obtain a second similarity; A similarity threshold is determined, and a similarity comparison is performed between the second similarity and the similarity threshold, and then the target is re-identified according to the similarity comparison result to obtain the secondary tracking result of the ship target.

7. A ship target tracking method based on visual twin coding according to claim 6, characterized in that: The performing a similarity comparison between the second similarity and the similarity threshold, and then performing target re-identification according to the similarity comparison result to obtain the secondary tracking result of the ship target, specifically includes: When the second similarity is greater than or equal to the similarity threshold, the new target is associated with the corresponding detection frame to obtain a secondary tracking result of the ship target; When the second similarity is less than the similarity threshold, an identification number is configured and a trajectory is generated for the new target to obtain a secondary tracking result of the ship target.

8. A ship target tracking system based on visual twin coding, characterized in that: include: The target detection module is used to obtain a video sequence of a ship to be identified, perform target detection on each frame image in the video sequence of the ship, and obtain a detection frame corresponding to each frame image; An update prediction module is used to update the detection frame to obtain a target prediction frame corresponding to each ship target in the image of the current frame and a target detection frame corresponding to each ship target in the image of the next frame; A target association module is used to construct a cost matrix according to each target prediction frame and each target detection frame through a pre-trained visual twin encoding model and an intersection-over-union calculation formula, and then perform target association according to the cost matrix through the Hungarian algorithm to obtain the initial tracking result of the ship target; The target re-identification module is used to classify the detection frame to obtain the ship target type, perform similarity comparison between the disappeared target and the new target in the ship target type, and then perform target re-identification based on the similarity comparison result to obtain the secondary tracking result of the ship target.

9. An electronic device, characterized in that: The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory. When the program is executed by the processor, the steps of the ship target tracking method based on visual twin coding as described in any one of claims 1 to 7 are realized.

10. A storage medium, the storage medium being a computer-readable storage medium, used for computer-readable storage, characterized in that: The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the ship target tracking method based on visual twin coding as described in any one of claims 1 to 7.

Images