Maritime target tracking method and device based on template updating SiameseRPN++, equipment and medium

Abstract

The application relates to the technical field of target tracking and discloses a maritime target tracking method and device based on a template updating SiameseRPN++, equipment and a medium. The method comprises the following steps: performing the following tracking operation until the current frame image is the last frame image in a video frame sequence: determining a tracking result according to template features extracted by a template branch and current frame features extracted by a detection branch according to the current frame image by using a twin branch; determining the similarity between a prediction image obtained by cutting the current frame image according to the tracking result and each template image by using a template updating branch; and determining whether the current state of the target is a new state and whether the input of the template branch needs to be replaced according to the preset range to which the similarity belongs. The target tracking model is trained based on a training method, the template updating branch is introduced into the target tracking model, the templates are continuously updated or retained based on a tracking algorithm, and then target tracking is performed, so that the target tracking accuracy is improved.

Description

Technical Field

[0001] This invention relates to the field of target tracking technology, and more specifically to a method, apparatus, equipment, and medium for maritime target tracking based on template-updated SiameseRPN++. Background Technology

[0002] In the field of computer vision, target tracking is a highly popular research area. It aims to build a model of the target's appearance and motion information based on a given target in an initial frame within a video or image sequence, and then predict the target's motion state and locate its position in subsequent frames. Maritime target tracking, as an extension of target tracking technology to specific maritime environments, plays a crucial role in marine environmental monitoring and safety assurance in numerous ocean-related fields such as ship tracking, border defense, and fisheries supervision.

[0003] Despite the rapid development of research on maritime target tracking in recent years, many problems remain unsolved.

[0004] For targets such as ships, attitude changes, occlusion, and background interference during movement have a significant impact on target tracking algorithms. Related technologies use methods such as data augmentation and data generation to expand the dataset, but these still have certain limitations for specific scenarios.

[0005] Therefore, there is an urgent need for a maritime target tracking method that can improve tracking accuracy. Summary of the Invention

[0006] Therefore, the technical problem to be solved by the present invention is to overcome the problem of low accuracy in tracking maritime targets in related technologies.

[0007] To address the aforementioned technical problems, in a first aspect, this invention provides a maritime target tracking method based on template-updated SiameseRPN++, applied to a target tracking model, which includes a template branch, a detection branch, a twin branch, and a template update branch; the maritime target tracking method based on template-updated SiameseRPN++ includes:

[0008] The video to be tested is processed to determine the video frame sequence;

[0009] Perform the following tracking operation until the current frame image is the last frame image of the video frame sequence:

[0010] The template branch is used to extract template features based on the target image for this tracking operation; the target image for the first tracking operation is obtained by cropping the first frame image; the target image for this tracking operation is stored as a template image in the template library; the template library includes template images of different target states;

[0011] The detection branch is used to extract the features of the current frame based on the current frame image targeted by this tracking operation; the current frame image targeted by the first tracking operation is the second frame image;

[0012] The twin branch is used to determine the tracking result of this tracking operation based on the template features extracted in this tracking operation and the features of the current frame. The tracking result of this tracking operation includes the target's position information and foreground / background classification results in the current frame image targeted by this tracking operation.

[0013] Based on the tracking results of this tracking operation, the current frame image targeted by this tracking operation is segmented to obtain the prediction image targeted by this tracking operation;

[0014] The similarity between the predicted image targeted by this tracking operation and each template image in the template library is determined using the template update branch.

[0015] When it is determined that the current frame image targeted by this tracking operation is not the last frame image, if there is at least one template image whose similarity to the predicted image targeted by this tracking operation is within a first preset range, then the predicted image targeted by this tracking operation is stored as a template image in the template library; the first preset range is greater than a first preset value and less than or equal to a second preset value.

[0016] In an optional implementation, after determining the similarity between the predicted image targeted by the current tracking operation and each template image in the template library using the template update branch, the method further includes:

[0017] When it is determined that the current frame image targeted by the current tracking operation is not the last frame image, if there is at least one template image whose similarity to the predicted image targeted by the current tracking operation is within the second preset range, then the template features extracted by the current tracking operation are determined as the template features extracted by the next tracking operation; the minimum value of the second preset range is greater than the second preset value.

[0018] In an optional implementation, after determining the similarity between the predicted image targeted by the current tracking operation and each template image in the template library using the template update branch, the method further includes:

[0019] If it is determined that the current frame image targeted by this tracking operation is not the last frame image, and if there is at least one template image whose similarity to the predicted image targeted by this tracking operation belongs to a third preset range, then the next tracking operation is performed; the maximum value of the third preset range is greater than or equal to the first preset value.

[0020] In one optional implementation, determining the tracking result for this tracking operation based on the template features extracted in this tracking operation and the current frame features using the twin branch includes:

[0021] Cross-correlation calculation is performed on the template features extracted in this tracking operation and the current frame features extracted in this tracking operation to obtain the response map targeted by this tracking operation; the response map includes the foreground and background classification results corresponding to each response point;

[0022] Extract response point images from the response image for which the foreground category probability is greater than a preset threshold, and determine them as sample images for this tracking operation.

[0023] The sample image targeted by this tracking operation is input into the region proposal regression model, and the tracking result for this tracking operation is determined using the region proposal regression model.

[0024] In one optional implementation, the cross-correlation calculation of the template features extracted in this tracking operation and the current frame features extracted in this tracking operation includes:

[0025] Cross-correlation calculations are performed on the shallow, medium, and deep features of the template features extracted in this tracking operation, and the shallow, medium, and deep features of the current frame features extracted in this tracking operation.

[0026] In an optional implementation, the method further includes:

[0027] When it is determined that the current frame image targeted by this tracking operation is the last frame image, the tracking operation ends and the target tracking result is obtained; the target tracking result includes the target's position information and foreground / background classification results in each frame image of the video frame sequence.

[0028] In one optional implementation, after processing the video to be detected to determine the video frame sequence, the method further includes:

[0029] The video frame sequence is preprocessed, including Wiener filtering, cropping, and linear interpolation.

[0030] Secondly, the present invention provides a maritime target tracking device based on template update SiameseRPN++, applied to a target tracking model, the target tracking model including a template branch, a detection branch, a twin branch, and a template update branch; the maritime target tracking device based on template update SiameseRPN++ includes:

[0031] The first processing module is used to process the video to be detected and determine the video frame sequence;

[0032] The second processing module is used to perform the following tracking operations until the current frame image is the last frame image of the video frame sequence:

[0033] The second processing module includes a first processing unit, a second processing unit, a third processing unit, a fourth processing unit, and a fifth processing unit;

[0034] The first processing unit is used to extract template features based on the target image of the current tracking operation using the template branch; the target image of the first tracking operation is obtained by cropping the first frame image; the target image of the current tracking operation is stored as a template image in the template library; the template library includes template images of different target states;

[0035] The second processing unit is used to extract current frame features based on the current frame image targeted by the current tracking operation using the detection branch; the current frame image targeted by the first tracking operation is the second frame image;

[0036] The third processing unit is used to determine the tracking result of the current tracking operation based on the template features extracted in the current tracking operation and the current frame features using the twin branch; the tracking result of the current tracking operation includes the target's position information and foreground / background classification results in the current frame image targeted by the current tracking operation.

[0037] The fourth processing unit is used to cut the current frame image targeted by the current tracking operation according to the tracking result targeted by the current tracking operation to obtain the prediction image targeted by the current tracking operation.

[0038] The fifth processing unit is used to determine the similarity between the predicted image targeted by this tracking operation and each template image in the template library using the template update branch;

[0039] When it is determined that the current frame image targeted by this tracking operation is not the last frame image, if there is at least one template image whose similarity to the predicted image targeted by this tracking operation is within a first preset range, then the predicted image targeted by this tracking operation is stored as a template image in the template library; the first preset range is greater than a first preset value and less than or equal to a second preset value.

[0040] Thirdly, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to perform the template-updated SiameseRPN++-based maritime target tracking method described in the first aspect or any corresponding embodiment thereof.

[0041] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions on a single computer-readable storage medium, the computer instructions being used to cause a computer to execute the template-updated SiameseRPN++ maritime target tracking method of the first aspect above or any corresponding embodiment thereof.

[0042] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the template-updated SiameseRPN++ maritime target tracking method described in the first aspect or any corresponding embodiment thereof.

[0043] The technical solution provided by this invention has the following technical effects:

[0044] By working collaboratively with template branches, detection branches, and twin branches, features can be effectively extracted from video frame sequences, and the target's position information and foreground / background classification results can be determined in each frame. In the initial tracking operation, the template branch extracts template features from the target image in the first frame, the detection branch extracts current frame features from the second frame, and the twin branch calculates accurate tracking results based on both features. Based on the tracking results, the second frame is segmented to obtain a predicted image. The similarity between the predicted image and template images in the template library is determined. If the similarity falls within a first preset range, the predicted image is stored as the template image in the template library, updating the template. This allows the target to be accurately identified and located in complex maritime environments, maintaining high tracking accuracy even if the target undergoes some degree of displacement or rotation.

[0045] The template update branch determines whether to update the template based on the similarity between the predicted image and template images in the template library. When a template image exists whose similarity to the predicted image is within a first preset range (greater than a first preset value and less than or equal to a second preset value), the predicted image is stored as a new template in the template library. This template update strategy enables the model to adapt to changes in the target during tracking, such as changes in the target's pose or recovery from partial occlusion. For example, when a ship is sailing at sea, it may present different angles due to turning. By dynamically updating the template, the model can better adapt to these changes, continuously and accurately track the target, and avoid tracking failures or accuracy degradation caused by a fixed template.

[0046] This invention can improve the stability of long-term tracking missions. As time goes on, the maritime environment becomes increasingly complex and changeable, and targets may experience various complex situations. The technical solution of this invention, through continuous template updates, can accurately track targets, reduce the risk of mistracking, and thus improve the stability of long-term tracking missions. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the specific embodiments or related technologies of the present invention, the drawings used in the description of the specific embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0048] Figure 1 This is a flowchart illustrating the maritime target tracking method based on template update SiameseRPN++ according to an embodiment of the present invention.

[0049] Figure 2 This is a schematic diagram of the network structure of the target tracking model according to an embodiment of the present invention;

[0050] Figure 3 This is a flowchart illustrating the template update strategy according to an embodiment of the present invention;

[0051] Figure 4 This is a flowchart illustrating the tracking algorithm according to an embodiment of the present invention;

[0052] Figure 5 This is a schematic diagram of the structure of the maritime target tracking device based on template update SiameseRPN++ according to an embodiment of the present invention;

[0053] Figure 6 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] Target tracking is a hot research topic in computer vision. It involves specifying a target in an initial frame of a video or image sequence, modeling its appearance and motion information, and then continuously predicting the target's motion state and pinpointing its location in subsequent video frames. Maritime target tracking, as an application and extension of target tracking technology in the marine environment, plays a crucial role in monitoring and protecting the marine environment and ensuring safety. It is widely used in areas such as ship tracking, border defense, and fisheries supervision. Compared to traditional land-based target tracking, the marine environment is complex and variable, including diverse vessel types, target occlusion, and background interference from the sea surface, which presents many challenges for target tracking.

[0056] Significant progress has been made in maritime target tracking algorithms in recent years, and the main algorithms can be divided into three categories.

[0057] The first category is tracking algorithms based on correlation filtering, such as the Kalman filter, Mean-Shift algorithm, and MOSSE algorithm. These algorithms primarily utilize the target's motion and feature information across consecutive frames for tracking, transforming the solver template from complex time-domain computations into Fourier-domain dot product calculations, significantly reducing computational load and improving tracker speed. However, these algorithms may fail to track in complex and ever-changing maritime environments.

[0058] The second category is feature-matching-based tracking algorithms, such as optical flow. This algorithm is a pixel-motion-based tracking algorithm that calculates the target's trajectory based on the optical flow information of pixel motion between adjacent frames. In maritime target tracking, optical flow can be used to estimate the ship's speed and direction, and combined with other tracking algorithms to achieve more accurate tracking.

[0059] The third category comprises deep learning-based tracking algorithms. The recently developed Siamese series of network tracking algorithms have achieved excellent performance in both accuracy and speed. The earliest SiamFC model utilized a fully convolutional network for fast matching, but lacked scale adaptability and struggled to handle occlusion and appearance changes. The subsequent SiamRPN introduced a Region Proposal Network (RPN) on top of SiamFC, enhancing adaptability to scale changes and improving tracking accuracy through anchor box mechanisms. Further improvements include SiamMask, which incorporates a semantic segmentation module, enabling the algorithm to generate pixel-level segmentation masks and expanding application scenarios, but increasing computational complexity. SiamRPN++ introduces deep networks and multi-layer fusion for feature extraction, improving the model's robustness in complex backgrounds. Ocean innovatively eliminated the anchor box design, improving the model's adaptability to target changes through end-to-end target localization combined with an online update strategy.

[0060] Despite the rapid development of research on maritime target tracking in recent years, many problems remain unsolved.

[0061] First, the cost of collecting maritime target data is high and the number of samples is small. Especially in small sample scenarios, existing well-performing supervised learning-based target tracking algorithms usually require a large amount of labeled data. Furthermore, issues such as data distribution differences and expired labeled data can greatly affect the training speed and effectiveness. Even when using transfer learning methods, it is difficult to achieve effective tracking of new types of targets.

[0062] Second: The target's posture changes, occlusion, and background interference during movement have a great impact on target tracking algorithms. Related technologies use methods such as data augmentation and data generation to expand the dataset, but they still have certain limitations for specific scenarios.

[0063] Third: When the model encounters a target category it has never seen before, existing high-performing algorithms can only identify it as one of the existing labels, making category expansion difficult and prone to tracking failures. Therefore, it is necessary to design an effective online template-updating target tracking method to address these three problems. This invention aims to provide a maritime target tracking method based on template-updating SiameseRPN++.

[0064] To address these issues, embodiments of the present invention provide a method, apparatus, device, and medium for tracking maritime targets based on template-updated SiameseRPN++.

[0065] According to an embodiment of the present invention, a method for tracking maritime targets based on template-updated SiameseRPN++ is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer device such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0066] Figure 1 This is a flowchart illustrating the maritime target tracking method based on template update SiameseRPN++ according to an embodiment of the present invention.

[0067] like Figure 1 As shown in the figure, this embodiment of the invention provides a maritime target tracking method based on template update SiameseRPN++, applied to a target tracking model. The target tracking model includes a template branch, a detection branch, a Siamese branch, and a template update branch. The network structure of the target tracking model is as follows: Figure 2 As shown.

[0068] This template-based SiameseRPN++ method for tracking maritime targets includes:

[0069] S101: Process the video to be detected to determine the video frame sequence.

[0070] In this embodiment, the video to be detected is fully sampled to obtain a video frame sequence, which includes multiple frames, such as a first frame, a second frame, a third frame, etc.

[0071] In this embodiment, after processing the video to be detected to determine the video frame sequence, the video frame sequence is preprocessed to obtain a preprocessed video frame sequence. The preprocessing includes Wiener filtering, cropping, and linear interpolation. The preprocessed video frame sequence is then applied to the technical solution described in S102 below.

[0072] Each frame in the video frame sequence is processed by Wiener filtering to remove motion blur caused by rapid movement of maritime targets, such as ships. Each filtered frame is then cropped and subjected to quadratic linear interpolation to obtain a template image of a first preset size, such as 127*127, after removing motion blur, and an image of a second preset size, such as 255*255, after removing motion blur.

[0073] S102: Perform the following tracking operation until the current frame image is the last frame image of the video frame sequence:

[0074] S1021: Use the template branch to extract template features from the target image for this tracking operation.

[0075] In this embodiment, the target image for the initial tracking operation is obtained by cropping the first frame image. The target image for this tracking operation is stored as a template image in the template library. The template library includes template images of different target states.

[0076] In this embodiment, for the initial tracking operation, the target image for the initial tracking operation can be obtained by selecting the target (e.g., a ship) in the first frame image and cropping the background. The target image includes the target. The target image for the initial tracking operation can be stored in the template library as the first state. The template library consists of multiple states of a single target. As an example, four states can be set, typically frontal, left, right, and rear. For example, the front, left, right, and rear of a ship.

[0077] During the initial tracking operation, there is only one sample in the template library. As the video is detected, more and more templates will be added. Templates of the same type will be merged and updated, while templates of different types will be separated, with a maximum of 4 states.

[0078] In this embodiment, the input to the template branch is the target image for this tracking operation, and the output of the template branch is the template features, specifically the template branch feature maps F of each convolutional layer. l (z). The template branch is used to extract features from the input of the template branch, which is specifically a ResNet-50 network. The target image is the template image and will be updated during the tracking process along with the algorithm of the template update branch.

[0079] S1022: Use the detection branch to extract the features of the current frame based on the current frame image targeted by this tracking operation.

[0080] In this embodiment, the first tracking operation targets the second frame image as the current frame image.

[0081] In this embodiment, the second tracking operation targets the third frame image. The input to the detection branch is the current frame image targeted by this tracking operation, and the output of the detection template branch is the current frame features, specifically the detection branch feature maps F of each convolutional layer. l (x). The detection branch is used to extract features from the search region in the current frame image (the current frame image obtained after preprocessing the search region). Specifically, the detection branch uses a ResNet-50 network. The template branch and detection branch use a weight-shared ResNet-50 network for feature extraction to obtain the template branch feature map F. l (z) and detection branch feature map F l (x).

[0082] S1023: Use twin branches to determine the tracking result for this tracking operation based on the template features extracted in this tracking operation and the current frame features.

[0083] In this embodiment, the tracking result for this tracking operation includes the target's position information and foreground / background classification result in the current frame image targeted by this tracking operation. The position information can be the coordinate information of the target bounding box, for example, the coordinate information of the four vertices.

[0084] In this embodiment, S1023 utilizes twin branches to determine the tracking result for this tracking operation based on the template features extracted in this tracking operation and the current frame features, specifically including:

[0085] Cross-correlation is performed on the template features extracted in this tracking operation and the current frame features extracted in this tracking operation to obtain the response map for this tracking operation. The response map includes the foreground and background classification results corresponding to each response point. The foreground and background classification results include foreground class probability and background class probability. The foreground is the image including the target, and the background is the image excluding the target.

[0086] Specifically, cross-correlation calculations are performed on the template features extracted in this tracking operation and the current frame features extracted in this tracking operation, including: cross-correlation calculations are performed on the shallow features, middle features, and deep features in the template features extracted in this tracking operation, and the shallow features, middle features, and deep features in the current frame features extracted in this tracking operation.

[0087] Extract response point images from the response image for which the foreground category probability is greater than a preset threshold, and determine them as sample images for this tracking operation.

[0088] The sample images targeted by this tracking operation are input into the region proposal regression model, and the tracking result for this tracking operation is determined using the region proposal regression model.

[0089] In this embodiment, the input to the Siamese branch is the feature maps of the detection branches of the shallow, middle, and deep convolutional layers, and the template branch feature maps of the shallow, middle, and deep convolutional layers. As an example, the shallow, middle, and deep layers could be the 7th, 13th, and 16th convolutional layers, respectively. The output of the Siamese branch is the tracking result for this tracking operation.

[0090] In this embodiment, the twin branch includes a preset number of SiameseRPN modules. For example, 3 SiameseRPN modules.

[0091] A single Siamese RPN module has two branches: a regression branch and a classification branch. Each branch receives... and The two feature maps, after their dimensions have been altered by convolutional layers, are used as input. After passing through three layers, a 25×25×2k classification branch response map is generated. Regression response plot for 25×25×4k

[0092]

[0093] Here, ★ represents the convolution operation, which is equivalent to using a convolution kernel. and Come to and Perform convolution operations.

[0094] For the three Siamese RPN modules, hierarchical aggregation is required. The classification feature maps in these outputs are designated S3, S4, and S5, and the regression feature maps are designated B3, B4, and B5. This invention directly applies a weighted sum to the RPN outputs:

[0095]

[0096] Among them, S all B is the aggregated classification feature map. all This is the aggregated regression feature map, where α and β are the corresponding weights.

[0097] In this embodiment, the SiameseRPN module incorporates a Region Proposal Regression (RPN) model into the Siamese network. It generates a set of predefined anchor boxes within the search region, predicts whether each anchor box contains the target, and regresses to adjust the position and size of the anchor boxes, thereby achieving efficient target localization. The Region Proposal Regression model includes candidate region generation, regression optimization, and classification scoring. For anchor selection, one anchor size is chosen, and four aspect ratios are set: [0.33, 0.5, 1, 2, 3]. Regarding the selection of positive and negative samples, similar to the RPN in Faster R-CNN, a threshold is set, and the IOU between the anchor and the ground truth is compared. If the IOU exceeds the threshold, it is considered a positive sample. Similarly, if the IOU is less than a certain threshold, it is considered a negative sample. In this invention, IOU > 0.6 is considered positive, and IOU < 0.3 is considered negative. Furthermore, to avoid imbalance between positive and negative samples, the ratio of positive to negative samples is set to approximately 1:3.

[0098] In this embodiment, each response point in the response image corresponds to a foreground class probability and a background class probability. If the foreground class probability IOU at a response point is greater than a preset threshold (e.g., 0.6), it indicates that the image at that response point has a high probability of being the location of the target, and it is identified as a positive sample. If the foreground class probability IOU at a response point is less than or equal to a preset threshold (or less than another preset threshold, e.g., 0.3), it indicates that the image at that response point is highly unlikely to be the location of the target, and it is identified as a negative sample. The preset threshold can be set and modified according to actual needs.

[0099] The response point images with foreground category probabilities (IOU) greater than a preset threshold are extracted as sample images for prediction regression. The regression target image is obtained by using a region proposal regression model. The regression target image refers to the target image obtained by cropping the current frame image according to the regression box. Generally, many target images will be obtained. The appropriate target image is selected according to the IOU threshold, and the detection box with the highest probability is taken as the target box.

[0100] The Siamese branch performs cross-correlation calculations on the image features input from the template branch and the detection branch, and uses three SiameseRPN modules to obtain the bounding box coordinates and foreground / background classification results of the target in the video to be detected. Response point images with foreground class probabilities greater than a preset threshold are extracted from the response image as sample images for prediction regression. The predicted bounding box coordinates (location information) and foreground / background classification results are then obtained using a region proposal regression model.

[0101] S1024: Based on the tracking result of this tracking operation, the current frame image targeted by this tracking operation is segmented to obtain the prediction image targeted by this tracking operation.

[0102] In this embodiment, the target is segmented from the current frame image based on its location information and foreground / background classification results, resulting in a predicted image for this tracking operation. The predicted image is the result of segmenting the target from the current frame image based on the tracking results, and it includes the target. Foreground and background can be labeled using custom classification tags.

[0103] S1025: Use the template update branch to determine the similarity between the predicted image and each template image in the template library for this tracking operation.

[0104] The template update branch consists of three parts: feature modeling, adaptive updating, and occlusion relocalization. It updates the matched template to represent the target state in the current time period when the target undergoes deformation, angle change, or occlusion. A Gaussian mixture model and perceptual hashing algorithm are used to model, cluster, and determine the target state, thereby establishing a template library. The prediction results for each frame are evaluated; if similar samples exist in the library, they can directly replace the input of the template branch in the network. This allows the algorithm to select the most reliable target state from past tracking results to cope with changes in the target's appearance, while avoiding redundant computation. The steps are as follows:

[0105] (1) Feature Modeling: Gaussian mixture model is used to model and cluster the state of the target in each frame, thereby establishing a template library. Let the random variable x, then the Gaussian mixture model can be expressed as:

[0106]

[0107] Where: N(x; μ) k I) represents the k-th type in the Gaussian mixture model. If there are K types that need to be clustered, they can be represented by K Gaussian distributions, μ k ∈X represents the mean, and the covariance matrix is ​​set to the identity matrix I to avoid complex calculations in high-dimensional sample spaces. π k Let be the mixing coefficient, equivalent to the weight of each type, and satisfy:

[0108]

[0109] If similar templates exist, then the two types k and l are merged into a single category sample set n. The merging method is as follows:

[0110] π n =π k +π l ,

[0111] (2) Adaptive Update: The perceptual hash algorithm is used to compare the similarity between the predicted target and the template target in the current frame, thereby determining whether the current frame image represents a new state of the target. The perceptual hash (pHash) algorithm performs DCT transformation on the image, obtains the mean of the DCT coefficients, and calculates image similarity based on its transform domain features. This algorithm converts the image into an 8×8 Hash fingerprint of 0 or 1. Counting the Hash fingerprints can determine the similarity between two images. Generally, if the number of identical bits in the Hash fingerprints of two images exceeds 59 (ratio greater than 0.92), the two images are considered to have high similarity. If it is less than 54 (ratio less than 0.84), the two images are considered to have low similarity. This invention uses the following discrimination values: when the number of identical bits n in two Hash fingerprints is greater than 59, it indicates that the two images have high similarity and can be considered as images of the same state. When the number of identical bits in two Hash fingerprints is less than 59 and greater than 54, it indicates that the two images are somewhat different but relatively similar, indicating that it is a new state image of the target. If the number of identical bits between two hash fingerprints is less than 54, it indicates that the images are far apart and have a low degree of similarity. This can be judged as an occlusion or tracking error, and no update operation should be performed.

[0112] (3) Occlusion relocation: When the template update branch determines that the current frame is occluded or lost, the template image before the update is input into the template branch, and cross-correlation calculation is performed with the detected video image until the target is found. Then the template update process is restarted.

[0113] like Figure 1 As shown, after determining the similarity between the predicted image and each template image in the template library for this tracking operation using the template update branch in S1025, the maritime target tracking method based on template update SiameseRPN++ includes the following: If the similarity falls within a first preset range, specifically, when it is determined that the current frame image for this tracking operation is not the last frame image, if at least one template image has a similarity to the predicted image for this tracking operation that falls within the first preset range, then the predicted image for this tracking operation is stored as a template image in the template library. The template update strategy corresponding to the template update branch is as follows: Figure 3 As shown.

[0114] In this embodiment, the first preset range is greater than a first preset value and less than or equal to a second preset value. A similarity score within the first preset range indicates that the target in the predicted image for the current tracking operation is in a new state. In this case, the predicted image for the current tracking operation is determined as the target image for the next tracking operation. Alternatively, a Gaussian mixture model is established or updated, and a template branch is used to extract target template features based on the predicted image for the current tracking operation. These target template features are then used as the template features extracted for the next tracking operation, serving as the input to the twin branch. This is equivalent to determining the predicted image for the current tracking operation as the target image for the next tracking operation, making the target image the most suitable template image for the current target state. Determining the predicted image for the current tracking operation as the target image for the next tracking operation, or determining the target template features for the current tracking operation as the template features extracted for the next tracking operation, and using these as the input to the twin branch allows for timely adjustment of the target template based on dynamic changes in the target.

[0115] In real-world scenarios, the appearance of a target can change. For example, in maritime target tracking, a ship's appearance may change depending on the cargo it carries, or it may present different visual effects under different lighting conditions. When the predicted image used for the current tracking operation is used as the target image for the next tracking operation, the model can capture these appearance changes. Because the predicted image is based on the current tracking results, it reflects the target's current actual state. If the target's appearance changes, this change will be reflected in the predicted image. Therefore, in the next tracking operation, the model can use the updated target image (i.e., the previous predicted image) as the new target image, thus adapting to changes in the target's appearance and maintaining accurate target tracking.

[0116] Target attitude changes are also a common dynamic situation. Taking ships at sea as an example, ships may turn or tilt during navigation. By defining the target template features used in the current tracking operation as the template features extracted for the next tracking operation, the model can respond to such attitude changes in a timely manner. When the target attitude changes, the corresponding target template features will also change. In the next tracking operation, using the updated target template features as input to the twin branch allows the model to better match the target's features under the new attitude, improving tracking accuracy and stability.

[0117] In complex environments, dynamic changes in the target can cause tracking interruptions in traditional tracking methods. However, by employing a method that adjusts the target template in real time based on these dynamic changes, the probability of tracking interruptions can be effectively reduced. For example, when a target undergoes significant changes in appearance or posture within a short period, a traditional fixed template may no longer accurately match the target, leading to tracking failure. However, by continuously updating the target image or target template features, the model can continuously adjust its tracking strategy as the target changes, maintaining target lock and ensuring tracking continuity.

[0118] Accurate target templates are crucial for target localization. When the target changes dynamically, a timely updated target template can more accurately reflect the target's current position and characteristics. Taking Siamese twins as an example, they determine the target's position by comparing the target template features with the features of the current frame. If the target template can be updated promptly as the target changes dynamically, the Siamese twin can obtain more accurate information when calculating the target's position, thereby improving the accuracy of target localization and making the tracking results more reliable.

[0119] During long-term tracking, the target may undergo multiple complex dynamic changes, while environmental factors may also constantly change. Through this mechanism of dynamically adjusting the target template, the model can continuously adapt to changes in the target and environment during long-term tracking. For example, when tracking a ship at sea for an extended period, the ship may constantly change its state due to factors such as weather and sea conditions, while environmental factors such as lighting and waves will also change. Timely updates to the target template allow the model to maintain good tracking performance in such complex long-term tracking scenarios, improving the model's robustness and adaptability.

[0120] Conventional SiameseRPN++ selects the target image (e.g., ship N) in the first frame and uses it as a template to find the most similar target in subsequent video frames. This invention adds a template update branch to address issues such as target loss, occlusion, and rapid background changes. To reduce computation, a hash fingerprint is used to determine whether to update the template. If an update is needed, a Gaussian mixture model is built for clustering. As an example, this invention sets eight cluster types; clusters belonging to the same type are merged, meaning the template library contains a maximum of eight templates.

[0121] In this embodiment, the predicted image targeted by this tracking operation is stored in the template library as a template image, and the matching template is updated to express the state of the target in the current time period.

[0122] In this embodiment, the similarity between the predicted image targeted by the current tracking operation and each template image in the template library can be calculated. The similarity can be the number n of identical bits in the hash fingerprint. Specifically, the similarity can be determined by calculating the number n of identical bits in the hash fingerprint (generally 64 bits). In this embodiment, the conventional method in the art is used to calculate the number of identical bits in the hash fingerprint. As an example, the first preset value can be set to 54, the second preset value can be set to 59, and the first preset range is 54 < n ≤ 59.

[0123] The template update branch cuts out the predicted image from the current frame image based on the target bounding box coordinates predicted by the network and the classification results, calculates the hash fingerprint of the predicted image, obtains the similarity with the template image, establishes or updates the Gaussian mixture model, and obtains the template image that is most suitable for the current target state.

[0124] Target tracking algorithms such as Figure 4 As shown, the process is as follows:

[0125] (1) Detect and crop the predicted image of the video, scale it to 127×127 pixels, and input the template to update the branch.

[0126] (2) A Gaussian mixture model is used to model and cluster the state of the predicted image, and a template library is established to construct k Gaussian distributions N(x; μ). k ;I) is used to represent k types of target states.

[0127] (3) Calculate the hash fingerprint (usually 64) of each frame of the predicted image and all template images, and calculate the number of the same bits n.

[0128] (4) If n > 59, it is considered similar to this template feature, and this template feature is directly used as the template feature extracted in the next tracking operation.

[0129] (5) If 54 < n ≤ 59, then establish or update the Gaussian mixture model and use the template branch to extract this feature as the template feature to be extracted in the next tracking operation.

[0130] (6) If n≤54, it is considered to be an occlusion or tracking error and the image is not considered.

[0131] In one alternative implementation, such as Figure 1As shown, after determining the similarity between the predicted image and each template image in the template library for this tracking operation using the template update branch in S1025, the maritime target tracking method based on template update SiameseRPN++ further includes: if the similarity falls within a second preset range, specifically, when it is determined that the current frame image for this tracking operation is not the last frame image, if at least one template image has a similarity with the predicted image for this tracking operation that falls within the second preset range, then the template features extracted in this tracking operation are determined as the template features extracted in the next tracking operation.

[0132] In this embodiment, the minimum value of the second preset range is greater than the second preset value.

[0133] As an example, if the second preset range is n>59, then it is considered that the image is similar to this template image, and the state is similar or the same. The template features extracted in this tracking operation are directly determined as the template features extracted in the next tracking operation, and used as the input of the twin branch to avoid repeated calculations and improve the target tracking efficiency.

[0134] When the similarity between the predicted image and the template image is within a second preset range (e.g., in the case where n>59), it means that the predicted image has a high similarity to the existing template image, and the target's state may not have changed significantly. In this case, using the template features extracted in the current tracking operation as the template features for the next tracking operation ensures that the twin branch uses stable and effective template information in subsequent target localization and classification processes. This helps maintain the continuity and accuracy of tracking when the target state is relatively stable, avoiding errors introduced by unnecessary template updates.

[0135] Updating the template every time, regardless of whether the target state changes substantially, increases computational load. However, by setting a second preset range to determine whether to update template features, when the similarity between the predicted image and the template image falls within this range, the current template features are directly used, reducing unnecessary feature re-extraction and update operations. This significantly saves computational resources and improves the efficiency of the tracking algorithm when handling target tracking tasks with a large number of video frames, enabling the algorithm to process each frame faster and achieve real-time or near real-time target tracking.

[0136] For targets with periodic state changes, such as those in maritime target tracking where certain ships may operate according to fixed routes and operating patterns, their appearance and attitude may periodically revert to similar states. In this case, when the similarity between the predicted image and the template image is within a second preset range, using the current template features can effectively adapt to this periodic change. The model does not need to relearn a new template every time the target state changes slightly; instead, it can utilize existing similar template features for tracking, enhancing the model's adaptability to tracking such targets.

[0137] In one alternative implementation, such as Figure 1 As shown, after determining the similarity between the predicted image and each template image in the template library for this tracking operation using the template update branch in S1025, the maritime target tracking method based on template update SiameseRPN++ further includes: if the similarity falls within a third preset range, specifically, when it is determined that the current frame image for this tracking operation is not the last frame image, if at least one template image and the predicted image for this tracking operation have a similarity within the third preset range, then the next tracking operation is performed. The maximum value of the third preset range is greater than or equal to the first preset value.

[0138] As an example, if the third preset range is n≤54, it is considered an occlusion or tracking error, and the image is ignored, and the next tracking operation is performed.

[0139] In an optional implementation, the maritime target tracking method based on template-updated SiameseRPN++ further includes:

[0140] Once it is determined that the current frame image being tracked is the last frame image, the tracking operation ends, and the target tracking result is obtained. The target tracking result includes the target's position information in each frame of the video frame sequence and the foreground / background classification result.

[0141] This invention employs a training mode of pre-training, segmented training, and weight freezing, enabling transfer learning of maritime scenarios with relatively small datasets and significantly shortening training time. The training method of this invention is as follows:

[0142] Step S1: Image preprocessing.

[0143] The video stream is fully sampled to obtain a video frame sequence. Each frame is then filtered by Wiener to remove motion blur caused by the rapid movement of the target at sea. Each frame is then scaled to obtain a 255*255 video frame sequence with motion blur removed.

[0144] Step S2: Data augmentation.

[0145] Data augmentation methods are used to expand the dataset of the network, including geometric transformation, color transformation, adding noise, and removing motion blur. Images with four flip angles (0°, 90°, 180°, and 270°) with randomly changed RGB channel values ​​and randomly added Gaussian noise are obtained, and corresponding angle labels are created.

[0146] Step S3: Train the ResNet-50 network.

[0147] Load the ResNet-50 pre-trained model on the COCO dataset and use the COCO dataset pre-trained model as the baseline weights. In order to transfer the COCO pre-trained model to the custom task, modify the output layer and the number of outputs of the specific classification head. Since the ResNet-50 pre-trained model on COCO is usually for 80-class classification tasks, while the target categories to be identified in the maritime scene task may be different, it is necessary to replace the fully connected layers (FC layers) of the network with classifiers adapted to the target task. At the same time, freeze the weights of the low-level feature layers (the first 4 layers) and only fine-tune the high-level feature layers (fully connected layers) or a few unfrozen high-level convolutional layers to avoid overfitting and reduce training time. Expand the dataset through data augmentation. Finally, train the fine-tuned ResNet-50 network for the maritime scene.

[0148] Step S4: Train the three SiameseRPN modules in the twin branch.

[0149] The fine-tuned ResNet-50 backbone network (with the classification head removed) is used as the feature extraction network for both the template and detection branches, and the weights of this feature extraction network are frozen. The outputs of the two branches are fed into the Siamese branch, which then passes through three SiameseRPN modules to input the bounding box regression and foreground / background classification results. These results are compared with the ground truth values, and the loss function is calculated. The weights are then updated using backpropagation gradient descent. The dataset used in this step is video footage of maritime target tracking.

[0150] During training, the loss function consists of two parts: classification loss and regression loss. The classification loss uses cross-entropy, while the regression loss uses smooth L1 with normalization. Let (Ax, Ay, Aw, Ah) represent the location information of a certain anchor, and (Tx, Ty, Tw, Th) represent the location information of the ground truth (GT). Then the corresponding normalized distance can be expressed as:

[0151]

[0152] Therefore, The loss function can be expressed as follows:

[0153] σ represents The parameters in the loss function.

[0154] Then the regression loss L reg It can be represented as:

[0155]

[0156] Classification loss L cls Using softmax, it can be expressed as:

[0157] L cls = [clog(c)+(1-c)log(1-c)], where c represents the classification probability, for example, the probability of the foreground class.

[0158] Finally, the total loss is:

[0159] loss = L cls +λL reg λ represents the hyperparameter that controls the weights of the regression loss.

[0160] Where λ is a hyperparameter introduced to balance the two parts of the loss.

[0161] For the outputs of the three SiameseRPN modules, multi-layer fusion is required, and this invention directly performs linear weighting.

[0162]

[0163] Step S5: Train the overall network.

[0164] The weights of the ResNet-50 and SiameseRPN modules in the template branch, detection branch, and twin branch are deregulated, and the entire network is trained using a maritime target tracking video dataset until convergence.

[0165] In summary, this invention discloses a template-updated SiameseRPN++ method for maritime target tracking based on an adaptive SiameseRPN++ network. It adds a template update branch to handle various morphological changes or obstacle occlusion during tracking and proposes an efficient and practical training method. First, the target image selected from the first frame of the video sequence is input into the template branch, while images from the video frame sequence excluding the first frame are input into the detection branch. These images are then processed by the SiameseRPN++ network to obtain the predicted image for the current video frame. Next, the predicted image is input into the template update branch, where a Gaussian mixture model and a perceptual hashing algorithm determine whether to update the template. Finally, the template update strategy is used to continuously update or retain the template for target tracking. Addressing the difficulty of current SiameseRPN++ models in handling target loss or occlusion, this invention designs a template update branch to improve the model's stability for long-term tracking tasks and avoid cumulative drift and accuracy degradation caused by a fixed template. Meanwhile, considering the characteristics of small targets and complex backgrounds in maritime scenarios, the training speed and operability are greatly improved by using a training mode that includes pre-training, segmented training, and weight freezing for template branches, detection branches, and twin branches.

[0166] This invention addresses the challenges of maritime scenarios with wide field of view, minimal environmental changes, and sparse targets. It employs a template-updated SiameseRPN++-based maritime target tracking method to track specific targets. The method involves: inputting the video to be detected into an adaptive SiameseRPN++ target tracking model to obtain target location bounding boxes (target box location information) and foreground / background classification results; then clustering previous prediction results using a Gaussian mixture model to establish a template library; and finally, replacing the input of the template branch in the adaptive SiameseRPN++ network with a template update strategy.

[0167] This invention aims to provide a maritime target tracking method based on template-updated SiameseRPN++: A deep Siamese network is constructed using the SiameseRPN++ architecture. Video frame features are extracted from the template and detection branches using ResNet-50. The template features are then convolved with the current frame features to obtain the target location. A Gaussian mixture model and a template library are used to update and select the most reliable target template, which is then used to update the matching template of the Siamese network, enabling it to adapt to changes in the target's appearance. Finally, a regression model is introduced to further refine the target location and reduce the impact of the background on network performance.

[0168] The beneficial effects of this invention are:

[0169] This invention employs a maritime target tracking model based on an adaptive SiameseRPN++ network to track targets in the video to be detected. Through methods such as few-shot learning, template update strategies, and occlusion relocalization, the target tracking model can stably track maritime targets with a limited number of samples. The invention introduces a template update branch into the SiameseRPN++ network, significantly enhancing the algorithm's robustness and adaptability. By adaptively updating the template to capture changes in target appearance, scale, and pose, the algorithm can better cope with lighting variations, partial occlusion, and background interference in complex scenes, reducing the risk of mistracking. The use of a Gaussian mixture model and perceptual hashing algorithm reduces the template update error rate, thereby improving the model's stability for long-term tracking tasks and avoiding cumulative drift and accuracy degradation caused by a fixed template. Furthermore, considering the characteristics of small targets and complex backgrounds in maritime scenes, the pre-training, segmented training, and weight freezing training modes for the template branch, detection branch, and twin branch greatly improve the training speed and operability of the method.

[0170] It should be noted that the contents not described in detail in this specification are common knowledge to those skilled in the art.

[0171] This embodiment also provides a maritime target tracking device based on template-updated SiameseRPN++. A single device is used to implement the above embodiments and optional implementation methods, and details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0172] Figure 5 This is a schematic diagram of the structure of the maritime target tracking device based on template update SiameseRPN++ according to an embodiment of the present invention.

[0173] This invention provides a maritime target tracking device based on template update SiameseRPN++, applied to a target tracking model. The target tracking model includes a template branch, a detection branch, a twin branch, and a template update branch. Figure 5 As shown, the maritime target tracking device based on template-updated SiameseRPN++ includes:

[0174] The first processing module 11 is used to process the video to be detected and determine the video frame sequence.

[0175] The second processing module 12 is used to perform the following tracking operation until the current frame image is the last frame image of the video frame sequence:

[0176] The second processing module 12 includes a first processing unit 121, a second processing unit 122, a third processing unit 123, a fourth processing unit 124, and a fifth processing unit 125.

[0177] The first processing unit 121 is used to extract template features from the target image for the current tracking operation using a template branch. The target image for the first tracking operation is obtained by cropping the first frame image. The target image for the current tracking operation is stored as a template image in a template library. The template library includes template images of different target states.

[0178] The second processing unit 122 is used to extract current frame features based on the current frame image targeted by the current tracking operation using the detection branch. The current frame image targeted by the first tracking operation is the second frame image.

[0179] The third processing unit 123 is used to determine the tracking result for this tracking operation based on the template features extracted in this tracking operation and the current frame features using the twin branch. The tracking result for this tracking operation includes the target's position information and foreground / background classification results in the current frame image for which the tracking operation is performed.

[0180] The fourth processing unit 124 is used to cut the current frame image targeted by the current tracking operation according to the tracking result targeted by the current tracking operation to obtain the prediction image targeted by the current tracking operation.

[0181] The fifth processing unit 125 is used to determine the similarity between the predicted image targeted by this tracking operation and each template image in the template library using the template update branch.

[0182] If it is determined that the current frame image targeted by this tracking operation is not the last frame image, and if at least one template image has a similarity to the predicted image targeted by this tracking operation within a first preset range, then the predicted image targeted by this tracking operation is stored as a template image in the template library. The first preset range is greater than a first preset value and less than or equal to a second preset value.

[0183] In an optional implementation, the fifth processing unit 125 is further configured to, after determining the similarity between the predicted image targeted by the current tracking operation and each template image in the template library using the template update branch, and when determining that the current frame image targeted by the current tracking operation is not the last frame image, if at least one template image has a similarity to the predicted image targeted by the current tracking operation that falls within a second preset range, then determine the template features extracted in the current tracking operation as the template features extracted in the next tracking operation. The minimum value of the second preset range is greater than a second preset value.

[0184] In an optional implementation, the fifth processing unit 125 is further configured to, after determining the similarity between the predicted image targeted by the current tracking operation and each template image in the template library using the template update branch, and when determining that the current frame image targeted by the current tracking operation is not the last frame image, perform the next tracking operation if at least one template image has a similarity to the predicted image targeted by the current tracking operation within a third preset range. The maximum value of the third preset range is greater than or equal to a first preset value.

[0185] In one optional implementation, the third processing unit 123 is specifically used to perform cross-correlation calculation on the template features extracted in this tracking operation and the current frame features extracted in this tracking operation to obtain the response map targeted by this tracking operation. The response map includes the foreground and background classification results corresponding to each response point.

[0186] Extract response point images from the response image for which the foreground category probability is greater than a preset threshold, and determine them as sample images for this tracking operation.

[0187] The sample images targeted by this tracking operation are input into the region proposal regression model, and the tracking result for this tracking operation is determined using the region proposal regression model.

[0188] In one optional implementation, the third processing unit 123 is specifically used to perform cross-correlation calculation on the shallow features, middle features and deep features of the template features extracted in the current tracking operation, and the shallow features, middle features and deep features of the current frame features extracted in the current tracking operation.

[0189] In an optional implementation, the second processing module 12 includes a sixth processing unit, configured to terminate the tracking operation and obtain the target tracking result when it is determined that the current frame image targeted by the current tracking operation is the last frame image. The target tracking result includes the target's position information and foreground / background classification results in each frame image of the video frame sequence.

[0190] In an optional implementation, the maritime target tracking device based on template update SiameseRPN++ further includes a preprocessing module for preprocessing the video frame sequence after processing the video to be detected to determine the video frame sequence. The preprocessing includes Wiener filtering, cropping, and linear interpolation.

[0191] Further functional descriptions of the above modules and units are the same as those in the corresponding embodiments described above, and will not be repeated here.

[0192] In this embodiment, the maritime target tracking device based on template update SiameseRPN++ is presented in the form of a functional unit. Here, a unit refers to an ASIC (Application Specific Integrated Circuit), a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0193] This invention also provides a computer device having the above-described features. Figure 5 The image shows a marine target tracking device based on template-updated SiameseRPN++.

[0194] Please see Figure 6 , Figure 6 This is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention, such as... Figure 6 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In an alternative implementation, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor device). Figure 6 Take a processor 10 as an example.

[0195] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GPA), or any combination thereof.

[0196] The memory 20 stores instructions executable by at least one processor 10 to cause at least one processor 10 to perform the method shown in the above embodiments.

[0197] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store application programs required for operating the device and at least one function. The data storage area may store data created based on the use of the computer device. Furthermore, the memory 20 may include high-speed random access memory and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In an alternative embodiment, the memory 20 may optionally include memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0198] Memory 20 may include volatile memory, such as random access memory. Memory may also include non-volatile memory, such as flash memory, hard disk, or solid-state drive. Memory 20 may also include combinations of the above types of memory.

[0199] The computer device also includes a communication interface 30 for communicating with other devices or communication networks.

[0200] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded over a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc. Further, the storage medium may also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code, which, when accessed and executed by the computer, processor, or hardware, implements the methods shown in the above embodiments.

[0201] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.

[0202] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A template-based SiameseRPN++ maritime target tracking method applied to a target tracking model, the target tracking model comprising a template branch, a detection branch, a twin branch and a template update branch; characterized in that, The method comprises the following steps of: processing a video to be detected to determine a video frame sequence; performing the following tracking operations until the current frame image is the last frame image of the video frame sequence: extracting a template feature from a target image for which the current tracking operation is performed by using the template branch; the target image for which the first tracking operation is performed is obtained by cropping the first frame image; the target image for which the current tracking operation is performed is stored in a template library as a template image; the template library comprises template images of different states of the target; extracting a current frame feature from the current frame image for which the current tracking operation is performed by using the detection branch; the current frame image for which the first tracking operation is performed is the second frame image; determining a tracking result for which the current tracking operation is performed by using the twin branch according to the template feature and the current frame feature extracted by the current tracking operation; the tracking result for which the current tracking operation is performed comprises position information of the target in the current frame image for which the current tracking operation is performed and a foreground-background classification result; cutting the current frame image for which the current tracking operation is performed according to the tracking result for which the current tracking operation is performed to obtain a predicted image for which the current tracking operation is performed; determining a similarity between the predicted image for which the current tracking operation is performed and each template image in the template library by using the template updating branch; when it is determined that the current frame image for which the current tracking operation is performed is not the last frame image, if there is at least one template image whose similarity to the predicted image for which the current tracking operation is performed belongs to a first preset range, the predicted image for which the current tracking operation is performed is stored in the template library as a template image; the first preset range is greater than a first preset value and less than or equal to a second preset value.

2. The method of claim 1, wherein, after the similarity between the predicted image for which the current tracking operation is performed and each template image in the template library is determined by using the template updating branch, the method further comprises the following steps of: when it is determined that the current frame image for which the current tracking operation is performed is not the last frame image, if there is at least one template image whose similarity to the predicted image for which the current tracking operation is performed belongs to a second preset range, the template feature extracted by the current tracking operation is determined as a template feature extracted by a next tracking operation; the minimum value of the second preset range is greater than the second preset value.

3. The method of claim 1, wherein, after the similarity between the predicted image for which the current tracking operation is performed and each template image in the template library is determined by using the template updating branch, the method further comprises the following steps of: when it is determined that the current frame image for which the current tracking operation is performed is not the last frame image, if there is at least one template image whose similarity to the predicted image for which the current tracking operation is performed belongs to a third preset range, a next tracking operation is performed; the maximum value of the third preset range is greater than or equal to the first preset value.

4. The method of claim 1, wherein, the method of determining the tracking result for which the current tracking operation is performed by using the twin branch according to the template feature and the current frame feature extracted by the current tracking operation comprises the following steps of: performing cross-correlation calculation on the template feature extracted by the current tracking operation and the current frame feature extracted by the current tracking operation to obtain a response map for which the current tracking operation is performed; the response map comprises a foreground-background classification result corresponding to each response point. extract a response point image with a foreground class probability greater than a preset threshold in a response map targeted by the current tracking operation, and determine the response point image as a sample image targeted by the current tracking operation; input the sample image targeted by the current tracking operation into a region proposal regression model, and determine a tracking result targeted by the current tracking operation by using the region proposal regression model.

5. The method of claim 4, wherein, the cross-correlation calculation on the template features extracted by the current tracking operation and the current frame features extracted by the current tracking operation includes: the cross-correlation calculation on the shallow features, the middle features and the deep features in the template features extracted by the current tracking operation, and the shallow features, the middle features and the deep features of the current frame features extracted by the current tracking operation.

6. The method of claim 1, wherein, the method further includes: when it is determined that the current frame image targeted by the current tracking operation is the last frame image, ending the tracking operation to obtain a target tracking result; the target tracking result includes position information of the target in each frame image in the video frame sequence and a foreground-background classification result.

7. The method of claim 1, wherein, after determining the video frame sequence by processing the video to be detected, the method further includes: preprocessing the video frame sequence, the preprocessing including Wiener filtering processing, cropping processing and linear interpolation processing.

8. A device for maritime target tracking based on template updating SiameseRPN++, applied to a target tracking model, the target tracking model comprising a template branch, a detection branch, a Siamese branch and a template updating branch; characterized in that, including: a first processing module configured to process a video to be detected to determine a video frame sequence; a second processing module configured to perform the following tracking operation until a current frame image is a last frame image of the video frame sequence: the second processing module includes a first processing unit, a second processing unit, a third processing unit, a fourth processing unit and a fifth processing unit; the first processing unit is configured to extract template features from a target image targeted by the current tracking operation by using the template branch; the target image targeted by the first tracking operation is obtained by cropping a first frame image; the target image targeted by the current tracking operation is stored in a template library as a template image; the template library includes template images of different states of the target; the second processing unit is configured to extract current frame features from a current frame image targeted by the current tracking operation by using the detection branch; the current frame image targeted by the first tracking operation is a second frame image; the third processing unit is configured to determine a tracking result targeted by the current tracking operation by using the twin branch according to the template features and the current frame features extracted by the current tracking operation; the tracking result targeted by the current tracking operation includes position information of the target in the current frame image targeted by the current tracking operation and a foreground-background classification result; the fourth processing unit is configured to cut the current frame image targeted by the current tracking operation according to the tracking result targeted by the current tracking operation to obtain a predicted image targeted by the current tracking operation; the fifth processing unit is configured to determine a similarity between the predicted image targeted by the current tracking operation and each template image in the template library by using the template updating branch; when it is determined that the current frame image targeted by the current tracking operation is not the last frame image, if there is at least one template image with a similarity to the predicted image targeted by the current tracking operation belonging to a first preset range, the predicted image targeted by the current tracking operation is stored in the template library as a template image. The first preset range is greater than a first preset value and less than or equal to a second preset value.

9. A computer device, comprising: Comprise: The memory and the processor are connected in communication with each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the offshore target tracking method based on template updating SiameseRPN++ in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing a computer to execute the offshore target tracking method based on template updating SiameseRPN++ in any one of claims 1 to 7.

Images