Rainy target tracking method based on adaptive rain streak removal and fusion dynamic template matching

Abstract

The present application relates to a kind of rain in target tracking method of adaptive rain streak removal and fusion dynamic template matching, by adaptive rain streak removal and contrast enhancement to input rain image, obtain clearer image for tracking, improve the probability of template matching to correct target, using fusion dynamic template matching strategy mechanism to adjust tracking template in time, make the template to be matched more close to the current real appearance state of target, greatly improve the tracking accuracy and robustness of algorithm when appearance change of target in rainy day condition;Using the present application can quickly and accurately track any target appearing in rainy day scene video sequence robustly, thereby significantly improve the completion efficiency of rainy day tracking task.

Description

Technical Field

[0001] This invention relates to the field of video target tracking, specifically to a method for tracking targets in rain using adaptive rain pattern removal and dynamic template matching. Background Technology

[0002] As a crucial application scenario for video target tracking, rain tracking plays a vital role in civilian fields such as intelligent transportation and flood relief, as well as military fields such as all-weather warfare and rain-based military reconnaissance. However, most current tracking algorithms are designed based on ideal weather conditions. In dynamic weather conditions like rain, the image quality captured by imaging devices often deteriorates significantly, with markedly reduced contrast and visibility, blurred images, and dense rain streaks partially or completely obscuring targets along the way, resulting in the loss of key information and chaotic global feature distribution. This makes conventional tracking algorithms ill-suited for rain tracking tasks.

[0003] Unlike tracking on traditional video sequences, rainy video sequences contain significant interference, making it difficult for conventional trackers to accurately distinguish the target from the chaotic background. This can easily lead to potential tracking inaccuracies and drift. The low contrast and rain streak noise in rainy video frames reduce the effectiveness of feature extraction, and the weaker feature information hinders subsequent matching and localization. Therefore, it is necessary to sharpen the acquired rain images. However, in actual tracking, not every frame needs sharpening; it depends on the degree of rain streak interference on the target. When the rain streak interference is insufficient to affect normal tracking, time-consuming sharpening is unnecessary, and descaling will cause some information loss. Therefore, sharpening is only required when the rain streak interference in the captured image is high to address the tracking drift caused by low contrast and severe rain streak noise. Thus, an efficient adaptive descaling strategy is key to optimizing tracking efficiency.

[0004] Based on a dual-branch deep convolutional neural network structure, Siamese network trackers possess powerful representation capabilities, effectively extracting high-level semantic information from deep target features. They transform the tracking problem into a template matching process, i.e., tracking is achieved by calculating the similarity between deep features. They also use large amounts of data to train the network model offline, achieving an end-to-end tracking process through parameter fixing. However, this structure makes the tracker's performance highly dependent on the quality of the selected template. When tracking under ideal weather conditions, a fixed single template faces challenges due to changes in target appearance and frequent background switching. Extending this to rainy weather tracking introduces the problem of template image quality degradation, further affecting the probability of the template matching the correct target. Therefore, Siamese networks relying solely on a fixed template tracking mechanism cannot achieve sufficiently ideal tracking results.

[0005] One pressing technical problem that needs to be solved by those skilled in the art is how to achieve both high precision and real-time tracking in rainy weather, given the challenges of rain streak noise pollution, image degradation due to reduced contrast, and the constantly changing appearance of the target in rainy video sequences. Summary of the Invention

[0006] To address the shortcomings of existing technologies, this invention provides a visual object tracking method based on adaptive rain streak removal and fused dynamic template matching (SiamDRS).

[0007] To achieve the above objectives, the technical solution adopted by this invention is: an adaptive rain pattern removal and dynamic template matching method for target tracking in rain, comprising the following steps:

[0008] S1. Based on the first frame of the video sequence, input image I1 and the ground truth bounding box (x0, y0, w0, h0) are used to crop the basic template Z1. Based on the subsequent frames, input image I1 is used to crop the template Z1. i Crop the target search region X i ,i∈[2,n];

[0009] S2, combine Z1 and X i The image is fed into an offline pre-trained adaptive rain ripple removal network for sharpening, resulting in a clear, rain-free template image Z1. free and target search image X i free ;

[0010] S3, Z1 free and X i free The feature φ(Z1) is obtained by feeding the data into an offline pre-trained Siamese network to extract features. free ) and φ(X i free Then, a deep cross-correlation operation is performed on it to obtain the multi-semantic feature map R. i ;

[0011] S4. Reduce the dimensionality of the multi-semantic feature map. The data is fed into a classification-regression subnetwork to obtain classification feature response maps. Centrality Feature Response Map and regression characteristic response plot And by combining these three, the predicted bounding box of the target in the current frame is obtained;

[0012] S5. Based on the response diagram and search image X i Calculate the average peak correlation energy and rain streak interference. If they meet the threshold requirements, search for image X from the rainless image based on the tracking results. i free Middle cropping candidate template Z i free And extract the feature φ(Z) i free Then, with the initial template φ(Z1) free Feature fusion is performed, and the resulting new template feature φ(Z) is used as the search feature matching object for the next frame; otherwise, no fusion is performed.

[0013] S6. If the average peak correlation energy value and rain ripple interference degree calculated in step S5 are within a certain threshold range, then the new template feature φ(Z) obtained in step S5 is used to replace the φ(Z1) obtained in step S3. free Otherwise, continue using φ(Z1) free Use it as a tracking template.

[0014] Furthermore, the adaptive rain ripple removal network described in step S2 comprises two parts: a rain ripple interference calculation mechanism and an image deraining sub-network. The specific steps are as follows:

[0015] S2.1 Calculate the template rain image Z1 and the search rain image X i The rain streak interference is approximated by using the rain streak density that approaches the target's surroundings, and consists of two density calculation steps.

[0016] The input rain image is cropped to a suitable size along its center point, making it as close as possible to the target region. Then, mean filtering is used to perform the first separation on the cropped rain image, resulting in a coarse binary image M containing only high-frequency features. This process can be represented as: Where I c (i,j) represents the input rain image, and f represents a median filter operation with a kernel size of (m,n).

[0017] The first density calculation is performed on the coarse binary image M. If the density of rain streaks around the target is lower than the threshold (0.03), it is directly output; otherwise, the second separation is performed.

[0018] For coarse binary images containing a large amount of high-frequency information, it is necessary to further separate the equally high-frequency detail information and rain line information. The second separation process is as follows: First, find all connected components in the coarse binary image M, and traverse all eigenvalues ​​and eigenvectors of each connected component. Based on the eigenvalue relationships, the shape and orientation properties of the principal components in the connected regions can be reflected. Principal component analysis conditions are then set to filter out a refined binary image M containing only rain line information.r The judgment condition is: Where λ i V represents the eigenvalues ​​of the connected matrix. max (i,j) are the coordinates of the eigenvector corresponding to the eigenvalue with the largest absolute value, T1, T2, and These are the thresholds for determining shape and orientation attributes, respectively.

[0019] For the thinned binary image M r A second density calculation is performed. If the rain pattern interference of the image is lower than the threshold (0.10), it is directly output; otherwise, the image sharpening process in step S2.2 is performed.

[0020] S2.2, For rain images Z1 and X that meet the rain removal conditions i First, the data is preprocessed by inputting it into the unsharpened mask channel. Rain images with more prominent rain streaks are more conducive to accurate rain removal. The data augmentation process can be represented as: I * =I r +β(I r -h(I r ), where I and I * Let h and β represent the input image and the enhanced image, respectively. h is a low-pass filter, and β is a weighting coefficient that controls the sharpening intensity.

[0021] Enhanced rain image Z1 U and X i U The image is fed into a Unet-like network for rain pattern removal and contrast enhancement, and then fused to obtain the final clean image Z1. free and X i free This process can be represented as: In the formula This indicates a pixel-wise filtering operation, where K is the multi-scale rain removal prediction filter kernel.

[0022] Furthermore, the Siamese network described in steps S3 to S4 has two main branches: a template branch and a search branch. Both branches use the ResNet50 network structure and share network parameters. The specific training steps are as follows:

[0023] S4.1. Use COCO, ImageNet DET, ImageNet VID and YouTube-BB datasets for training. Based on the annotation information, crop two frames of images to 127×127 and 255×255 respectively with the target as the center, and denoted as Z and X, as inputs to the template branch and search branch.

[0024] S4.2. The Z and X obtained from S4.1 are fed into the backbone network ResNet50 for feature extraction to obtain the feature map T. Z and T X The two feature maps are subjected to deep cross-correlation to obtain a multi-semantic feature map, which is then fed into a classification-regression network to output a classification response map, a centrality response map, and a regression response map.

[0025] S4.3 Calculate the loss for each of the three branches, using cross-entropy loss for classification and IOU loss for regression. If (x0,y0) and (x1,y1) represent the coordinates of the top-left and bottom-right corners of the ground truth bounding box, and (l,t,r,b) represent the distance from the corresponding position (i,j) in the regression response map to the four sides of the bounding box in the input search region, then the regression objective at this point can be expressed as:

[0026]

[0027] use We can calculate the IOU loss between the actual value and the predicted value as follows: The index function Λ(·) in the formula is defined as:

[0028] Centrality branch networks can remove outliers far from the target center, avoiding the influence of low-quality predicted bounding boxes. The centrality loss can be written as:

[0029] In the formula, C (i,j) The centrality score representing the distance from the target at coordinate (i,j) in the centrality response graph is calculated using the following formula:

[0030] The total loss function is: L = L cls +σ1L cen +σ2L reg σ1 and σ2 are hyperparameters used to balance the three-part loss;

[0031] S4.4. Initialize the network parameters according to a normal distribution. Set the initial learning rate to 0.001 and the training batch size to 96. Use the Adam optimizer to iterate and train 20 times to optimize the network parameters and save the results of each iteration.

[0032] S4.5 Test the results of 15 to 20 iterations on the OTB dataset, and fix the optimal parameters as the final training result.

[0033] Furthermore, in step S5, φ(Z1) free ) and φ(Z i freeFeature fusion is performed using a linear weighted fusion method, with the specific formula being φ(Z)=σφ(Z1 ... free )+(1-σ)φ(Z i free ), where α is a weighting coefficient, which is determined to be 0.75 based on experience.

[0034] Furthermore, the average peak correlation energy threshold and rain pattern interference threshold in step S5 are set to 35 and 0.10, respectively. If the average peak correlation energy value of the current frame is higher than 35 and the rain pattern interference is higher than 0.10, the current frame is considered to contain high-quality rainless historical features, and template acquisition and fusion will be performed at this time; otherwise, acquisition and fusion will not be performed.

[0035] Beneficial Effects: The rain scene video target tracking algorithm provided by this invention, based on a Siamese network framework, adaptively removes rain streaks and enhances contrast in the input rain image to obtain a clearer image for tracking, increasing the probability of template matching the correct target. Addressing the problem that traditional tracking algorithms' static single-template tracking mechanism cannot handle changes in target appearance, a fusion dynamic template matching strategy is used to adjust the tracking template in real time, making the template to be matched closer to the target's current true appearance. This significantly improves the tracking accuracy and robustness of the algorithm when the target's appearance changes under rainy conditions. As a high-precision and real-time tracking algorithm, this invention enables fast and accurate robust tracking of any target appearing in a rain scene video sequence, thereby significantly improving the efficiency of rainy tracking tasks. Attached Figure Description

[0036] Figure 1 This is a schematic diagram of the overall structure of the tracking algorithm network in this invention;

[0037] Figure 2 This is a schematic diagram illustrating the principle of the rain pattern interference calculation mechanism in this invention;

[0038] Figure 3 This is a schematic diagram of the adaptive rain ripple removal network structure in this invention;

[0039] Figure 4 This is a comparison chart of the accuracy and success rate of the method of this invention (SiamDRS) with other mainstream tracking methods in the simulation experiment of the UAV123-RL / H synthetic rain tracking dataset;

[0040] Figure 5 This is a comparison chart of the accuracy and success rate of the method of this invention (SiamDRS) with other mainstream tracking methods in the simulation experiment of the UAV20L-RL / H synthetic rain tracking dataset;

[0041] Figure 6This is a comparison chart of the accuracy and success rate of the method of this invention (SiamDRS) with other mainstream tracking methods in the simulation experiment of the DTB70-RL / H synthetic rain tracking dataset;

[0042] Figure 7 This is a comparison chart of the accuracy and success rate of the method of this invention (SiamDRS) with other mainstream tracking methods in a simulation experiment on the RainyTrack75 real rainy day tracking dataset. Detailed Implementation

[0043] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments.

[0044] like Figure 1 As shown, the adaptive rain pattern removal and dynamic template matching method for target tracking in rain specifically includes the following steps S1 to S5.

[0045] S1. Based on the first frame of the video sequence, input image I1 and the ground truth bounding box (x0, y0, w0, h0) are used to crop the original template Z1. Based on the subsequent frames, input image I1 is used to crop the template Z1. i Crop the target search region X i ,i∈[2,n].

[0046] S2, such as Figure 2 As shown, Z1 and X i The image is fed into an offline pre-trained adaptive rain ripple removal network for sharpening, resulting in the enhanced template image Z1. free and target search area X i free The specific implementation methods are S2.1 to S2.2.

[0047] S2.1, such as Figure 3 As shown, the template rain image Z1 and the search rain image X are obtained using a rain pattern interference calculation mechanism. i The rain streak interference is approximated by using the rain streak density that approaches the target's surroundings, and consists of two density calculation steps.

[0048] The input rain image is cropped to a suitable size along its center point, making it as close as possible to the target region. Then, mean filtering is used to perform the first separation on the cropped rain image, resulting in a coarse binary image M containing only high-frequency features. This process can be represented as: In the formula, I c (i,j) represents the input template rain image and the search rain image, and f represents the median filtering operation with a filter kernel size of (m,n);

[0049] The first density calculation is performed on the coarse binary image M. If the density of rain streaks around the target is lower than the threshold (0.03), it is directly output; otherwise, the second separation is performed.

[0050] For a coarse binary image M containing a large amount of high-frequency information, it is necessary to further separate the equally high-frequency detail information and rain line information. The second separation process is as follows: First, find all connected components in the coarse binary image M, and traverse all eigenvalues ​​and eigenvectors of each connected component. Based on the eigenvalue relationships, the shape and orientation properties of the principal components in the connected regions can be reflected. Principal component analysis conditions are then set to filter out a refined binary image M containing only rain line information. r The filtering conditions are: Where λ i V represents the eigenvalues ​​of the connected matrix. max (i,j) are the coordinates of the eigenvector corresponding to the eigenvalue with the largest absolute value, T1, T2, and These are the thresholds for determining shape and orientation attributes, respectively.

[0051] For the thinned binary image M r A second density calculation is performed. If the rain pattern interference of the image is lower than the threshold (0.10), it is output directly; otherwise, it enters the image sharpening processing channel in step S2.2.

[0052] S2.2, For rain images Z1 and X that meet the rain removal conditions i First, the data is preprocessed by inputting it into the unsharpened mask channel. Rain images with more prominent rain streaks are easier to de-rain. The data augmentation process can be represented as: I * =I r +β(I r -h(I r ), where I and I * Let h and β represent the input image and the enhanced image, respectively. h is a low-pass filter, and β is a weighting coefficient that controls the sharpening intensity.

[0053] Enhanced rain image Z1 U and X i U Feeding the image into a Unet-like network for rain pattern removal and contrast enhancement will result in a clean, restored image; this process can be represented as: In the formula This represents a pixel-wise filtering operation, where K is the multi-scale rain removal prediction filter kernel, and I... r and I o These are the input image and the cleaned image after restoration, respectively.

[0054] In practice, we set up four pixel-wise rain removal filter kernels at four different scales to fully satisfy the rain pattern scale of real-world scenes. Each scale's rain removal filter kernel outputs its reconstructed rain-free image. By using a 1×1 convolution kernel to adaptively fuse the four reconstructed images, we obtain the final rain-free template image Z1. free and search image X i free .

[0055] S3, Z1 free and X i free The features are extracted from the offline pre-trained Siamese network to obtain feature φ(Z1). free ) and φ(X i free Then, a deep cross-correlation operation is performed on it to obtain the multi-semantic feature map R. i .

[0056] S4. Reduce the dimensionality of the multi-semantic feature map. The data is fed into a classification-regression subnetwork to obtain classification feature response maps. Centrality Feature Response Map and regression characteristic response plot And based on this, the predicted bounding box of the target in the current frame is obtained; the specific offline pre-training methods are S4.1 to S4.5.

[0057] S4.1. Use COCO, ImageNet DET, ImageNet VID and YouTube-BB datasets for training. Based on the annotation information, crop two frames of images to 127×127 and 255×255 respectively with the target as the center, and denoted as Z and X, as inputs to the template branch and search branch.

[0058] S4.2. The Z and X obtained from S4.1 are fed into the backbone network ResNet50 for feature extraction to obtain the feature map T. Z and T X The two feature maps are subjected to deep cross-correlation to obtain a multi-semantic feature map, which is then fed into a classification-regression network to output a classification response map, a centrality response map, and a regression response map.

[0059] S4.3 Calculate the loss for each of the three branches, using cross-entropy loss for classification and IOU loss for regression. If (x0,y0) and (x1,y1) represent the coordinates of the top-left and bottom-right corners of the ground truth bounding box, and (l,t,r,b) represent the distance from the corresponding position (i,j) in the regression response map to the four sides of the bounding box in the input search region, then the regression objective at this point can be expressed as:

[0060]

[0061] use We can calculate the IOU loss between the actual value and the predicted value as follows: The index function Λ(·) in the formula is defined as:

[0062] Setting up a centrality branch can remove outliers far from the target center, avoiding the influence of low-quality predicted bounding boxes. The centrality loss can be written as:

[0063] In the formula, C (i,j) The centrality score representing the distance from the target at coordinate (i,j) in the centrality response graph is calculated using the following formula:

[0064] The total loss function is: L = L cls +σ1L cen +σ2L reg σ1 and σ2 are hyperparameters used to balance the losses of the three branches, L cls L represents the classification loss. cen L represents the centrality loss. reg This represents the regression loss.

[0065] S4.4. Initialize the network parameters according to a normal distribution. Set the initial learning rate to 0.001 and the training batch size to 96. Use the Adam optimizer to iterate and train 20 times to optimize the network parameters and save the results of each iteration.

[0066] S4.5 Test the results of 15 to 20 iterations on the OTB dataset, and fix the optimal parameters as the final training result.

[0067] S5. Based on the response diagram and search image X i Calculate the average peak correlation energy and rain streak interference. If they meet the threshold requirements, search for image X from the rainless image based on the tracking results. i free Middle cropping candidate template Z i And extract the feature φ(Z) i free Then, with the initial template φ(Z1) free Feature fusion is performed, and the resulting new template feature φ(Z) is used for matching in the next frame; otherwise, no fusion is performed. The fusion and update process of the tracking template can be expressed as: φ(Z) = σφ(Z1) free )+(1-σ)φ(Z i free ), where σ is a constant factor that controls the degree of fusion.

[0068] The mean peak correlation energy (MCE) is a measure of the fluctuation in the response graph, reflecting the variability in the target's appearance. Its mathematical expression is: Where F max F min and F i,j These are the maximum, minimum, and values ​​at (i,j) in the response graph;

[0069] When the target is occluded, causing a change in appearance, the response map exhibits a multi-peak pattern with relatively low peak values; conversely, when the target is clearly visible within the search range, the response map exhibits a single-peak pattern with relatively high peak values. To adapt to the target's appearance in real time and avoid using low-quality historical features as templates, we use the average peak correlation energy (MRE) as a key indicator for the candidate template acquisition strategy. Furthermore, to avoid candidate templates that may contain rain streaks, we only acquire them from the sharpened search image. If the MRE of the current frame is higher than a set threshold, the candidate template is considered to be of high quality, and weighted fusion is performed to update the tracking template; otherwise, no update is performed.

[0070] S6. If the average peak correlation energy value and rain ripple interference value calculated in step S5 are both within the threshold range, then the new template feature φ(Z) obtained in step S5 is used to replace the φ(Z1) obtained in step S3. free Otherwise, continue using φ(Z1) free ).

[0071] The above S1-S4 is the target tracking process, S2 is the adaptive rain pattern removal process, and S5-S6 is the dynamic template matching process. Together they constitute a complete target tracking process. In the actual tracking process, steps S1-S6 are repeated until the tracking task of the entire video sequence is completed. The bounding box of the tracking result is obtained from step S4.

[0072] The effectiveness of this invention is verified through simulation experiments. The simulations were conducted on three synthetic common rain tracking datasets (UAV123-RL, UAV20L-RL, DTB70-RL), three synthetic challenging rain tracking datasets (UAV123-RH, UAV20L-RH, DTB70-RH), and one real rain tracking dataset, RainyTrack75. Performance was compared with some high-performing target tracking methods in recent years. Performance tests were conducted on a PC platform consisting of a Silver4110 2.1GHz CPU and an NVIDIA GeForce RTX2080 GPU, and the results were implemented using PyTorch. The comparison results were reproduced from the original paper, and the performance was evaluated and compared under the same environment.

[0073] Among them, SiamDRS is the method of this invention. This invention only compares its performance with tracking algorithms that also use deep feature extraction networks on the same scale, including the following 6 types:

[0074] 1. PVT++, see literature [1] Li B, Huang Z, Ye J, et al. PVT++: A Simple End-to-EndLatency-Aware Visual Tracking Framework [C] / / Proceedings of the IEEE / CVFInternational Conference on Computer Vision.2023:10006-10016.

[0075] 2. SiamPW-BRO, see literature [2] Feng T, Qiang L. Ranking-based Siamese visual tracking [C] / / Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, New Orleans, LA, USA. 2022: 18-24.

[0076] 3. SiamGAT, see literature [3] Guo D, Shao Y, Cui Y, et al.Graph attentiontracking[C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition.2021:9543-9552.

[0077] 4. LightTrack, see literature [4] Yan B, Peng H, Wu K, et al. Lighttrack: Finding lightweight neural networks for object tracking via one-shot architecturesearch[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition.2021:15180-15189.

[0078] 5. Ocean, see literature [5] Zhang Z, Peng H, Fu J, et al. Ocean: Object-awareanchor-free tracking [C] / / Computer Vision-ECCV 2020:16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16. Springer InternationalPublishing,2020:771-787.

[0079] 6. SiamCAR, see literature [6] Guo D, Wang J, Cui Y, et al. Siamese fullyconvolutional classification and regression for visual tracking [C] / / Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2020: 6269-6277.

[0080] Figure 4-7 The simulation experiments on UAV123-RL / H, UAV20L-RL / H, DTB70-RL / H, and RainyTrack75 benchmarks show a comparison of the accuracy, normalized accuracy, and success rate of the proposed method (SiamDRS) with other tracking methods. The left, middle, and right figures respectively show the accuracy comparison, normalized accuracy comparison, and success rate comparison. As can be seen from the figures, the proposed method significantly outperforms other methods in terms of tracking accuracy and success rate when dealing with various rainy weather scenarios.

[0081] The method described in this invention possesses excellent anti-interference feature extraction capabilities and appearance adaptation capabilities, effectively overcoming the difficulties faced in video target tracking in rainy scenes. While ensuring real-time performance, this method significantly improves the tracking accuracy and robustness of the tracker under rainy conditions, thereby expanding the tracker's application range to all weather conditions.

[0082] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for adaptive rain pattern removal and dynamic template matching for target tracking in rain, characterized in that, Includes the following steps: S1. Input an image based on the first frame of the video sequence. and truth bounding box Cut out the basic template Input images based on subsequent frames Crop out the target search area ; S2, will and The image is fed into an offline pre-trained adaptive rain pattern removal network for sharpening, resulting in a clear, rain-free template image. and target search image ; The adaptive rain pattern removal network comprises two parts: a rain pattern interference calculation mechanism and an image deraining sub-network. The specific steps are as follows: S2.1 Calculate the template image and search area image The rain pattern interference level is determined. If the rain pattern interference level of the image is lower than the threshold, it is directly output; otherwise, subsequent sharpening and restoration processing is performed. S2.2 For rain images that meet the rain removal conditions and First, the data is preprocessed by inputting it into the unsharpened mask channel to obtain a rain image that is more easily perceived by the rain descaling filter kernel. and ; S2.3, will and The input image is processed using a Unet network for rain streak removal and contrast enhancement, and then fused to obtain the final clean image. and ; Step S2.1 includes two density calculation steps: First density calculation: Median filtering is used to perform initial separation on the cropped rain image to obtain a coarse binary image containing only high-frequency features. At this point, for coarse binary images Density measurements are performed, and the results are used to determine whether to perform finer separation processing. A second density calculation is then performed: the coarse binary image M, containing a large amount of high-frequency information, undergoes a second separation based on the set filtering criteria. Principal component analysis was performed to obtain a refined binary image containing only rain line information. At this time, targeting Density measurements are performed, and based on the measurements, it is determined whether to feed the input template and the search rain image into the image deraining sub-network; the initial separation process can be represented as: ,in To input a rain image, Indicates the filter kernel size is The median filtering operation; the filtering criteria set in the second separation are: ,in The eigenvalues ​​of the connected matrix are... These are the coordinates of the eigenvector corresponding to the eigenvalue with the largest absolute value. , and These are the thresholds for determining shape and orientation attributes, respectively. S3, will and Its features are extracted by feeding it into an offline pre-trained Siamese network. and Then, a deep cross-correlation operation is performed between these two features to obtain a multi-semantic feature map. ; S4. Reduce the dimensionality of the multi-semantic feature map. The data is fed into a classification-regression subnetwork to obtain classification feature response maps. Centrality characteristic response map and regression characteristic response plot And by combining these three, the predicted bounding box of the target in the current frame is obtained; S5. Based on the response diagram and search images Calculate the average peak correlation energy (APCE) and rain streak interference. If they meet the threshold requirements, search for images from rain-free areas based on the tracking results. Middle cropping candidate template And extract features Then with the initial template Perform feature fusion and obtain the new template features If a feature is used as the target for matching in the next frame, then no fusion is performed. S6. If the average peak correlation energy value and rain ripple interference degree calculated in step S5 are within a certain threshold range, then the new template features obtained in step S5 are adopted. Replace the result obtained in step S3 Otherwise, continue to use Create a tracking template.

2. The method for adaptive rain pattern removal and dynamic template matching for target tracking in rain as described in claim 1, characterized in that, In step S2.2, for rain images that meet the rain removal conditions... and Data preprocessing is performed by setting an unsharpened mask enhancement channel. The calculation formula for data preprocessing is as follows: In the formula and These represent the input image and the enhanced image, respectively. It is a low-pass filter. It is a weighting coefficient that controls the sharpening intensity.

3. The method for adaptive rain pattern removal and dynamic template matching for target tracking in rain as described in claim 1, characterized in that, In step S2.3, the enhanced rain image is... and The image is fed into a Unet-like network for rain streak removal and contrast enhancement, and then fused to obtain the final clean image. and This process can be represented as: In the formula This represents a pixel-by-pixel filtering operation, where K is the multi-scale rain removal prediction filter kernel. and These are the input image and the cleaned image after restoration, respectively.

4. The method for adaptive rain pattern removal and dynamic template matching for target tracking in rain as described in claim 1, characterized in that, The Siamese network described in steps S3 to S4 has two main branches: a template branch and a search branch. Both branches use the ResNet50 architecture and share network parameters. The specific training steps are as follows: S4.

1. Train using COCO, ImageNet DET, ImageNet VID, and YouTube-BB datasets. Based on the annotation information, crop two frames of images centered on the target. and The dimensions, denoted as Z and X, serve as inputs for the template branch and the search branch; S4.

2. The Z and X obtained from S4.1 are fed into the backbone network ResNet50 for feature extraction to obtain the feature map. and The two feature maps are subjected to deep cross-correlation to obtain a multi-semantic feature map, which is then fed into a classification-regression network to output a classification response map, a centrality response map, and a regression response map. S4.3 Calculate the loss for each of the three branches, using cross-entropy loss for classification and IOU loss for regression; the final loss function is: , Represents classification loss. This represents the centrality loss. Indicates regression loss, and It is a hyperparameter used to balance the three parts of the loss; S4.4 Initialize the network parameters randomly according to a normal distribution, use the Adam optimizer to iteratively train and optimize the network parameters, and save the results of each iteration; S4.5 Test the iteration results on the OTB dataset and select the optimal parameters as the final training result.

5. The method for adaptive rain pattern removal and dynamic template matching for target tracking in rain as described in claim 1, characterized in that, In step S5, the average peak correlation energy (MCE) is a measure of the fluctuation in the response map, reflecting the variability of the target's appearance. It serves as an important indicator for the candidate template acquisition strategy, and its calculation formula is: ,in , and These represent the maximum, minimum, and maximum values ​​in the response graph, respectively. The value at the given point; in this calculation formula, the numerator reflects the reliability of the current response map, and the denominator reflects the average fluctuation of the response map. The smaller the APCE value, the smaller the maximum peak value of the response map or the more peaks in the response map, that is, the higher the degree of target occlusion, and the lower the target feature quality at this time.

6. The method for adaptive rain pattern removal and dynamic template matching for target tracking in rain as described in claim 1, characterized in that, In step S5, for and Feature fusion is performed using a linear weighted fusion method, with the specific formula being: ,in The weighting factor is 0.75, which is determined empirically.

7. The method for adaptive rain pattern removal and dynamic template matching for target tracking in rain as described in claim 1, characterized in that, In step S5, the average peak correlation energy threshold and the rain pattern interference threshold are set to 35 and 0.10 respectively based on experience. If the average peak correlation energy value of the current frame is higher than 35 and the rain pattern interference value is higher than 0.10, it is considered that there are high-quality rainless historical features in the current frame, and template acquisition and fusion will be performed. Otherwise, acquisition and fusion will not be performed.

Images