Foggy day target tracking system based on unsupervised domain adaptation
Through the unsupervised domain adaptation method, a video training set of synthetic foggy days and real foggy days was established. The improved domain classifier and adversarial training method were used to solve the performance degradation of target tracking in foggy days scenarios, and efficient tracking in real foggy days environments was achieved.
Patent Information
- Application Number
- CN202311464122.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-11-03
- Publication Date
- 2025-05-06
AI Technical Summary
The existing target tracking technology performs poorly in foggy scenes, resulting in a degradation of tracking performance, especially inadequate generalization capabilities in real foggy environments.
The foggy day target tracking system based on unsupervised domain adaptation is adopted. By establishing a video training set of synthetic foggy days and real foggy days, and using improved domain classifiers and adversarial training methods, we can achieve gradual domain adaptation from clear days to synthetic foggy days and then to real foggy days.
The generalization ability of the target tracking model in real foggy environments is improved, the tracking performance is enhanced, the dependence on a large amount of labeled data is avoided, and the model's inference speed is improved.
Smart Images
Figure HDA0004533041070000011 
Figure HDA0004533041070000012 
Figure HDA0004533041070000013
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of computer vision and unsupervised domain adaptation, and in particular to a foggy target tracking system based on unsupervised domain adaptation. Background Art
[0002] With the development of deep learning in recent years, the field of target tracking has also been booming. Driven by large-scale datasets and supervised by meticulous manual annotations, emerging deep learning trackers have continued to create state-of-the-art performance in recent years. Although target tracking has made great progress, current trackers are trained in clear scenes. Compared with clear scenes, foggy scenes will cause occlusion, image blur, unrecognizable features and other problems, which will greatly reduce the tracking performance of conventional target tracking methods or even make them ineffective. Due to the existence of domain offset between clear and foggy days, the generalization ability of trackers trained in clear scenes in foggy days will drop sharply, thus affecting practical applications.
[0003] One of the most direct solutions is to collect a large-scale foggy dataset and perform detailed manual annotation, and then use it to train the tracker. This will undoubtedly make the tracker's tracking ability stronger in foggy days, but this process is extremely difficult. Foggy scenes are not common, and it is not realistic to shoot a large number of tracking videos in foggy scenes. In addition, the annotation of video datasets is more time-consuming than the annotation of clear sky images.
[0004] In the field of target detection, there have been many precedents that have focused on the above-mentioned issues, but in the field of target tracking, this issue has not received enough attention.
[0005] Fog object detection algorithms can be mainly divided into two categories: algorithms based on image restoration and algorithms based on unsupervised domain adaptation.
[0006] The algorithm frameworks based on image restoration are relatively complex, and the restored clear images sometimes only improve visually, but do not significantly improve the subsequent target detection performance.
[0007] Algorithms based on unsupervised domain adaptation can effectively solve the problem that foggy datasets are difficult to collect and label without affecting the reasoning speed of the model.
[0008] Even though labeling can be avoided through unsupervised methods, it is still difficult to collect a large number of real foggy videos. Therefore, most foggy object detection works based on unsupervised domain adaptation use synthetic fog datasets to achieve domain adaptation from clear days to synthetic foggy days.
[0009] Since there is still a large domain offset between synthetic fog data and real fog data, the model after domain adaptation from clear sky to synthetic fog cannot obtain good generalization ability under real foggy days. Summary of the invention
[0010] The present invention aims to solve the problem that the existing technology cannot solve the poor tracking performance in real foggy weather, and provides a foggy weather target tracking system based on unsupervised domain adaptation. The system includes the following steps:
[0011] Establish a video training set, which includes: a synthetic fog training set without labeled information to achieve domain adaptation of the model from clear sky to synthetic foggy sky, a synthetic fog training set with labeled information, and a real foggy training set without labeled information to achieve domain adaptation of the model from synthetic foggy sky to real foggy sky;
[0012] Construct a foggy tracking network model based on the unsupervised domain adaptation night target tracking (UDAT) algorithm. It uses an improved domain classifier and uses adversarial training to solve the problem of target tracking performance degradation in real foggy weather.
[0013] The method of introducing the intermediate domain is to use the clear sky video training set as the source domain and the synthetic foggy video training set without labeled information as the target domain. Through the first adversarial training, the domain adaptation of the model from clear sky to synthetic foggy sky is achieved. Similarly, the synthetic foggy video training set with labeled information is used as the source domain and the real foggy video training set without labeled information is used as the target domain. Through the second adversarial training, the domain adaptation of the model from synthetic foggy sky to real foggy sky is achieved.
[0014] Manually annotate real foggy videos and use the trained model to track real foggy targets in real time.
[0015] Compared with the prior art, the technical solution provided by the present invention is innovative in that: 1. This paper uses the improved synthetic fog algorithm to propose a synthetic fog video training set for deep learning model training and collects a real fog video training set to further improve the generalization ability of the tracking algorithm in real foggy conditions; 2. The present invention optimizes the network structure of the existing night tracking UDAT algorithm, adopts an improved domain classifier, and forces the model to generate features similar to those of foggy days through adversarial training, thereby effectively solving the problem of target tracking performance degradation in foggy days; 3. The present invention uses the synthetic foggy video training set as the intermediate domain and further improves the generalization ability of the target tracking model in real foggy weather through two adversarial trainings; 4. The present invention uses the trained model to track real foggy videos in real time to demonstrate the effectiveness of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 Schematic diagram of the flow of the foggy weather target tracking system in the embodiment;
[0017] Figure 2 is the atmospheric light scattering model;
[0018] Figure 3 is the transmittance formula;
[0019] Figure 4 Schematic diagram of the original image and the depth map;
[0020] Figure 5 Schematic diagram of medium fog image and heavy fog image;
[0021] Figure 6 It is a schematic diagram of a foggy target tracking system model based on unsupervised domain adaptation;
[0022] Figure 7 Schematic diagram of the intermediate domain training process;
[0023] Figure 8 Schematic diagram of the real foggy target tracking effects of different methods. DETAILED DESCRIPTION
[0024] In order to make the purpose, technical solutions and advantages of the present invention more clear, the following describes the implementation methods of the present invention in detail in combination with specific examples and calculation formulas.
[0025] See also Figure 1 , a foggy target tracking system based on unsupervised domain adaptation, the method comprises the following steps:
[0026] Since there is currently no public foggy data set in the field of target tracking for training the present invention, the primary task of the present invention is to establish a foggy video training set that can be effectively trained. The training set includes: a synthetic fog training set without labeled information to achieve domain adaptation of the model from clear days to synthetic foggy days, a synthetic fog training set with labeled information, and a real foggy training set without labeled information to achieve domain adaptation of the model from synthetic foggy days to real foggy days.
[0027] The method to construct a synthetic fog dataset is as follows: Atmospheric light scattering model reference Figure 2 , where: I(x) is the foggy image, x is the coordinate value of the image pixel, J(x) is the clear image, L is the atmospheric illumination, and t(x) is the transmittance. Transmittance formula see Figure 3 , where: β is the scattering coefficient of haze particles, and d is the distance between the object and the camera.
[0028] By estimating the depth map information, d can be estimated, and then a foggy picture that is consistent with the actual situation can be synthesized, with high fog density in the distance and low fog density in the near distance. Figure 4 .
[0029] In order to solve the problem of large domain shift and further improve the generalization of the model, this paper sets both β and L in a random range and adds noise terms to synthesize two training sets with different concentrations of medium fog and heavy fog, while ensuring that the fog effect of each frame of a video is slightly different. Figure 5 .
[0030] A foggy weather tracking network model is constructed. The model is based on the unsupervised domain adaptation night target tracking (UDAT) algorithm and uses an improved domain classifier to solve the problem of target tracking performance degradation in real foggy days through adversarial training.
[0031] Network Model Reference Figure 6 , the model is improved based on the unsupervised domain adaptation night tracking UDAT. In addition, the present invention proves through experiments that the optimal position and number of domain classifiers are different for different problems. For shallow features such as foggy days, the domain classifier should be placed at a relatively front position of the model, and the number of domain classifiers is not the more the better. Therefore, the present invention places a domain classifier after the last three layers of features in the front-end feature extraction part of the model to replace the domain classifier after the Transformerbridging layer in the original UDAT.
[0032] Specifically, the present invention reduces the latest image classification algorithm fastvitsa36 from 1000 classification tasks to 1 category, and adds a layer of Sigmoid activation function at the end of the model to ensure the stability of the classifier and the output is a number between 0 and 1. The improved classifier is used as the domain classifier of the present invention. During the adversarial training process, the model can better align the feature distribution of clear sky and foggy sky images, further improving the tracking performance of the model in foggy days.
[0033] In order to speed up the training convergence speed, the present invention simplifies the alternating training of the generator and the discriminator in the UDAT algorithm to directly optimizing the generator and the discriminator simultaneously only through GRL.
[0034] A method for introducing an intermediate domain, wherein a synthetic fog training set is used as an intermediate domain, thereby achieving gradual domain adaptation from clear sky to synthetic fog and then to real fog. Specifically, during the first training, the present invention uses clear sky images with labeled information as the source domain and synthetic fog images without labeled information as the target domain to achieve feature alignment between clear sky and synthetic fog. Considering that the foggy images synthesized by the present invention have labeled information, in order to reasonably utilize the labeled information of the synthetic foggy images, during the second training, the present invention uses synthetic foggy images with labeled information as the source domain and real foggy images without labeled information as the target domain to achieve feature alignment between synthetic fog and real fog. For the training process of the intermediate domain, please refer to Figure 7 .
[0035] See also Figure 8 ,The present method can achieve better target tracking effect than other ,methods in real foggy environment.
[0036] Unless otherwise specified, the models of the components in the embodiments of the present invention are not limited, and any device that can perform the above functions may be used.
[0037] Those skilled in the art will appreciate that the accompanying drawing is only a schematic diagram of a preferred embodiment, and the serial numbers of the embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0038] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.
Claims
1. A foggy target tracking system based on unsupervised domain adaptation, characterized in that: The following steps are involved: Establish a video training set, which includes: a synthetic fog training set without labeled information to achieve domain adaptation of the model from clear sky to synthetic foggy sky, a synthetic fog training set with labeled information, and a real foggy training set without labeled information to achieve domain adaptation of the model from synthetic foggy sky to real foggy sky; Construct a foggy tracking network model based on the unsupervised domain adaptation night target tracking (UDAT) algorithm. It uses an improved domain classifier and uses adversarial training to solve the problem of target tracking performance degradation in real foggy weather. The method of introducing the intermediate domain is to use the clear sky video training set as the source domain and the synthetic foggy video training set without labeled information as the target domain. Through the first adversarial training, the domain adaptation of the model from clear sky to synthetic foggy sky is achieved. Similarly, the synthetic foggy video training set with labeled information is used as the source domain and the real foggy video training set without labeled information is used as the target domain. Through the second adversarial training, the domain adaptation of the model from synthetic foggy sky to real foggy sky is achieved. Manually annotate real foggy videos and use the trained model to track real foggy targets in real time.
2. The foggy weather target tracking system based on unsupervised domain adaptation as claimed in claim 1, characterized in that: The video training set established is to set the parameters in the synthetic fog algorithm within a random range and add noise items, synthesizing two training sets with different concentrations of medium fog and heavy fog, while ensuring that the fog effect of each frame of a video is slightly different.
3. The foggy weather target tracking system based on unsupervised domain adaptation as claimed in claim 1, characterized in that: The constructed foggy weather tracking network model places a domain classifier after the last three layers of features in the front-end feature extraction part of the model to replace the domain classifier after the Transformer bridging layer in the original UDAT.
4. The foggy weather target tracking system based on unsupervised domain adaptation as claimed in claim 3, characterized in that: The placed classifier reduces the latest image classification algorithm fastvitsa36 from 1000 classification tasks to 1 category, and adds a layer of Sigmoid activation function at the end of the model to ensure the stability of the classifier and the output is a number between 0 and 1.