An outdoor RGB-T target tracking algorithm based on generated pseudo-fusion features

Abstract

The present application relates to an outdoor RGB-T target tracking algorithm based on generating pseudo-fusion features, comprising a pseudo-fusion feature generation network and a weather adaptability enhancement network. The pseudo-fusion feature generation network comprises a method of first channel-by-channel convolution and then dot product for fusion features to obtain more abundant fusion features, and a method of low-computational-load channel exchange to realize information exchange between multi-modalities. The weather adaptability enhancement network focuses on the adaptability of the tracker to weather type changes when performing outdoor tracking tasks, adopts a multi-branch weather type prediction structure, calculates the weather type prediction probability of the feature map by multiple weather feature extraction branches, and adopts a double-modality parallel prediction method to improve the adaptability of the tracker to weather changes and ensure the reliability of the information. The present application combines the heterogeneity of multi-modal data to realize complementary advantages, improve the representation ability of features, and improve the ability to cope with weather changes, thereby realizing robust tracking.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision image processing, and particularly relates to a visible light-thermal infrared (RGB-T) target tracking algorithm based on generated pseudo-fusion features and improved outdoor weather adaptability. BACKGROUND

[0002] Visual target tracking technology has very wide applications and plays an important role in automatic driving, human-computer interaction, battlefield reconnaissance, security monitoring and the like. Based on the continuous development of sensors, the acquisition cost of multi-modal data is reduced, and the use of multi-modal data processing for tracking problems shows a strong development prospect. At present, visible light data and thermal infrared data are mainly applied in the field of multi-modal tracking. The visible light modality can reflect rich scene detail information and retain more texture features, but its imaging is sensitive to changes in light intensity and is easily affected by light intensity, resulting in target blur. In comparison, the thermal infrared modality can capture the thermal radiation distribution of the scene and is not limited by light intensity. Combining the complementary advantages of visible light and thermal infrared data can overcome the limitations in feature representation using only single modal data to a certain extent and achieve better tracking effect.

[0003] At present, multi-modal visual target tracking algorithms often use a relatively complex fusion method to realize mutual attention between modalities in combination with the advantages of visible light-infrared dual modalities. Meanwhile, existing RGB-T target tracking algorithms pay more attention to the fusion between visible light and thermal infrared modalities, and ignore the robustness of the tracker. When performing outdoor target tracking tasks, there are problems such as complex and changeable weather, strong randomness of scenes and the like, and existing multi-modal target tracking algorithms are difficult to cope with the multiple challenges of outdoor tracking tasks.

[0004] In combination with the above problems, the existing multi-modal tracking algorithms generally have problems of complex fusion structure, large amount of calculation and poor weather adaptability. When used for outdoor tracking tasks, the efficiency and accuracy are difficult to guarantee. It is a problem to be solved at present to design an RGB-T tracking algorithm with simple and effective fusion mode and high adaptability to weather changes to realize all-day and all-weather target tracking tasks. SUMMARY

[0005] In order to solve the problems of complex fusion structure, poor scene adaptability, difficult to guarantee tracking efficiency and difficult to adapt to outdoor weather changes when existing multi-modal tracking algorithm is executed in outdoor target tracking task, based on the potential problems that may exist in the actual application scene of the existing RGB-T multi-modal visual target tracking algorithm, on the one hand, the fusion quality and the calculation efficiency of the RGB-T multi-modal fusion are comprehensively considered, the pseudo-fusion feature map of visible light and thermal infrared is obtained in the form of convolution generation, the feature information is further enriched, and the information exchange between the two modes is realized by using the low-computing-capacity channel exchange mode. On the other hand, in order to improve the ability of the RGB-T multi-modal tracker to deal with outdoor weather changes, a weather adaptability enhancement network is designed to enhance the adaptability of the feature to the weather condition to improve the tracking robustness.

[0006] In order to realize the above-mentioned purpose, the specific technical scheme of the present application is as follows:

[0007] An RGB-T outdoor target tracking algorithm combining pseudo-fusion feature generation and weather adaptability enhancement, the method mainly includes designing a pseudo-fusion feature generation network and a weather adaptability enhancement network. Mainly includes four parts of data preprocessing, network model design, offline training, online tracking. As shown in the accompanying Figure 1 The network model structure of the RGB-T multi-modal target tracking algorithm proposed by the present application is shown, and the overall network architecture includes a multi-modal feature extraction network, a pseudo-fusion feature generation network, a weather adaptability enhancement network and a classification network.

[0008] Data preprocessing: the present application uses a weather type dataset to construct a pseudo-RGB-T dataset to participate in the network training process;

[0009] Network model design: the outdoor RGB-T target tracking algorithm based on the generation of pseudo-fusion features designed by the present application has an overall structure of the network model including four parts: a multi-modal feature extraction network, a pseudo-fusion feature generation network, a weather adaptability enhancement network and a target background classification network.

[0010] (1) Multi-modal feature extraction network: the present application uses a double-flow feature extraction network with the same structure but different parameters of visible light mode and thermal infrared mode to extract multi-modal features.

[0011] (2) Generating a pseudo-fusion feature network: a pseudo-fusion feature network is designed to obtain more abundant multi-modal features in a low computational manner and promote information exchange between multi-modal features. Specifically, the multi-modal features after each layer of feature extraction are first channel-connected, and then a low computational cost method of first channel-by-channel and then dot product is used to generate a pseudo-feature map of the fusion features, ensuring that the generated pseudo-feature map of the fusion features and the feature map obtained after the multi-modal feature extraction network operation are the same size and have consistent channel numbers. Subsequently, the pseudo-feature map of the fusion features and the visible light feature map and the thermal infrared feature map obtained after the multi-modal feature extraction network operation are exchanged in channels, respectively, to realize the mutual introduction of visible light information and thermal infrared information between modalities.

[0012] (3) Weather adaptability enhancement network: after the pseudo-fusion feature network processing, in order to improve the adaptability of the tracker to weather changes when performing outdoor tracking tasks, the present application proposes a weather adaptability enhancement network.

[0013] Weather adaptability enhancement network in offline training process:

[0014] The weather adaptability enhancement network in the offline training process aims to obtain a feature extraction branch adapted to multiple weather categories, providing robustness for online tracking. The features V3 D and I3 D are input into the weather adaptability enhancement network, and according to the weather category labels contained in the data set, the features V3 D and I3 D are input into the weather type evaluation branch t j corresponding to the label, where j represents weather category information, including cloudy, rainy, snowy, foggy, and thunderstorm weather. All feature maps after operation of different weather type evaluation branches are sampled for positive and negative samples in the target region and input into the classification network to train the target background classifier.

[0015] Weather adaptability enhancement network in online tracking process:

[0016] The weather adaptability enhancement network in the online tracking process mainly enhances the adaptability of the network to weather and improves the robustness of tracking. The implementation is to evaluate the weather category of the features V3 D and I3 D obtained after the multi-modal feature extraction network and the pseudo-fusion feature network operation, and input the features V3 D and I3 D into the parallel multi-branch weather category evaluation branch t j , where j represents different weather categories. Taking the visible light modality as an example, the visible light modality is evaluated by the weather evaluation branch tj the feature V obtained after operation tj D performing weather type prediction, according to the prediction score, selecting the feature map output by the weather evaluation branch with the highest probability adding it to the feature V3 D to obtain the feature V T , and the feature map I is obtained in the same way T , so as to complete the parallel prediction of the weather type in the multi-modal, and improve the prediction accuracy and adaptability of the image to the weather type.

[0017] (4) Classification network: The classification network includes 2 fully connected layers and 1 classification layer. In the training process, the classification network mainly calculates the target background classification loss and the instance embedding loss. The target background classification loss is mainly used to distinguish the target and the background in each video. The binary cross-entropy loss is used as the loss function, and the classification score output by the network is used to distinguish the target and the background. The instance embedding loss is mainly used to expand the distance of the foreground targets with similar semantic information in the shared feature space in different video sequences. Specifically, for the targets in different sequence branches, the classification scores are different, and the score in the current sequence should be significantly greater than the scores in other sequence branches.

[0018] Offline training: including pre-training model loading, network parameter setting, network loss function setting and model acquisition.

[0019] (1) Pre-training model loading: the first three convolutional layers of the lightweight convolutional neural network VGG-M pre-trained on a large-scale target classification dataset ImageNet are loaded into the double-flow network structure of the present application.

[0020] (2) Network parameter setting: setting the number of iterations, batch size, different learning rates, momentum and weight decay parameters in the training process.

[0021] (3) Network loss function setting: including classification loss and instance embedding loss. The classification loss is used to distinguish the target and the background in each video sequence; the purpose of the instance embedding loss is to distinguish the target in different video sequences and expand the distance of the target between different video sequences. The hyperparameter λ1 is used to balance the instance embedding loss term.

[0022] (4) Model acquisition: according to the training result, the best network parameter model is obtained.

[0023] Online tracking: including model loading, weather type evaluation, regression of initial frame target box, generation of candidate samples, selection of best candidate samples and online update;

[0024] (1) Model loading: loading the best network model obtained by training.

[0025] (2) Weather type evaluation: the feature V3 obtained after the generated pseudo-fusion feature network operation is input into the weather adaptability enhancement network, and the weather type of each feature map is evaluated by using a multi-branch structure. D and I3 D Input into the weather adaptability enhancement network, and the weather type of each feature map is evaluated by using a multi-branch structure.

[0026] (3) Regression of the initial frame target box: the initial frame target tracking result is obtained by using the target box regression method of the initial frame.

[0027] (4) Generation of candidate samples: the candidate sample set {X n m} m＝1,...,256 .

[0028] (5) Selection of the best candidate sample: the classification result of the generated candidate sample is obtained, the samples are sorted according to the classification score, and the sample region with the highest score is selected as the tracking result of the current frame.

[0029] (6) Online update: online update includes short-time update and long-time update. The short-time update is determined by the sample classification score obtained by the classification network. If the score is less than 0, the network is updated for a short time; and the network is updated for a long time every time a fixed interval tracking task is completed.

[0030] Characteristics of the application:

[0031] The application proposes an RGB-T outdoor target tracking algorithm based on generated pseudo-fusion features and weather adaptability enhancement. The application generates the fusion of the features of two modalities, obtains the corresponding pseudo-fusion features, enhances the diversity of information, and uses a low-computing-channel exchange method to exchange information between the two modalities. At the same time, considering that the weather is complex and changeable when the tracker performs an outdoor tracking task, improving the adaptability of the tracker to weather changes is crucial to improve the tracking effect, so a weather adaptability enhancement network is designed to improve the adaptability of the tracker to weather changes. Specifically, a multi-branch weather type evaluation structure is used to extract features in parallel, calculate the evaluation value of the multi-branch weather type, and combine the weighted feature with the highest probability obtained by evaluation with the output of the generated pseudo-fusion feature network. Moreover, the weather adaptability enhancement network uses a dual-modal parallel evaluation method to improve the reliability of information to further enhance the adaptability of the tracker to weather changes. The generated pseudo-fusion feature network and the weather adaptability enhancement network designed in the application realize corresponding optimization in the fusion feature representation of the tracker and the robustness of the tracker, respectively, can complement each other by combining the heterogeneity of multi-modal data, enhance the ability to cope with weather changes in the tracking process, and achieve the purpose of robust tracking. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 . Network model structure diagram

[0033] Figure 2 . Data preprocessing flowchart

[0034] Figure 3 . Offline training process flowchart

[0035] Figure 4 . Online tracking process flowchart

[0036] Figure 5 . One of the experimental results of generating pseudo-fusion feature network

[0037] Figure 6 . Two of the experimental results of generating pseudo-fusion feature network DETAILED DESCRIPTION

[0038] The embodiments of the present application are described in detail below in conjunction with the accompanying drawings of the specification:

[0039] An RGB-T outdoor target tracking method based on generating pseudo-fusion features and weather adaptability enhancement, the algorithm is divided into data preprocessing, network structure design, offline training and online tracking process. The network overall flowchart is shown in Figure 2 .

[0040] The specific content of the data preprocessing part is as follows:

[0041] Data preprocessing: the data preprocessing flowchart is shown in the accompanying Figure 2 The present application uses weather type data set to participate in network training process, the categories include cloudy, rainy, snowy, misty and thunderstorm weather. In order to make the weather type data set suitable for tracking task, the target information such as human and animal in the weather type data set is labeled in order to train the classifier of the tracking algorithm proposed in the present application, and in order to adapt to the multi-modal tracking model, the weather type data set is expanded, the gray image is generated as pseudo-infrared modal data by using the visible light modal data, and the corresponding pseudo-RGB-T image is constructed by combining the original weather type data set of visible light modal, which is applied to the training of the RGB-T multi-modal tracker of the present application.

[0042] The network structure design part is as follows:

[0043] The network structure of the present application is composed of multi-modal feature extraction network, pseudo-fusion feature network, weather adaptability enhancement network and classification network.

[0044] (1) Firstly, the feature extraction is completed by using a multi-modal feature extraction network, the network structure for extracting visible light and thermal infrared features is adopted, the feature extraction network structures of each mode are same but parameters are different, and the structure is referred to as a double-flow network structure. Each mode network branch in the double-flow network structure includes 3 convolution layers, 1 maximum value pooling layer, 3 nonlinear activation ReLU layers and 2 local corresponding normalization layers. Parameters are learned in the training process according to the characteristics of different modal images, so that the weights and biases of the convolution layer are adapted to the characteristics of different modes, and thus the double-flow structure can better utilize the characteristics of multi-modal to extract features, and provide a good foundation for obtaining more representative fused features.

[0045] (2) The generated pseudo-fused feature network is used to fuse the modal features after each layer of feature extraction. The generated pseudo-fused feature network can utilize the multi-modal information to further generate more rich multi-modal fusion information, compared with the fused features obtained by only using the original multi-modal image information, more information with reference significance can be provided, and meanwhile the generated pseudo-fused feature network adopts a simple and effective way, which can provide more beneficial information while reducing the complex calculation cost. The specific implementation is that after inputting the visible light and thermal infrared images, the features V l ∈C×W×H and I l ∈C×W×H are obtained by convolution layer processing, which represent the visible light features and thermal infrared features obtained after the convolution processing of the lth layer respectively, wherein l∈{1, 2, 3}, C represents the channel number of the feature map, specifically, after the first layer of convolution extraction, the channel number of the feature map is 96, after the second layer of convolution extraction, the channel number of the feature map is 256, and after the third layer of convolution extraction, the channel number of the feature map is 512. W represents the width of the feature map, and H represents the height of the feature map. After obtaining the features V l and I l at each feature extraction layer, the feature information of each channel is integrated by using a lightweight per-channel convolution combined with a point-by-point convolution, to obtain the generated pseudo-fused feature A l at each layer. l The generated pseudo-fused feature A l is subjected to a channel exchange operation with the features V l and I l respectively, to obtain the visible light modal features V C and the thermal infrared modal features I l after channel exchange. C The features V l and I C after channel exchange are subjected to a channel exchange operation with the features V l and I C respectively. l lCorresponding addition, get the feature V of each layer after the generated pseudo-fusion feature network operation l D ∈C×W×H and I l D ∈C×W×H.

[0046] (3) Weather adaptability enhancement network mainly improves the adaptability to weather changes in outdoor tracking scenarios, and improves the robustness of tracking. The application adopts a dual-mode parallel structure to identify the weather information contained in the image, improves the discrimination ability of the network to the image, and enhances the robustness of the network.

[0047] In the weather adaptability enhancement network, a multi-branch structure is adopted, according to the category information contained in the different weather type data set, an additional feature extraction layer is added after the main feature extraction network layer, and the weather category information is distinguished. The specific implementation mode is to input the feature V3 D and I3 D into the weather adaptability enhancement network, V3 D and I3 D respectively represent the visible light feature and the thermal infrared feature obtained after the 3rd layer convolution processing, according to the weather category information contained in the feature V3 D and I3 D , the feature map is input into the corresponding weather type evaluation branch t j , wherein j represents the weather category, including cloudy, rainy, snowy, foggy and thunderstorm weather. The feature map obtained after the weather type evaluation branch t j operation is added to the feature V3 D and I3 D , to obtain the required weather adaptability enhancement feature F V and F I , finally the feature maps of the two modalities are combined through addition operation, as shown in formula (1), to obtain the feature map F A input into the classification network.

[0048] F A = F V +F I (1)

[0049] (4) Classification network; the classification network contains 3 fully connected layers in total, the first fully connected layer has 512 output units, and then is sent to the second fully connected layer after ReLU module processing; the second fully connected layer also has 512 output units, and uses ReLU and Dropouts module. The third fully connected layer is used to distinguish the target and the background, adopts binary classification cross-entropy loss as the loss function, and uses the classification score output by the network to realize the distinction between the target and the background.

[0050] The specific steps of the offline training process are as follows: including pre-training model loading; network parameter setting; network loss function setting; model acquisition. The offline training process flow is shown in FIG. 2. Figure 3

[0051] (1) Loading of the pre-training model: the network of the present application adopts a network structure of two branches for extracting visible light image features and thermal infrared image features, respectively, and inputs the images of the visible light and thermal infrared modalities, respectively. The first three convolutional layers of the lightweight convolutional neural network VGG-M pre-trained on the large-scale target classification dataset ImageNet are loaded in the two branches of the feature extraction network. The first layer of convolution includes a 7x7 convolution kernel, a step of 2, a nonlinear activation layer ReLU and a local response normalization layer, and outputs 96 channels; the second layer of convolution includes a 5x5 convolution kernel, a step of 2, a nonlinear activation layer ReLU and a local response normalization layer, and outputs 256 channels; and the third layer of convolution includes a 3x3 convolution kernel, a step of 1, a nonlinear activation layer ReLU and outputs 512 channels.

[0052] (2) Network parameter setting: 100 iterations of the network are set in the training process, and the batch size is set to 8. In each batch, 32 positive samples and 96 negative samples are generated for different frames and sent into the classification network. The learning rate of the feature extraction and feature fusion part is set to 0.0001. The classification network includes a total of 3 fully connected layers, of which the learning rate of the first two fully connected layers is set to 0.0001, and the learning rate of the last fully connected layer is set to 0.001. The momentum and weight decay are set to 0.9 and 0.0005, respectively. Based on the above, the loading of the network pre-training model and the initial model parameter setting are completed, and the iterative training process is started until the training is completed.

[0053] (3) Network loss function setting: in the training process, the network uses the combination of the classification loss and the instance embedding loss as the total classification loss function. The classification loss is mainly used to distinguish the foreground and background in each domain, obtain the classification score of the network output, and use the binary classification cross-entropy loss as the loss function; the instance embedding loss is mainly used to expand the distance of the foreground targets with similar semantic information in different domains in the shared feature space. For the targets in different sequence branches, the score in the current sequence should be significantly greater than that in other sequence branches, and the optimal network parameters are solved by the stochastic gradient descent method.

[0054] The classification loss function is shown in formula (2):

[0055]

[0056] ​where N represents the number of samples, y represents the binary classification label, y e {0, 1}, y i represents the binary classification label corresponding to the i-th sample, y i e {0, 1}. c represents the class label of positive and negative samples, if the binary classification label is consistent with the label of positive and negative samples, (y i ) c = 1, otherwise 0. k represents different video sequences, which can be described as different domains. S represents the binary classification score, which is calculated in different domains represents the binary classification score of the i-th sample in the k-th domain, represents the normalized result calculated by the softmax layer.

[0057] The instance embedding loss aims to distinguish targets in different domains and expand the distance between targets in different domains. Only positive samples are used in the calculation process, which can be represented as formula (3):

[0058]

[0059] In the formula, represents the binary classification score of the i-th positive sample, represents the classification score of the i-th positive sample calculated by the normalization layer, represents the binary classification label corresponding to the i-th positive sample, N represents the number of samples, k represents different video sequences, and K represents the total number of video sequences.

[0060] In summary, the loss function of the overall network training process can be represented as formula (4):

[0061] L = L cls + λ1L inst (4)

[0062] where λ1 is a hyperparameter that controls the balance between loss terms, set to 0.1.

[0063] (4) Model acquisition: For positive and negative samples in different domains k, the corresponding binary classification score S can be obtained in the classification network. Calculate the accuracy score of each positive and negative sample in each domain k in each round, and sum and average to obtain the average classification accuracy score for the domain k. In each round of training, the classification accuracy average score is calculated for all domains. After all domains are calculated, the classification average accuracy of all domains is summed and averaged to obtain the accuracy prediction score S pre of this round of training. As shown in formula 5. In the training process, the value of S pre is compared in each round, and after training, the network training model corresponding to the training round with the maximum S pre value is retained.

[0064]

[0065] where N represents the number of samples in each domain, N k represents the number of domains, represents the binary classification score of the i-th sample in the k-th domain.

[0066] The online tracking process includes model loading, weather type estimation, initial frame target box regression, candidate sample generation, best candidate sample selection, and online updating. The online tracking process flow is shown in FIG. 8, and the specific steps are as follows: Figure 4

[0067] (1) Model loading: the best network model obtained after training according to the network structure shown in FIG. 7 is loaded into the online tracking process to perform the tracking task. Figure 1

[0068] (2) Weather type estimation: the visible light feature V3 D and the thermal infrared feature obtained after the pseudo-fusion feature network operation of the third feature extraction layer are input into the weather adaptability enhancement network, and the weather type of each feature map is evaluated by using the multi-branch structure. The specific implementation manner is that the features V3 D and are input into the weather adaptability enhancement network, each feature map is input into all weather type evaluation branches t j of each modality, and the features corresponding to each weather category are obtained. where j represents the serial number of the weather category information, and respectively corresponds to cloudy, rainy, snowy, foggy, and thunderstorm weather. The feature maps obtained by passing each feature map through all weather type evaluation branches t j are channel concatenated to obtain the feature V t D as shown in equations (6) and (7).

[0069]

[0070]

[0071] The feature maps V t D of each modality after channel concatenation are input into the weather adaptability enhancement network, and the weights w j of each branch are calculated by using the softmax operation. The feature map output by the branch t j with the highest weight w j is selected, as shown in equations (8) and (9).

[0072] ​​

[0073]

[0074] The feature map with the highest weight is fused with the obtained weight w j The visible light feature obtained after the weighted fusion and the operation of the generated pseudo-fusion feature network of the third feature extraction layer and the thermal infrared feature The modality corresponding addition is performed to obtain the required weather adaptability enhanced feature F V and F I , as shown in formula (10) and formula (11).

[0075]

[0076]

[0077] Finally, the feature maps of the two modalities are combined through an addition operation, as shown in formula (1), to obtain a feature map F A which is input into a classification network.

[0078] (3) Regression of the initial frame target box: when performing a tracking task, first, the weights and biases of the classification layer are learned by using the initial frame to improve the adaptability of the tracking target in the new domain. The specific implementation manner is that the true value boundary box of the initial frame target is taken as the center, 500 positive samples are collected according to the overlapping degree (IOU)>0.7, and 5000 negative samples are collected according to the overlapping degree (IOU)<0.5. After passing through the two fully connected layers in the classification network in sequence, a feature vector with a size of 512 is obtained and input into the classification layer. In the classification layer, first, a Dropouts layer is used to prevent network overfitting, and then a softmax loss is used to obtain the classification scores of each sample, both of which have a size of 2 dimensions. In this way, the classification layer for the new domain is obtained, and the subsequent target tracking task is continued to be completed.

[0079] (4) Generating candidate samples: a candidate sample set is collected from the predicted position of the previous frame through Gaussian sampling

[0080] (5) Selecting the best candidate sample: the target candidate sample is processed through the fully connected layer and the classification layer, and the classification result of the candidate sample is output. According to the classification scores of the samples, the samples are sorted according to the scores, and the sample region with the highest score is selected as the tracking result of the current frame. The selection of the best sample is shown in formula (12):

[0081]

[0082] wherein m is the candidate sample label m=1, 2,..., 256, represents the classification score of the mth positive sample, This represents the candidate sample with the highest classification score after network calculation.

[0083] (6) Online Update: The network is updated in two ways during online tracking. First, when the classification network obtains the classification score of the best sample... A value less than 0 indicates tracking failure, and the network is updated once; this is called a short-term update. Secondly, when a tracking task is completed at fixed intervals, the network is updated once; this is called a long-term update. Both types of online updates involve the construction and updating of the sample library. The required sample set differs depending on the situation. During online tracking, when tracking is successful, random sampling is performed around the predicted target location in the current frame to collect data. (50) positive samples were added to the positive sample set In the middle. Collection Add one negative sample to the negative sample set. In the middle. Number of frames successfully tracked f + (100~120) is the set threshold, and the number of positive sample frames f - (20~30) is the set threshold. When the number of successfully tracked frames f... + When the set threshold of 100-120 successfully tracked frames is reached, samples collected from the earliest frame in the positive sample set are deleted. When the number of successfully tracked frames f... + When the set threshold of 20-30 successful tracking frames is reached, the negative samples collected in the earliest frame of the negative sample set are deleted. If the number of successful tracking frames f... + If the set threshold of 20-30 frames is not met, a positive sample set is used. and negative sample set All samples in the dataset have been updated online in a short time; when the tracking success rate is f, the number of frames has been successfully tracked. + When the number of frames exceeds the set threshold, the positive samples used for short-term updates come from the positive sample set. Positive samples are collected from the latest frames that are the same number as the threshold, while negative samples come from the negative sample set. The network collects all negative sample sets. For long-term online updates, all positive and negative sample sets within a set threshold frame number are used. For short-term and long-term updates, the classification loss is calculated using the binary cross-entropy loss function based on the corresponding positive and negative sample sets. Based on the calculated classification loss, gradient descent is used to optimize the weights and biases of the classification network to ensure robustness and adaptability of target tracking. The classification network consists of two fully connected layers and one classification layer. The first fully connected layer has 512 output units, which are processed by a ReLU module and fed into the second fully connected layer to obtain 512 output units. ReLU and Dropout modules are used to prevent overfitting. The classification layer is 2D, obtaining the classification results for the target and background.

[0084] Experimental result evaluation:

[0085] The main contents of the present application include adding weather category dataset to tracking, designing a pseudo-fusion feature network and a weather adaptability enhancement network.

[0086] The experimental results of generating the pseudo-fusion feature network are as follows, wherein the experimental results of generating the pseudo-fusion feature network are named part1, and the experimental results show that, as shown in Figure 5 , compared with the basic RGB-T target tracking network structure RT-MDNet+RGBT, only using the generated pseudo-fusion feature network, the accuracy is improved by 3.1% as shown in Figure 6 , only using the generated pseudo-fusion feature network, the success rate is improved by 3.8% compared with the RT-MDNet+RGBT algorithm.

[0087] The weather category dataset and the weather adaptability enhancement network designed in the present application are mainly used to enhance the weather adaptability of the outdoor RGB-T tracker, improve the robustness of the tracker, and in combination with the good performance of the generated pseudo-fusion feature network, the overall network structure designed in the present application will be able to maintain high tracking performance while improving the adaptability of the outdoor RGB-T visual target tracker.

Claims

1. An outdoor RGB-T target tracking method based on generating pseudo-fusion features, characterized in that: The network structure is composed of a multi-modal feature extraction network, a pseudo-fusion feature generation network, a weather adaptability enhancement network, and a classification network. (1) First, the multi-modal feature extraction network is used to complete the extraction of features, and a network structure for extracting visible light and thermal infrared features respectively is adopted. The feature extraction network structure of each modality is the same, which is called a double-flow network structure. Each modality network branch in the double-flow network structure includes 3 convolutional layers, 1 maximum value pooling layer, 3 nonlinear activation ReLU layers, and 2 local response normalization layers. (2) The pseudo-fusion feature generation network is used to fuse the deep features between modalities after each layer of feature extraction. The specific implementation involves inputting visible light and thermal infrared images, which are then processed by a convolutional layer to obtain feature V. l ∈C×W×H and I l Let C×W×H represent the visible light and thermal infrared features obtained after the l-th convolutional layer, respectively, where l∈{1,2,3}, C represents the number of channels in the feature map. Specifically, after the first convolutional layer, the feature map has 96 channels; after the second convolutional layer, it has 256 channels; and after the third convolutional layer, it has 512 channels. W represents the width of the feature map, and H represents the height of the feature map. Feature V is obtained at each feature extraction layer. l and I l Then, lightweight channel-wise convolution is used, combined with pointwise convolution to integrate the feature information of each channel, resulting in the generated pseudo-fusion feature A for each layer. l This will generate pseudo-fusion feature A. l Respectively with feature V l and I l Perform a channel swap operation to obtain the visible light mode features V after the channel swap. l C ∈C×W×H and thermal infrared modal characteristics Then swap the channels and the feature V l C and Respectively with feature V l and I l By adding the corresponding features, we obtain the feature V of each layer after the pseudo-fusion feature network operation. l D ∈C×W×H and (3) In the weather adaptability enhancement network, a multi-branch structure is adopted, and according to the category information contained in different weather type data sets, an additional feature extraction layer is added after the main feature extraction network layer to distinguish the weather category information; the specific implementation is to input the feature and into the weather adaptability enhancement network, and respectively represent the visible light feature and the thermal infrared feature obtained after the third layer of convolution processing when l=3, according to the weather category information contained in the features and , the feature maps are input into the corresponding weather type evaluation branch t j , where j represents the weather category, including cloudy, rainy, snowy, foggy and thunderstorm weather; the feature maps obtained after the operation of the weather type evaluation branch t j are added to the features and , and the required weather adaptability enhancement feature F V is obtained. I Finally, the feature maps of the two modalities are combined through addition operation, as shown in formula (1), to obtain the feature map F A input into the classification network. F A = F V + F I (1) (3) The classification network includes 3 fully connected layers. The first fully connected layer has 512 output units, which are processed by the ReLU module and sent to the second fully connected layer. The second fully connected layer also has 512 output units and uses the ReLU and Dropouts modules. The third fully connected layer is used to distinguish targets and backgrounds, adopts a binary cross-entropy loss as the loss function, and uses the classification score output by the network to distinguish targets and backgrounds. The specific steps of the offline training process are as follows: including pre-trained model loading; network parameter setting; network loss function setting; model acquisition. (1) Loading of the pre-trained model: the network structure of the two branches for extracting visible light image features and thermal infrared image features is adopted, the images of the visible light and thermal infrared modalities are input, and the first three convolutional layers of the lightweight convolutional neural network VGG-M pre-trained on the large-scale target classification dataset ImageNet are loaded in the two branches of the feature extraction network. The first layer of convolution includes a 7×7 convolution kernel, a step of 2, a nonlinear activation layer ReLU, and a local response normalization layer, outputting 96 channels. The second layer of convolution includes a 5×5 convolution kernel, a step of 2, a nonlinear activation layer ReLU, and a local response normalization layer, outputting 256 channels. The third layer of convolution includes a 3×3 convolution kernel, a step of 1, a nonlinear activation layer ReLU, and outputs 512 channels. (2) Network parameter setting: 100 iterations of the network are set during the training process, the batch size is set to 8, and 32 positive samples and 96 negative samples are generated for different frames in each batch and sent to the classification network. The learning rate of the feature extraction and feature fusion part is set to 0.0001. The classification network includes 3 fully connected layers, the learning rate of the first two fully connected layers is set to 0.0001, the learning rate of the last fully connected layer is set to 0.001, the momentum and weight decay are set to 0.9 and 0.0005, respectively. Based on the above, the network pre-trained model loading and initial model parameter setting are completed, and the iterative training process is started until the training is completed. (3) Network loss function setting: in the training process, the network uses the combination of classification loss and instance embedding loss as the total classification loss function; the classification loss function is shown in formula (2): where N denotes the number of samples, y denotes the binary classification label, y e {0, 1}, y i represents the binary classification label corresponding to the i-th sample, y i e {0, 1}; c represents the class label of positive and negative samples, if the binary classification label is consistent with the label of positive and negative samples, then (y i c = 1, otherwise 0; k represents different video sequences, which can be described as different domains; S represents the binary classification score, which is calculated in different domains represents the binary classification score of the i-th sample in the k-th domain, represents the normalized result calculated by the softmax layer;​ The instance embedding loss is calculated only by positive samples in the calculation process, which can be represented by formula (3): wherein denotes the binary classification score of the i-th positive sample, denotes the classification score of the i-th positive sample calculated by the normalization layer, denotes the binary classification label corresponding to the i-th positive sample, N denotes the number of samples, k denotes different video sequences, and K denotes the total number of video sequences; In summary, the loss function of the whole network training process can be represented by formula (4): L = L cls + λ1L inst (4) where λ1is a hyperparameter that controls the balance between the loss terms and is set to 0.

1. (4) Model acquisition: for positive and negative samples in different domains k, the corresponding binary classification scores S can be obtained in the classification network, the accuracy score of each positive and negative sample in each round is calculated, summed and averaged to obtain the average classification accuracy score for the domain k; the classification accuracy average score of all domains is calculated step by step in each round of training, and after the calculation of all domains is completed, the classification average accuracy of all domains is summed and averaged to obtain the accuracy prediction score S of this round of training pre ; as shown in formula 5; compare the size of S pre value in each round during training, and after training, the network training model corresponding to the training round with the maximum S pre value is retained; where N represents the number of samples in each domain, N k represents the number of domains, represents the binary classification score of the i-th sample in the k-th domain; The online tracking process includes model loading, weather type estimation, initial frame target box regression, candidate sample generation, best candidate sample selection, and online updating. (1) Model loading: load the network model obtained after training into the online tracking process to perform the tracking task; (2) Weather type estimation: the visible light feature and the thermal infrared feature obtained after the generated pseudo-fusion feature network of the third feature extraction layer is operated are input into a weather adaptability enhancement network, and a multi-branch structure is used to evaluate the weather type of each feature map; a specific implementation manner is that the features and the features are input into the weather adaptability enhancement network, each feature map is input into all weather type evaluation branches t of each modality, and features corresponding to each weather category are obtained j wherein j represents the serial number of weather category information, and respectively corresponds to cloudy, rainy, snowy, foggy and thunderstorm weather; the feature maps obtained after each feature map is subjected to all weather type evaluation branches t are channel concatenated to obtain a feature V j t D as shown in formulas (6) and (7);​​​​​​ The feature map V of each modality after channel concatenation t D The weight w of each branch is calculated by a softmax operation j The weight w of each branch is calculated by a softmax operation j The feature map output by the branch t with the highest weight w is selected j The feature map output by the branch t with the highest weight w is selected The feature map with the highest weight is fused with the obtained weight w j The visible light feature obtained after the weighted fusion and the operation of the generated pseudo-fusion feature network of the third feature extraction layer and the thermal infrared feature The modality corresponding addition is performed to obtain the required weather adaptability enhancement feature F V and F I as shown in formula (10) and formula (11); Finally, the feature maps of the two modalities are merged by an addition operation, as shown in equation (1), to obtain a feature map F A into the classification network; (3) Regression of the initial frame target box: when performing the tracking task, first learn the weights and bias of the classification layer using the initial frame to improve the adaptability to the tracking target in the new domain; the specific implementation is to collect 500 positive samples with an overlap degree (IOU)>0.7 and 5000 negative samples with an overlap degree (IOU)<0.5 around the true value boundary box of the initial frame target as the center, then pass them through two fully connected layers in the classification network to obtain a feature vector of 512 size, then pass the feature vector through the Dropouts layer in the classification layer to prevent network overfitting, and then use the softmax loss to obtain the classification score of each sample, both of which are 2-dimensional; thus, the classification layer for the new domain is obtained, and the subsequent target tracking task is continued; (4) Generate candidate samples: collect a candidate sample set from the predicted position of the previous frame by means of Gaussian sampling (5) Selecting the best candidate sample: the target candidate sample is processed by the full connection layer and the classification layer, and the classification result of the candidate sample is output; according to the classification score of the sample, the samples are sorted according to the score, and the sample region with the highest score is selected as the tracking result of the current frame, and the best sample The selection of the best sample is shown in formula (12): wherein m is a candidate sample label m = 1, 2,..., 256, denotes a classification score of the mth positive sample, denotes a candidate sample with the highest classification score after the network computation; (6) Online updating.

2. The method of claim 1, wherein, The online updating is specifically: During online tracking, the network is updated in two scenarios: first, when the classification network obtains the classification score of the best sample. If the value is less than 0, it indicates that the tracking has failed, and the network is updated once, which is called a short update. Secondly, when the tracking task is completed at a fixed interval, the network is updated once, which is called a long update. Both types of online updates of the network involve the construction and updating of the sample library. The sample set required for the update is different for different situations. During online tracking, when tracking is successful, random sampling is performed around the predicted target location in the current frame to collect data. (50) positive samples were added to the positive sample set In China; collection (200) negative samples were added to the negative sample set. In; number of frames successfully tracked f + (100~120) is the set threshold, and the number of positive sample frames f - (20~30) is the set threshold; when the number of successfully tracked frames f + When the set threshold of 100-120 successful tracking frames is reached, the samples collected from the earliest frame in the positive sample set are deleted; when the number of successful tracking frames f... + When the set threshold of 20-30 successful tracking frames is reached, the negative samples collected in the earliest frame of the negative sample set are deleted; if the number of successful tracking frames f + If the set threshold of 20-30 frames is not met, a positive sample set is used. and negative sample set All samples in the dataset have been updated online in a short time; when the tracking success rate is f, the number of frames has been successfully tracked. + When the number of frames exceeds the set threshold, the positive samples used for short-term updates come from the positive sample set. Positive samples are collected from the latest frames that are the same number as the threshold, while negative samples come from the negative sample set. The network collects all negative sample sets. During long-term online updates, all positive and negative sample sets within a set threshold frame number are used. For short-term and long-term updates, the classification loss is calculated using the binary cross-entropy loss function based on the corresponding positive and negative sample sets. Based on the calculated classification loss, the weights and biases of the classification network are optimized using gradient descent to ensure the robustness and adaptability of target tracking. The classification network consists of two fully connected layers and one classification layer. The first fully connected layer has 512 output units, which are processed by the ReLU module and fed into the second fully connected layer to obtain 512 output units. ReLU and Dropout modules are used to prevent overfitting. The classification layer is 2D and obtains the classification results of the target and background.

Images