Visible Light and Infrared Target Tracking Method and Device Based on Modal Competition and Collaboration Network

Abstract

The present invention discloses a visible light and infrared target tracking method and device based on a modal competition and cooperation network. The method includes: generating a first candidate sample set centered on the predicted position of the target in the previous frame of the current frame to be tracked; inputting the first candidate sample set into a trained tracking model to obtain the target tracking result of the current frame; the tracking model includes a feature extraction network and a target classifier; the feature extraction network includes a first extraction network, a second extraction network, and a weight allocation network. Both the first extraction network and the second extraction network include the first three convolutional layers of VGG-M, and each convolutional layer contains N convolutional kernels of the same size to form a convolutional kernel group. By introducing the convolutional kernel group and multiple weight allocation networks to generate a multi-core dynamic cooperative convolutional kernel, and using several convolutional kernels included in the convolutional kernel group, more diverse features can be extracted, achieving a good multi-modal video tracking effect.

Description

Technical Field

[0001] The present invention relates to the technical field of image recognition, and particularly relates to a visible light and infrared target tracking method and device based on a modal competition and cooperation network. Background Art

[0002] Object tracking is a very important research field in computer vision and has received a great deal of attention in recent years. Object tracking has a wide range of application fields in the industrial community, such as monitoring, autonomous driving, industrial robots, etc.

[0003] Currently, object tracking based on a detection and tracking framework is essentially a binary classification problem of an object and the background based on a well-performing classifier. First, at the given object position in the first frame, a threshold is set to confirm the object and background samples to initialize the object tracking model, and then the object in subsequent frames is tracked. At the same time, the size of the object is scaled differently in the first frame to obtain a large number of object sample sets to train the size regression model. In subsequent frames, Gaussian distribution sampling is performed near the object position predicted in the previous frame, and the classifier is used to distinguish between object samples and background samples, and the sample with the highest score is selected as the predicted position of the object in the next frame. After obtaining the rough orientation of the object, the size regression model trained in the first frame is used to regress and predict the length and width of the object to obtain a more refined result.

[0004] Although the performance of current single-modal (visible light) object tracking models is continuously improving, in some specific challenges, such as under illumination changes, fast motion, occlusion, etc., the performance of single-modal object tracking models is often poor. In order to obtain an environmentally robust tracking model, thermal infrared modal data is introduced into the object tracking field due to its characteristic of being insensitive to illumination changes as it reflects the surface temperature distribution in the scene. However, since thermal infrared modal data reflects the temperature information of the scene, a large amount of detailed information is missing, and the resolution is low, and it often cannot correctly distinguish similar objects in some scenes.

[0005] Therefore, taking advantage of the complementary characteristics of visible light modality and thermal infrared modality data, from the perspective of extracting more robust features and more effective feature fusion, more robust RGB-T object tracking in different scenarios is achieved by extracting modality-adaptive features. However, how to effectively extract the two modality-adaptive features is a problem that needs to be solved currently. The existing RGB-T object tracking models mainly use a simple three-layer convolutional network to extract the features of the object. The reason is that the given object in object tracking is often relatively small, so the depth of the feature extraction network in network design is relatively shallow. Such a design makes the feature extraction ability of the feature extraction network weaker compared to, for example, the ResNet-Series, thus becoming a bottleneck in improving the performance of the object tracking model. And the existing methods use convolutional kernels with invariant parameters to extract the features of the object. Such convolutional kernels cannot effectively face the changes in the environment and often cannot extract valuable features when the environment is bad.

[0006] For example, the invention patent application with the application number 202110359997.X discloses an RGBT object tracking method based on cross-modal sharing and specific representation forms. By registering and annotating the visible light video and the corresponding thermal infrared video, and dividing them into a training set and a test set according to requirements; using the VGG-M network to extract modality-specific features of different modalities, using a shared feature extraction module to extract modality-shared features between the two modalities, and at the same time using a multi-scale feature fusion strategy in the shared feature extraction module to enhance the features to improve their robustness. The object can be tracked under extreme conditions such as strong light changes and nights. By using the cross-entropy loss function, the trained model pays more attention to difficult-to-classify samples, improving the robustness of the model. Its essence is to achieve object tracking through multi-scale and feature fusion.

[0007] To sum up, the related technologies have the following problems:

[0008] (1) In visible light and infrared visual object tracking, since the size of the object in the tracking task is relatively small, the feature extraction networks adopted by mainstream methods are often relatively shallow considering the receptive field problem. Therefore, the feature extraction ability is relatively weak compared to top-notch models, thus limiting the improvement of the performance of the object tracking model.

[0009] (2) Regarding the problem of how to make good use of the features of the visible light and infrared modalities in visible light and infrared visual tracking, the mainstream methods adopt learning the weights of the modalities to guide the feature fusion of visible light and infrared. However, they ignore a fact: only by extracting rich and valuable features can it be better to judge whether the object is the target. Therefore, the features they extract are often of poor quality in some harsh environments, resulting in the final performance being difficult to reach the optimal.

[0010] (3) Due to the dynamic characteristics of its network parameters, the effect of the dynamic convolution network is often better than that of the traditional CNN. However, in the dynamic convolution network, during training, the learning of one convolution kernel in each layer may inhibit the learning of other convolution kernels, resulting in invalid parameters for some convolution kernels, thus leading to poor training effects. Summary of the Invention

[0011] The technical problem to be solved by the present invention is how to improve the feature extraction ability of the feature extraction network.

[0012] The present invention solves the above technical problems through the following technical means:

[0013] On the one hand, a visible light and infrared target tracking method based on a modal competition and cooperation network is adopted. The method includes:

[0014] Taking the predicted position of the target in the previous frame of the current frame to be tracked as the center, a first candidate sample set is generated, and the first candidate sample set includes visible light samples and thermal infrared samples;

[0015] Inputting the first candidate sample set into the trained tracking model to obtain the target tracking result of the current frame;

[0016] Among them, the tracking model includes a feature extraction network and a target classifier, and the output of the feature extraction network is connected to the target classifier; the feature extraction network includes a first extraction network, a second extraction network, and a weight allocation network. The visible light samples and the thermal infrared samples are respectively used as the inputs of the first extraction network and the second extraction network. Both the first extraction network and the second extraction network include the first three convolutional layers of VGG-M, and each convolutional layer contains N convolutional kernels of the same size to form a convolutional kernel group; the first candidate sample set is used as the input of the weight allocation network. The first weight group output by the weight allocation network is connected to each convolutional kernel group in the first extraction network through a multiplication operation and then used as the input of the first extraction network. The second weight group output by the weight allocation network is connected to each convolutional kernel group in the second extraction network through a multiplication operation and then used as the input of the second extraction network.

[0017] The present invention generates a multi-core dynamic collaborative convolution kernel by introducing a convolutional kernel group and multiple weight allocation networks. By using the several convolutional kernels included in the convolutional kernel group, more diverse features can be extracted. The weight allocation network is used to allocate the most suitable weights for the convolutional kernels in the convolutional kernel group in the current modality and current environment. After fusion, the best convolutional kernel in the current modality can be obtained. After convolving the convolutional kernel with the input, the most suitable feature representation in the current modality can be obtained, thereby achieving a good multi-modal video tracking effect.

[0018] Further, a rectified linear unit and a local response normalization layer are connected after each of the convolutional kernel groups in the first extraction network and the second extraction network.

[0019] Further, the weight assignment network includes a global average pooling layer, a modality-shared fully connected layer, and a modality-specific fully connected layer. The first candidate sample set is used as the input of the global average pooling layer. The output of the global average pooling layer is connected to the input of the modality-specific fully connected layer through the modality-shared fully connected layer. The modality-specific fully connected layer is used to output the first weight group and the second weight group.

[0020] Further, the target classifier includes two domain-shared fully connected layers and one domain-specific fully connected layer. The two domain-shared fully connected layers are connected and then connected to the input of the domain-specific fully connected layer. The output of the domain-specific fully connected layer is used as the target tracking result of the current frame;

[0021] The domain-shared fully connected layer includes a fully connected layer, a rectified linear unit, and a neuron random activation function connected in sequence. The domain-specific fully connected layer includes a fully connected layer, a neuron random activation function, and a Softmax layer connected in sequence.

[0022] Further, after inputting the first candidate sample set into the trained tracking model to obtain the target tracking result of the current frame, the method further includes:

[0023] Taking the target tracking result of the current frame as a rough result, and determining whether the tracking is successful based on the rough result;

[0024] If so, inputting the rough result into a target size regressor, and using the output of the target size regressor as the accurate tracking result of the current frame, and updating each target classifier with the target features in the m frames closest to the current frame;

[0025] If not, updating the parameters of the target classifier with the target features in the n frames closest to the current frame, and then tracking the target in the next frame, where m > n.

[0026] Further, before generating the first candidate sample set centered on the predicted position of the target in the previous frame of the current frame to be tracked, the method further includes:

[0027] Obtaining the first frame of the visible light-thermal infrared video to be tracked, and generating a second candidate sample set and a target sample set through the ground truth box of the target in the first frame;

[0028] Training the tracking model and the target size regressor by using the second candidate sample set and the target sample set respectively.

[0029] Further, the method further includes:

[0030] Concatenate and flatten the features output by the first extraction network and the second extraction network in the channel dimension, and input them into the target classifier to obtain the target tracking result of the current frame.

[0031] Further, the method further includes:

[0032] Use the Imagenet-Vid dataset to pre-train the parameters of the first extraction network, the second extraction network, and the domain-shared fully connected layer as the pre-trained parameters in the network;

[0033] Use Gaussian distribution to initialize the parameters of the domain-specific fully connected layer and the weight assignment network;

[0034] Utilize the labeled and aligned visible-light thermal-infrared dataset to train the tracking model to obtain the trained tracking model.

[0035] Further, the training of the tracking model using the labeled and aligned visible-light thermal-infrared dataset includes:

[0036] Use max-preservation and random convolution kernel discarding to train each convolution kernel in the tracking model to combine the target classifier to obtain a pre-trained model in the first stage;

[0037] Load the pre-trained model in the first stage, and use the randomly constructed training data to train the network parameters of the pre-trained model in the first stage.

[0038] On the other hand, a visible-light infrared target tracking device based on a modal competition and cooperation network is adopted. The device includes:

[0039] A sample generation module, configured to generate a first candidate sample set centered on the target prediction position in the previous frame of the current frame to be tracked. The first candidate sample set includes visible-light samples and thermal-infrared samples;

[0040] A target tracking module, configured to input the first candidate sample set into the trained tracking model to obtain the target tracking result of the current frame;

[0041] Among them, the tracking model includes a feature extraction network and a target classifier, and the output of the feature extraction network is connected to the target classifier; the feature extraction network includes a first extraction network, a second extraction network, and a weight allocation network. The visible light sample and the thermal infrared sample are respectively used as the inputs of the first extraction network and the second extraction network. Both the first extraction network and the second extraction network include the first three convolutional layers of VGG-M, and each convolutional layer contains N convolutional kernels of the same size to form a convolutional kernel group. The first candidate sample set is used as the input of the weight allocation network. The first weight group output by the weight allocation network is connected to each convolutional kernel group in the first extraction network through a multiplication operation and then used as the input of the first extraction network. The second weight group output by the weight allocation network is connected to each convolutional kernel group in the second extraction network through a multiplication operation and then used as the input of the second extraction network.

[0042] The advantages of the present invention are as follows:

[0043] (1) In the present invention, target tracking is achieved by using a tracking model. The network structure of the tracking model includes a feature extraction network and a target classifier. The output of the feature extraction network is connected to the target classifier. The feature extraction network includes a first extraction network, a second extraction network, and a weight allocation network. Both the first extraction network and the second extraction network include the first three convolutional layers of VGG-M, and each convolutional layer contains N convolutional kernels of the same size to form a convolutional kernel group. The first weight group output by the weight allocation network is connected to each convolutional kernel group in the first extraction network through a multiplication operation and then used as the input of the second extraction network. The second weight group output by the weight allocation network is connected to each convolutional kernel group in the second extraction network through a multiplication operation and then used as the input of the second extraction network. By introducing convolutional kernel groups and multiple weight allocation networks to generate dynamically cooperative multi-core convolutional kernels, and using several convolutional kernels included in the convolutional kernel group, more diverse features can be extracted. The weight allocation network is used to allocate the most suitable weights for the convolutional kernels in the convolutional kernel group in the current modality and current environment. After fusion, the best convolutional kernel in the current modality can be obtained. After convolving this convolutional kernel with the input, the most suitable feature representation in the current modality can be obtained, thereby achieving a good multi-modal video tracking effect.

[0044] (2) The tracking result accuracy of the present invention is significantly better than other methods, and its tracking performance has been greatly improved.

[0045] Additional aspects and advantages of the present invention will be given in part in the following description, will become apparent in part from the following description, or will be understood through the practice of the present invention. Description of the Drawings

[0046] Figure 1It is the flowchart of the visible light and infrared target tracking method based on the modal competition and cooperation network in the first embodiment of the present invention;

[0047] Figure 2 It is the schematic diagram of the network model structure of the tracking model in the first embodiment of the present invention;

[0048] Figure 3 It is the schematic diagram of the structure of the weight assignment network in the first embodiment of the present invention;

[0049] Figure 4 It is the overall flowchart of the visible light and infrared target tracking method based on the modal competition and cooperation network in the first embodiment of the present invention;

[0050] Figure 5 It is the schematic diagram of the test result of the present invention tested on the publicly available dataset GTOT;

[0051] Figure 6 It is the schematic diagram of the test result of the present invention tested on the publicly available dataset RGBT234;

[0052] Figure 7 It is the schematic diagram of the structure of the visible light and infrared target tracking device based on the modal competition and cooperation network in the second embodiment of the present invention. Detailed implementation manners

[0053] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0054] As Figures 1 to 2 shown, the first embodiment of the present invention proposes a visible light and infrared target tracking method based on a modal competition and cooperation network. The method includes the following steps:

[0055] S10. Taking the predicted position of the target in the previous frame of the current frame to be tracked as the center, generate a first candidate sample set, where the first candidate sample set includes visible light samples and thermal infrared samples;

[0056] It should be noted that in this embodiment, Gaussian distribution sampling is performed centered on the target position of the previous frame to generate 256 candidate samples, which form the first candidate sample set.

[0057] S20. Input the first candidate sample set into the trained tracking model to obtain the target tracking result of the current frame;

[0058] Among them, the tracking model includes a feature extraction network and a target classifier, and the output of the feature extraction network is connected to the target classifier; the feature extraction network includes a first extraction network, a second extraction network, and a weight assignment network. The visible light sample and the thermal infrared sample are respectively used as the inputs of the first extraction network and the second extraction network. Both the first extraction network and the second extraction network include the first three convolutional layers of VGG-M, and each convolutional layer contains N convolutional kernels of the same size to form a convolutional kernel group. The first candidate sample set is used as the input of the weight assignment network. The first weight group output by the weight assignment network is connected to each convolutional kernel group in the first extraction network through a multiplication operation and then used as the input of the first extraction network. The second weight group output by the weight assignment network is connected to each convolutional kernel group in the second extraction network through a multiplication operation and then used as the input of the second extraction network.

[0059] Among them, the tracking model includes three major parts: a candidate box cropping module, a feature extraction module, and a target classification module. The candidate box cropping module is used to crop the sample picture into individual picture blocks as the input of the network. The feature extraction module is a stacked dynamic cooperation layer responsible for feature extraction of a specific modality. The target classifier module is responsible for identifying whether the candidate sample in the current frame is a target or a background.

[0060] It should be noted that in this embodiment, multiple convolutional kernels are used in each layer of the first extraction network and the second extraction network to model the features of the target. At the same time, the weight assignment network is used to assign weights to each convolutional kernel, and then all the convolutional kernels are fused into one kernel. The parameters of this kernel are dynamically changed for different inputs, which can better capture the features of the target. Its essence is to obtain the most suitable feature representation in the current modality through kernel fusion, so as to achieve a good multi-modal video tracking effect.

[0061] In some embodiments, step S20 includes the following sub-steps:

[0062] The visible light sample and the thermal infrared sample in the first candidate sample set are respectively input into the first extraction network and the second extraction network, and the samples in the first candidate sample set are input into the weight assignment network;

[0063] The weight assignment network respectively assigns a first weight group based on the visible light sample and a second weight group based on the thermal infrared sample;

[0064] The first weight group is multiplied by each convolutional kernel in the first extraction network to obtain a first dynamic cooperation convolutional kernel, and then convolved with the visible light sample to obtain visible light sample features;

[0065] Multiply the second weight combination with each convolutional kernel in the second extraction network to obtain a second dynamic collaborative convolutional kernel, and then perform convolution with the thermal infrared sample to obtain thermal infrared sample features;

[0066] Input the visible light sample features and the thermal infrared sample features into the target classifier to obtain the target tracking result of the current frame.

[0067] It should be noted that the weights in the first weight combination are the most suitable weights corresponding to each convolutional kernel in the convolutional kernel group in the visible light modality under the current environment; the weights in the second weight combination are the most suitable weights corresponding to each convolutional kernel in the convolutional kernel group in the thermal infrared modality under the current environment.

[0068] In one embodiment, as Figure 2 shown, a rectified linear unit and a local response normalization layer are connected after each convolutional kernel group in the first extraction network and the second extraction network.

[0069] It should be noted that after the modality-specific convolutional kernel group, there are a ReLU layer (rectified linear unit), an LRN layer (local response normalization), and a MaxPool layer (maximum pooling). The ReLU layer is to increase the non-linear fitting ability of the network, and the LRN layer is a normalization layer that can enable the network to converge quickly and increase the generalization ability of the network.

[0070] Particularly, a MaxPool layer (maximum pooling layer) is also set after the first two convolutional networks in the first extraction network and the second extraction network. The MaxPool layer is to make the feature map smaller, and the MaxPool layer is connected after the LRN layer.

[0071] In one embodiment, as Figure 3 shown, the weight allocation network includes a global average pooling layer, a modality-shared fully connected layer, and a modality-specific fully connected layer. The first candidate sample set is used as the input of the global average pooling layer. The output of the global average pooling layer is connected to the input of the modality-specific fully connected layer through the modality-shared fully connected layer. The modality-specific fully connected layer is used to output the first weight combination and the second weight combination.

[0072] It should be noted that the data of two modalities (visible light sample data and thermal infrared sample data) in the first candidate sample set are respectively input into the modality - shared fully - connected layer after global average pooling to map the multi - modal data to the same subspace to capture the global information of multi - modalities; the features after subspace mapping are respectively input into the modality - specific fully - connected layers to obtain the weights of each convolution kernel in the final modality - specific lower convolutional kernel group. In this embodiment, by setting the weight assignment network, all information of the two modalities is considered, and at the same time, the softmax function and the temperature coefficient are used to make the input more average, which is beneficial to the initial training.

[0073] It should be noted that in this embodiment, both the first extraction network and the second extraction network include 3 dynamic cooperation layers. Each modality dynamic cooperation layer consists of a convolution kernel group under a specific modality and a weight assignment network. Each convolution kernel group under a specific modality in each layer contains 4 convolution kernels, that is, 4 for the visible light branch and 4 for the thermal infrared branch. The size of the convolution kernels in the first layer is 96x7x7, the size of the convolution kernels in the second layer is 256x5x5, and the size of the convolution kernels in the third layer is 512x3x3. After each convolution kernel group, there are also an activation function (ReLU), a local response normalization layer (LRN), and a max - pooling layer (MaxPool) (except for the last layer). Taking the first layer as an example: visible light samples and thermal infrared samples are input into the multi - modal weight routing network to respectively obtain the weights of each convolution kernel in the visible light modality convolution kernel group and each convolution kernel in the thermal infrared modality convolution kernel group, and then the weights are multiplied by the convolution kernels in the corresponding modality convolution kernel group and fused into a dynamically cooperative convolution kernel. Using this convolution kernel to convolve with their respective inputs, visible light sample features and thermal infrared sample features can be obtained respectively. These features will be used as the input of the next layer for the same operation. Finally, the features output by the first extraction network and the second extraction network are used as the input of the target classifier.

[0074] In one embodiment, as Figure 2 shown, the target classifier includes two domain - shared fully - connected layers and K domain - specific fully - connected layers. After the two domain - shared fully - connected layers are connected, they are connected to the inputs of the K domain - specific fully - connected layers. The output of the domain - specific fully - connected layer is used as the target tracking result of the current frame;

[0075] The domain - shared fully - connected layer includes a fully - connected layer, a rectified linear unit, and a neuron stochastic activation function connected in sequence. The domain - specific fully - connected layer includes a fully - connected layer, a neuron stochastic activation function, and a Softmax layer connected in sequence.

[0076] It should be noted that the target classifier includes three fully connected layers. The first two are domain - shared fully connected layers, and the last one is a domain - specific fully connected layer. Among them, the domain - shared fully connected layer includes a fully connected layer, ReLU (Rectified Linear Unit), and Dropout (neuron random activation). The domain - specific fully connected layer only includes a fully connected layer, Dropout (neuron random activation), and a Softmax layer. The Softmax layer is used to normalize the input of the fully connected layer and calculate its score: and , is the score that the network predicts the sample to be the target, is the score that the network predicts the sample to be the background; finally, the sample corresponding to the maximum score is taken as the position of the target in the current frame.

[0077] It should be noted that the last layer of the target classifier is a domain - specific classifier. Each video corresponds to a domain, which can effectively solve the problem that the target may be the foreground or the background in different sequences.

[0078] It should be noted that the dimensions of the fully connected layers are 1024x3x3, 512x512, 512x2. The features of the sample will output two values for each sample after passing through the target classifier, which are the scores of it being the target and the background respectively. The higher the score, the greater the possibility that it is the target / background.

[0079] In one embodiment, the step S20 further includes the following steps:

[0080] Concatenate and flatten the features output by the first extraction network and the second extraction network in the channel dimension, and input them into the target classifier to obtain the target tracking result of the current frame.

[0081] In one embodiment, as Figure 4 shown, after the step S20, the following steps are further included:

[0082] S30: Take the target tracking result of the current frame as a rough result, and judge whether the tracking is successful based on the rough result. If yes, execute step S40; if not, execute step S50;

[0083] S40: Input the rough result into the target size regressor, and take the output of the target size regressor as the accurate tracking result of the current frame, and use the target features in the m frames closest to the current frame to update the target classifier;

[0084] S50: Use the target features in the n frames closest to the current frame to update the parameters of the target classifier, and then track the target in the next frame, where m > n.

[0085] It should be noted that according to this score, it is judged whether the current frame is successfully tracked. If successful, the target size regressor corrects the position of the target to obtain a more refined representation. If it fails, a short-term update strategy is adopted, and the target features of the last 30 frames are used to update the parameters of the target classifier. In addition, in order to capture the changes in the target appearance, a long-term update strategy is adopted every 10 frames, that is, the target features of the last 100 frames are used to update the parameters of the target classifier.

[0086] Specifically, in this embodiment, the sample with the largest target score among the candidate samples is selected as the prediction result of the current frame. If this score is greater than 0, it indicates that the tracking is successful. At this time, the target size regressor trained with the first frame is used to perform scale regression on this sample to obtain a more refined result. At the same time, 50 positive samples (IOU>0.7) and 200 negative samples (IOU<0.3) are collected in the current frame and added to the total positive and negative sample datasets. If the long-term update threshold (every 10 frames) is reached at this time, the target features of the last 100 frames are used to fine-tune the parameters of the target classifier so that the network can capture the changes in the target appearance. The network is trained 15 times using SGD at this stage, and the learning rate is 0.01. If the score of the prediction result is less than 0, it means that the tracking fails, and the prediction result is used as the final result. Then the short-term update strategy is used: the target features of the last 30 frames are used to fine-tune the parameters of the target classifier, and the network is also trained 15 times using SGD, and the learning rate is 0.01. After training, directly track the next frame.

[0087] It should be noted that in this embodiment, the target size regressor trained with the first frame is used to correct the target tracking position of the current frame, and finally a more refined tracking result is obtained.

[0088] It should be understood that after the tracking of the current frame is completed, directly enter the tracking of the next frame, and repeat steps S10 to S50 to complete the tracking until the last frame of the video.

[0089] In one embodiment, before step S10, the following steps are further included:

[0090] Obtain the first frame of the visible light-thermal infrared video to be tracked, and generate a second candidate sample set and a target sample set through the ground truth box of the target in the first frame;

[0091] Train the tracking model and the target size regressor respectively using the second candidate sample set and the target sample set.

[0092] It should be noted that based on the visible light-thermal infrared image of the first frame provided by the tracking video sequence and the corresponding ground truth bounding box of the target, 500 positive samples and 5000 negative samples were randomly generated according to the following strategy: 500 positive samples were obtained by sampling around the target using a Gaussian distribution at a certain threshold (IOU>0.7). 500 positive samples were obtained by sampling around the target and the entire image using a uniform distribution and global sampling at a certain threshold (IOU<=0.5). The positive and negative samples were combined to form the second candidate sample set.

[0093] The target classifier is trained using samples from the second candidate set in a batch size of 128. A difficult sample mining strategy is employed for negative samples, meaning only difficult samples are used to train the network. During this phase, the convolutional layer parameters are fixed, meaning they are not trained. Only the target classifier parameters are trained, with a learning rate of 0.005, for a total of 30 iterations.

[0094] Based on the ground-truth bounding box of the target in the first frame, 1000 samples (with IOU > 0.6) are randomly collected around the target and scaled to form a target sample set. These samples are then fed into the feature extractor to obtain the corresponding features. Ridge regression is then used to train the target scale regressor.

[0095] In one embodiment, before performing target tracking, the method further includes the following steps:

[0096] (1) using the Imagenet-Vid dataset to pre-train the parameters of the first extraction network, the second extraction network, and the domain-shared fully connected layer as pre-trained parameters in the network;

[0097] (2) Initializing the parameters of the domain-specific fully connected layer and the weight distribution network using Gaussian distribution;

[0098] In this embodiment, the parameters of the first three layers of the MDNet network pre-trained using the Imagenet-Vid dataset and the domain-shared fully connected layer are used as the pre-trained parameters in the network, and the parameters used for each convolutional kernel group in each layer are the same. Then, the parameters of the domain-specific layer and the multi-modal weight routing subnet are initialized using a Gaussian distribution. Among them, the first layer uses a 7x7 convolutional kernel, the second layer uses a 5x5 convolutional kernel, and the third layer uses a 3x3 convolutional kernel. The same number of convolutional kernels is used for each modality in each layer, which is set to 4 in the experiment, that is, the number of convolutional kernels in the convolutional kernel group is 4. An activation function ReLU is connected after each convolutional kernel group, and the first two layers also have an LRN (local response normalization layer) and a MaxPool (maximum pooling layer). The multi-modal weight routing subnet consists of a global average pooling layer, a modality-shared fully connected layer, and a modality-specific fully connected layer. A Softmax function is also connected after the modality-specific fully connected layer to obtain the weight of each convolutional kernel in the convolutional kernel group.

[0099] (3) Use the labeled and aligned visible light-thermal infrared dataset to train the tracking model to obtain the trained tracking model.

[0100] In one embodiment, the training process of the tracking model is divided into two stages:

[0101] (3-1) Use maximum retention and random convolutional kernel dropout to train each convolutional kernel combination in the tracking model and the target classifier to obtain the pre-trained model in the first stage;

[0102] The first stage: Use Maxout (maximum retention) and Kernel Dropout (random convolutional kernel dropout) to maximize the learning of convolutional kernels in the convolutional kernel group. Note: In this stage, the multi-modal weight routing subnet is not trained, only the modality-specific convolutional kernel group and the target classifier are trained. During training, a video domain is randomly selected, and 8 frames of pictures are selected as training data in each iteration. Then, 32 positive samples (IOU>0.7) and 96 negative samples (IOU<=0.5) are selected from them according to the threshold, where IOU is the intersection over union between the candidate sample and the ground truth box. After obtaining the positive and negative samples, we construct a domain-specific fully connected layer for the current video domain to discriminate whether the target belongs to the target or the background in the current domain. In each iteration of training, after the sample is input into the network, it is first convolved with each convolutional kernel in each convolutional kernel group of each layer to obtain several feature maps. Then, MaxOut and Dropout are used to fuse these feature maps so that the maximum response value on these feature maps can be retained and the non-maximum response values are suppressed. In this way, different convolutional kernels can focus on different features, thus avoiding the inability to play the role of multiple convolutional kernels due to feature redundancy. After passing through three convolutional layers, the features of the training samples can be obtained. These features are input into the domain-shared fully connected layer and the domain-specific fully connected layer corresponding to this domain, and the probability value of it being the target / background is obtained through softmax. Then, the BCE loss function is used to calculate its loss, and the SGD optimization method is used to optimize the entire network, with the number of optimization times being 200 epochs. In this stage, the learning rate of the convolutional layer is set to 0.0001, the learning rate of the fully connected layer is set to 0.0001, and the parameter of Dropout is 0.5. Save the final trained model as the pre-trained model for the next stage.

[0103] (3-2) Load the pre-trained model of the first stage and train the network parameters of the pre-trained model of the first stage using randomly constructed training data.

[0104] The second stage: Load the model parameters pre-trained in the first stage and train all the parameters of the entire network. Note: Maxout and Dropout are not used in this stage. During training, a video domain is randomly selected, and 8 frames of pictures are selected as training data in each iteration. Then, 32 positive samples (IOU > 0.7) and 96 negative samples (IOU <= 0.5) are selected from them according to the threshold, where IOU is the intersection over union between the candidate sample and the ground truth box. After obtaining the positive and negative samples, we construct a domain-specific fully connected layer for the current video domain to discriminate whether the target belongs to the target or the background in the current domain. In each iteration of training, after the sample is input into the network, the input of each layer will obtain the weights of each convolution kernel in the modality-specific convolution kernel group through the multi-modal weight routing subnet. Then, the parameters of the convolution kernel are fused according to the weights to obtain a dynamically cooperative convolution kernel. After convolving this convolution kernel with the corresponding modality input, a feature map is obtained as the input of the next layer. After passing through three convolutional layers, the features of the training samples can be obtained. These features are input into the domain-shared fully connected layer and the domain-specific fully connected layer corresponding to this domain, and the probability value of it being the target / background is obtained through softmax. Then, the BCE loss function is used to calculate its loss, and the SGD optimization method is used to optimize the entire network, with the number of optimization times being 200 epochs. In this stage, the learning rate of the convolutional layer is set to 0.0001, the learning rate of the fully connected layer is set to 0.0001, and the learning rate of the multi-modal weight routing subnet is set to 0.0005. The number of convolution kernels in the modality-specific convolution kernel group is set to 4. Save the finally obtained training model (excluding the parameters of the domain-specific fully connected layer) as the initial model for the tracking stage.

[0105] It should be noted that the model training in this embodiment is divided into two stages. The first stage is to fully train each convolution kernel so that they can extract rich and discriminative features. The second stage is to train the weight network to allocate appropriate weights to each convolution kernel in different scenarios.

[0106] This embodiment was tested on the publicly available datasets GTOT and RGBT234 respectively, and the test results (as shown respectively in Figures 5 to 6 shown) were evaluated with other trackers in terms of SR (success rate) and PR (precision). Among them, ours represents the tracking result accuracy of the solution of the embodiment of the present invention. It can be clearly seen that compared with other methods, its tracking performance has been improved to a large extent.

[0107] As Figure 7 shown, the second embodiment of the present invention proposes a visible light and infrared target tracking device based on a modality competition and cooperation network. The device includes:

[0108] The sample generation module 10 is configured to generate a first candidate sample set centered on the predicted position of the target in the previous frame of the current frame to be tracked. The first candidate sample set includes visible light samples and thermal infrared samples;

[0109] The target tracking module 20 is configured to input the first candidate sample set into a trained tracking model to obtain the target tracking result of the current frame;

[0110] Wherein, the tracking model includes a feature extraction network and a target classifier, and the output of the feature extraction network is connected to the target classifier; the feature extraction network includes a first extraction network, a second extraction network and a weight assignment network. The visible light samples and the thermal infrared samples are respectively used as the inputs of the first extraction network and the second extraction network. Both the first extraction network and the second extraction network include the first three convolutional layers of VGG-M, and each convolutional layer contains N convolutional kernels of the same size to form a convolutional kernel group; the first candidate sample set is used as the input of the weight assignment network. The first weight group output by the weight assignment network is connected to each convolutional kernel group in the first extraction network through a multiplication operation and then used as the input of the first extraction network. The second weight group output by the weight assignment network is connected to each convolutional kernel group in the second extraction network through a multiplication operation and then used as the input of the second extraction network.

[0111] In an embodiment, a rectified linear unit, a local response normalization layer and a max pooling layer are connected after each convolutional kernel group in the first extraction network and the second extraction network;

[0112] The weight assignment network includes a global average pooling layer, a modality-shared fully connected layer and a modality-specific fully connected layer. The first candidate sample set is used as the input of the global average pooling layer. The output of the global average pooling layer is connected to the input of the modality-specific fully connected layer through the modality-shared fully connected layer. The modality-specific fully connected layer is configured to output the first weight group and the second weight group;

[0113] The target classifier includes two domain-shared fully connected layers and a domain-specific fully connected layer. The two domain-shared fully connected layers are connected and then connected to the input of the domain-specific fully connected layer. The output of the domain-specific fully connected layer is used as the target tracking result of the current frame;

[0114] The domain-shared fully connected layer includes a fully connected layer, a rectified linear unit and a neuron stochastic activation function connected in sequence. The domain-specific fully connected layer includes a fully connected layer, a neuron stochastic activation function and a Softmax layer connected in sequence.

[0115] In an embodiment, the target tracking module 20 includes

[0116] An input unit, configured to respectively input the visible light samples and the thermal infrared samples in the first candidate sample set into the first extraction network and the second extraction network, and input the samples in the first candidate sample set into the weight assignment network;

[0117] A weight assignment unit, configured to enable the weight assignment network to respectively assign a first weight group based on the visible light samples and a second weight group based on the thermal infrared samples;

[0118] A collaborative convolution unit, configured to multiply the first weight group by each convolution kernel in the first extraction network to obtain a first dynamic collaborative convolution kernel, and then perform convolution with the visible light samples to obtain visible light sample features;

[0119] A feature acquisition unit, configured to multiply the second weight group by each convolution kernel in the second extraction network to obtain a second dynamic collaborative convolution kernel, and then perform convolution with the thermal infrared samples to obtain thermal infrared sample features;

[0120] A target tracking unit, configured to splice and flatten the visible light sample features and the thermal infrared sample features from the channel dimension, and input them into the target classifier to obtain the target tracking result of the current frame.

[0121] In one embodiment, the apparatus further includes:

[0122] A judgment module, configured to use the target tracking result of the current frame as a rough result, and judge whether the tracking is successful based on the rough result;

[0123] A refinement processing module, configured to, when the result output by the judgment module is yes, input the rough result into a target size regressor, and use the output of the target size regressor as the accurate tracking result of the current frame, and update the target classifier with the target features in the m frames closest to the current frame;

[0124] An update module, configured to, when the result output by the judgment module is no, update the parameters of the target classifier with the target features in the n frames closest to the current frame, and then track the target in the next frame, where m > n.

[0125] In one embodiment, the apparatus further includes:

[0126] An acquisition module, configured to acquire the first frame of the visible light-thermal infrared video being tracked, and generate a second candidate sample set and a target sample set through the ground truth box of the target in the first frame;

[0127] The initialization training module is used to train the tracking model and the target size regressor by using the second candidate sample set and the target sample set respectively.

[0128] In one embodiment, the device further includes:

[0129] The pre-training module is used to pre-train the parameters of the first extraction network, the second extraction network, and the domain-shared fully connected layer using the Imagenet-Vid dataset as the pre-training parameters in the network; initialize the parameters of the domain-specific fully connected layer and the weight assignment network using a Gaussian distribution;

[0130] The training module is used to train the tracking model using the labeled and aligned visible light-thermal infrared dataset to obtain the trained tracking model.

[0131] In one embodiment, the training module includes:

[0132] The first training unit is used to train each convolution kernel in the tracking model and combine the target classifier using maximum retention and random convolution kernel discarding to obtain a pre-trained model in the first stage;

[0133] The second training unit is used to load the pre-trained model in the first stage and train the network parameters of the pre-trained model in the first stage using randomly constructed training data.

[0134] It should be noted that other embodiments or implementation methods of the visible light and infrared target tracking device based on the modal competition and cooperation network of the present invention can refer to the above method embodiments, and will not be repeated here.

[0135] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a definite sequence list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, apparatus, or device (such as a computer-based system, a system including a processor, or other systems that can fetch and execute instructions from the instruction execution system, apparatus, or device), or in combination with these instruction execution systems, apparatuses, or devices. For the purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by or in combination with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer diskette (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other media, followed by editing, interpretation, or other suitable processing as necessary, and then stored in a computer memory.

[0136] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or combinations thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one or a combination of the following techniques well known in the art can be used: discrete logic circuits having logic gate circuits for implementing logical functions on data signals, application-specific integrated circuits having appropriate combinational logic gate circuits, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), and the like.

[0137] In the description of this specification, the description referring to terms such as "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials, or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0138] Furthermore, the terms "first" and "second" are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include at least one such feature. In the description of the present invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0139] Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those of ordinary skill in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

Claims

1. A visible light and infrared target tracking method based on a modal competition and cooperation network, characterized in that The method includes: Centering on the predicted position of the target in the previous frame of the current frame to be tracked, generating a first set of candidate samples, where the first set of candidate samples includes visible light samples and thermal infrared samples; Inputting the first set of candidate samples into a trained tracking model to obtain the target tracking result of the current frame; Among them, the tracking model includes a feature extraction network and a target classifier, and the output of the feature extraction network is connected to the target classifier; the feature extraction network includes a first extraction network, a second extraction network, and a weight allocation network. The visible light samples and the thermal infrared samples are respectively used as the inputs of the first extraction network and the second extraction network. Both the first extraction network and the second extraction network include the first three convolutional layers of VGG-M, and each convolutional layer contains N convolutional kernels of the same size to form a convolutional kernel group; the first set of candidate samples is used as the input of the weight allocation network. The first weight group output by the weight allocation network is connected to each convolutional kernel group in the first extraction network after a multiplication operation and used as the input of the first extraction network. The second weight group output by the weight allocation network is connected to each convolutional kernel group in the second extraction network after a multiplication operation and used as the input of the second extraction network; The weight allocation network includes a global average pooling layer, a modality-shared fully connected layer, and a modality-specific fully connected layer. The first set of candidate samples is used as the input of the global average pooling layer. The output of the global average pooling layer is connected to the input of the modality-specific fully connected layer through the modality-shared fully connected layer. The modality-specific fully connected layer is used to output the first weight group and the second weight group; The target classifier includes two domain-shared fully connected layers and a domain-specific fully connected layer. The two domain-shared fully connected layers are connected and then connected to the input of the domain-specific fully connected layer. The output of the domain-specific fully connected layer is used as the target tracking result of the current frame; The domain-shared fully connected layer includes a fully connected layer, a rectified linear unit, and a neuron stochastic activation function connected in sequence. The domain-specific fully connected layer includes a fully connected layer, a neuron stochastic activation function, and a Softmax layer connected in sequence; The method further includes: Pre-training the parameters of the first extraction network, the second extraction network, and the domain-shared fully connected layer using the Imagenet-Vid dataset as the pre-training parameters in the network; Initializing the parameters of the domain-specific fully connected layer and the weight allocation network using a Gaussian distribution; Training the tracking model using a labeled and aligned visible light-thermal infrared dataset to obtain the trained tracking model, including: Training each convolutional kernel combination in the tracking model and the target classifier using maximum retention and random convolutional kernel discarding to obtain a pre-training model in the first stage; Loading the pre-training model in the first stage and training the network parameters of the pre-training model in the first stage using randomly constructed training data.

2. The visible light and infrared target tracking method based on a modal competition and cooperation network according to claim 1, wherein After each of the convolution kernel groups in the first extraction network and the second extraction network, a rectified linear unit and a local response normalization layer are connected.

3. The visible light and infrared target tracking method based on the modal competition and cooperation network according to claim 1, wherein After inputting the first candidate sample set into the trained tracking model to obtain the target tracking result of the current frame, the method further includes: Regarding the target tracking result of the current frame as a rough result, and determining whether the tracking is successful based on the rough result; If so, inputting the rough result into a target size regressor, and regarding the output of the target size regressor as the accurate tracking result of the current frame, and using the target features in the m frames closest to the current frame to update each of the target classifiers; If not, using the target features in the n frames closest to the current frame to update the parameters of the target classifier, and then tracking the target in the next frame, where m > n.

4. The visible light and infrared target tracking method based on a modal competition and cooperation network according to claim 3, wherein Before generating the first candidate sample set centered on the target prediction position in the previous frame of the current frame to be tracked, the method further includes: Obtaining the first frame of the visible light-thermal infrared video to be tracked, and generating a second candidate sample set and a target sample set through the ground truth box of the target in the first frame; Respectively training the tracking model and the target size regressor by using the second candidate sample set and the target sample set.

5. The visible light and infrared target tracking method based on the modal competition and cooperation network according to claim 1, characterized in that The method further includes: Concatenating and flattening the features output by the first extraction network and the second extraction network in the channel dimension, and inputting them into the target classifier to obtain the target tracking result of the current frame.

6. A visible light and infrared target tracking device based on a modal competition and cooperation network, characterized in that, The device includes: A sample generation module, configured to generate a first candidate sample set centered on the target prediction position in the previous frame of the current frame to be tracked, where the first candidate sample set includes visible light samples and thermal infrared samples; A target tracking module, configured to input the first candidate sample set into the trained tracking model to obtain the target tracking result of the current frame; Wherein, the tracking model includes a feature extraction network and a target classifier, and the output of the feature extraction network is connected to the target classifier; the feature extraction network includes a first extraction network, a second extraction network and a weight allocation network, the visible light samples and the thermal infrared samples are respectively used as the inputs of the first extraction network and the second extraction network, both the first extraction network and the second extraction network include the first three convolutional layers of VGG-M, and each convolutional layer contains N convolutional kernels of the same size to form a convolution kernel group; the first candidate sample set is used as the input of the weight allocation network, and the first weight group output by the weight allocation network is connected to each of the convolution kernel groups in the first extraction network after a product operation and used as the input of the first extraction network, and the second weight group output by the weight allocation network is connected to each of the convolution kernel groups in the second extraction network after a product operation and used as the input of the second extraction network; The weight assignment network includes a global average pooling layer, a modality-shared fully connected layer, and a modality-specific fully connected layer. The first candidate sample set serves as the input of the global average pooling layer. The output of the global average pooling layer is connected to the input of the modality-specific fully connected layer via the modality-shared fully connected layer. The modality-specific fully connected layer is used to output the first weight set and the second weight set; The target classifier includes two domain-shared fully connected layers and a domain-specific fully connected layer. After being connected, the two domain-shared fully connected layers are connected to the input of the domain-specific fully connected layer. The output of the domain-specific fully connected layer serves as the target tracking result of the current frame; The domain-shared fully connected layer includes a fully connected layer, a rectified linear unit, and a neuron stochastic activation function connected in sequence. The domain-specific fully connected layer includes a fully connected layer, a neuron stochastic activation function, and a Softmax layer connected in sequence; The device further includes: a pre-training module, configured to pre-train the parameters of the first extraction network, the second extraction network, and the domain-shared fully connected layer using the Imagenet-Vid dataset as pre-training parameters in the network; initialize the parameters of the domain-specific fully connected layer and the weight assignment network using a Gaussian distribution; a training module, configured to train the tracking model using the labeled and aligned visible light-thermal infrared dataset to obtain the trained tracking model; The training module includes: a first training unit, configured to train each convolution kernel in the tracking model and the target classifier using maximum retention and random convolution kernel discarding to obtain a pre-trained model in the first stage; a second training unit, configured to load the pre-trained model in the first stage and train the network parameters of the pre-trained model in the first stage using randomly constructed training data.

7. The visible light and infrared target tracking device based on the modal competition and cooperation network according to claim 6, characterized in that After each convolution kernel group in the first extraction network and the second extraction network, a rectified linear unit and a local response normalization layer are connected.

8. The visible light and infrared target tracking device based on the modal competition and cooperation network according to claim 6, characterized in that, The device further includes: a judgment module, configured to use the target tracking result of the current frame as a rough result and determine whether the tracking is successful based on the rough result; a refinement processing module, configured to, when the output result of the judgment module is yes, input the rough result into a target size regressor, use the output of the target size regressor as the precise tracking result of the current frame, and update each target classifier using the target features in the m frames closest to the current frame; an update module, configured to, when the output result of the judgment module is no, update the parameters of the target classifier using the target features in the n frames closest to the current frame, and then track the target in the next frame, where m > n.

9. The visible light and infrared target tracking device based on the modal competition and cooperation network according to claim 8, characterized in that, The device further includes: an acquisition module, configured to acquire the first frame of the tracked visible light-thermal infrared video, and generate a second candidate sample set and a target sample set through the ground truth box of the target in the first frame; an initialization training module, configured to train the tracking model and the target size regressor using the second candidate sample set and the target sample set respectively.

10. The visible light and infrared target tracking device based on the modal competition and cooperation network according to claim 6, characterized in that, The features output by the first extraction network and the second extraction network are concatenated and flattened in the channel dimension and input into the target classifier to obtain the target tracking result of the current frame.

Images