Cross-modal target tracking method and system based on multi-teacher joint guidance

Abstract

The application discloses a cross-modal target tracking method and system based on multi-teacher joint guidance, and the method comprises the following steps: training a first teacher network and a second teacher network by using RGB modal data samples and NIR modal data samples respectively, so as to obtain a first teacher feature learning network and a second teacher feature learning network; training a student network by using mixed modal data samples containing RGB modal data and NIR modal data, and guiding the student network by using the first teacher feature learning network or the second teacher feature learning network; and adopting a ternary tuple loss function to constrain the features of the student network, so as to obtain a tracker used for cross-modal target tracking. The application effectively solves the modal difference problem in the tracking process while reducing the calculation complexity.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, and particularly relates to a cross-modal target tracking method and system based on multi-teacher joint guidance. BACKGROUND

[0002] Target tracking is a basic task in the field of computer vision, aiming to locate any target in a video sequence. Due to its wide application prospects in video monitoring, intelligent robots and other fields, it has attracted more and more attention. Although impressive progress has been made in recent years, most trackers are based on visible light images, and due to the limitations of visible light imaging, the tracker is prone to tracking failure when encountering extreme lighting, rain, snow, fog and other challenges in the tracking process. In order to overcome the limitations of tracking based on visible light modal, to realize all-day and all-weather tracking, people have made a lot of efforts, and the multi-modal target tracking task has been put forward. By effectively mining and utilizing the complementary information of other modal (for example: thermal infrared, depth, language) and RGB modal, more robust tracking than simply based on RGB data is realized. Unfortunately, all existing multi-modal target tracking tasks need to perform representation alignment of different modal, which brings complexity to the task and increases the amount of calculation.

[0003] Near-infrared imaging has become a component of many surveillance cameras, and its imaging can switch between visible light and near-infrared according to the change of light. When the light is strong, it is visible light imaging; when the light is weak, it will automatically switch to near-infrared imaging, which well solves the imaging limitations of visible light sensors in poor light, and avoids the imaging and platform problems brought by existing multi-modal visual systems. However, while visible light modal and near-infrared modal are complementary, there are significant modal differences, which brings great challenges to cross-modal target tracking tasks. Therefore, how to handle modal differences has become the key to realizing robust and stable tracking.

[0004] In the related art, the document "Chenglong Li et al., Cross-Modal Object Tracking: Modality-Aware Representations and A Unifified Benchmark. Association for the Advancement of Artificial Intelligence. 2022" proposes a cross-modal object tracker MArMOT, which adopts a three-stage training method. First, the baseline tracker is fine-tuned using a cross-modal dataset, so that the tracker can achieve preliminary tracking of cross-modal data. Then, two parallel modality-aware branches are trained, so that the two parallel modality-aware branches can learn specific representations of objects in different modalities. Finally, the integration layer is trained to weight and fuse the features from the two modality-aware branches. By learning the modality-aware object representation, the modal difference between the RGB data and the NIR data in the tracking process is reduced. However, in the three-stage training process, the tracker may fall into a local optimum, thereby limiting the performance of the tracker, and at the same time, a large computational overhead is brought.

[0005] The patent application document with publication number CN115100235A proposes a target tracking method that uses a global and local knowledge distillation training method to guide training. However, this scheme solves the problem of thermal infrared target tracking, not cross-modal target tracking. It enhances the features obtained in the feature extraction stage by adding a self-attention-cross-attention feature enhancement module, a detail enhancement module, and a cross-attention enhancement module. Moreover, this scheme uses a larger-scale backbone network as the teacher network and adopts global-based knowledge distillation. SUMMARY

[0006] The technical problem to be solved by the present application is how to effectively solve the modal difference problem in the tracking process while reducing the computational complexity.

[0007] The present application solves the above technical problems by the following technical means:

[0008] On the one hand, the present application proposes a cross-modal object tracking method based on multi-teacher joint guidance, which comprises:

[0009] The first teacher network and the second teacher network are trained using RGB modality data samples and NIR modality data samples respectively to obtain the first teacher feature learning network and the second teacher feature learning network;

[0010] The student network is trained using a mixed modality data sample containing RGB modality data and NIR modality data, and the student network is guided using the first teacher feature learning network or the second teacher feature learning network.

[0011] The student network is trained using a mixed modality data sample containing RGB modality data and NIR modality data, and the student network is guided using the first teacher feature learning network or the second teacher feature learning network.

[0012] Further, the first teacher network, the second teacher network and the student network all adopt a convolutional neural network based on a residual network ResNet-50.

[0013] Further, the student network is trained using a mixed modality data sample containing RGB modality data and NIR modality data, and the student network is guided using the first teacher feature learning network or the second teacher feature learning network, comprising:

[0014] When the student network is trained using the RGB modality data in the mixed modality data sample, the student network is guided using the first teacher feature learning network.

[0015] When the student network is trained using the NIR modality data in the mixed modality data sample, the student network is guided using the second teacher feature learning network.

[0016] During the training of the student network, the parameters of the first teacher feature learning network and the second teacher feature learning network remain unchanged.

[0017] Further, when the student network is trained using a mixed modality data sample containing RGB modality data and NIR modality data, and the student network is guided using the first teacher feature learning network or the second teacher feature learning network, the following formula is used for constraint:

[0018]

[0019]

[0020] In the formula, L KD is the knowledge distillation loss; is the feature map extracted by the teacher feature learning network; is the feature map extracted by the student network; is the feature map extracted by the second teacher feature learning network; is the feature map extracted by the first teacher feature learning network; i is the row sequence number of the feature map; j is the column sequence number of the feature map; and n is the width of the feature map.

[0021] Further, the formula of the triplet loss function is represented as:

[0022] D S_R ||F S_R -F T_R || 2 -||F S_R -F T_N || 2

[0023] D S_N ||F S_N -F T_N || 2 -||F S_N -F T_R || 2

[0024] L DC = {max(D S_R + margin, 0), max(D S_N + margin, 0)}

[0025] In the formula, D S_R is the difference between the student network feature and the two teacher network features when the input is RGB data; D S_N is the difference between the student network feature and the two teacher network features when the input is NIR data; L dC is the distance constraint loss, and when the input is RGB data, the distance constraint loss value is max(D S_R + margin, 0), and when the input is NIR data, the distance constraint loss value is max(D S_N + margin, 0); F S_R is the feature extracted by the student network when the input is RGB data; F S_N is the feature extracted by the student network when the input is NIR data; F T_R is the feature extracted by the RGB teacher network when the input is RGB data; F T_N is the feature extracted by the NIR teacher network when the input is NIR data; || || 2 is the L2 loss; and margin is a constant.

[0026] In another aspect, the present application provides a cross-modal target tracking system based on multi-teacher joint guidance, which comprises:

[0027] The first training module is used to train the first teacher network and the second teacher network using RGB modal data samples and NIR modal data samples respectively, so as to obtain the first teacher feature learning network and the second teacher feature learning network.

[0028] The second training module is used to train the student network using mixed modal data samples containing RGB modal data and NIR modal data, and to guide the student network using either the first teacher feature learning network or the second teacher feature learning network.

[0029] The constraint module is used to constrain the features of the student network by using a triplet loss function, taking the features extracted by the student network as anchors, the features extracted by the teacher feature learning network of the same modality as positive samples, and the features extracted by the teacher feature learning network of different modalities as negative samples, so as to obtain a tracker for cross-modal target tracking.

[0030] Furthermore, the first teacher network, the second teacher network, and the student network all employ convolutional neural networks based on the ResNet-50 residual network.

[0031] Furthermore, the second training module is specifically used for:

[0032] When training the student network using RGB modal data from the mixed modal data samples, the first teacher feature learning network is used to guide the student network.

[0033] When training the student network using NIR modal data from the mixed modal data samples, the second teacher feature learning network is used to guide the student network.

[0034] During the training of the student network, the parameters of the first teacher feature learning network and the second teacher feature learning network remain unchanged.

[0035] Furthermore, the second training module employs the following formula for constraint during the training of the student network:

[0036]

[0037]

[0038] In the formula: L KD It is a loss due to knowledge distillation; These are feature maps extracted by the teacher feature learning network; These are feature maps extracted from the student network; It is a feature map extracted by the second teacher feature learning network; is the feature map extracted by the first teacher feature learning network; i is the row sequence number of the feature map; j is the column sequence number of the feature map; and n is the width of the feature map.

[0039] Further, the formula of the triplet loss function is represented as:

[0040] D S_R =||F S_R -F T_R || 2 -||F S_R -F T_N || 2

[0041] D S_N =||F S_N -F T_N || 2 -||F S_N -F T_R || 2

[0042] L DC = {max(D S_R +margin, 0), max(D S_N +margin, 0)}

[0043] In the formula, D s_R is the difference value of the distance between the student network feature and the two teacher network features when the input is RGB data; D S_N is the difference value of the distance between the student network feature and the two teacher network features when the input is NIR data; L DC is the distance constraint loss, the distance constraint loss value is max(D S_R +margin, 0) when the input is RGB data, and the distance constraint loss value is max(D S_N +margin, 0) when the input is NIR data; F S_R is the feature extracted by the student network when the input is RGB data; F S_N is the feature extracted by the student network when the input is NIR data; F T_R is the feature extracted by the RGB teacher network when the input is RGB data; F T_N is the feature extracted by the NIR teacher network when the input is NIR data; || || 2 is the L2 loss; and margin is a constant.

[0044] The present application has the following advantages:

[0045] (1) The application uses RGB modal data and NIR modal data to train two modal corresponding teacher feature learning networks, then uses mixed modal data and utilizes knowledge distillation loss to supervise the learning of the student network, finally in order to strengthen the feature alignment of the student network and the corresponding teacher network, a triplet loss function is adopted, the feature of the student network is further constrained and the feature space distance of the student network feature and the features of the two modal teacher networks is considered;The application gets rid of the previous complex network structure design, through the ingenious use of the cross-modal knowledge distillation method, the modal difference problem in the tracking process is effectively solved while reducing the computational complexity;In addition, the triplet loss function in the face recognition field is also used to ensure the further improvement of the tracking performance, so that when the sequence switches the mode in the tracking process, the target can still be accurately tracked.

[0046] Additional aspects and advantages of the application will be set forth in part in the description which follows, and in part will become apparent to those skilled in the art upon examination of the following description and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0047] Figure 1 is a flowchart of a cross-modal target tracking method based on multi-teacher joint guidance according to an embodiment of the application;

[0048] Figure 2 is a segmented training process diagram of a cross-modal target tracking model in the embodiment of the application;

[0049] Figure 3 is a triplet loss function constraint diagram in the embodiment of the application;

[0050] Figure 4 is the overall network framework of the cross-modal target tracking model based on knowledge distillation and triplet loss in the embodiment of the application;

[0051] Figure 5 is a cross-modal target tracking result diagram in the embodiment of the application;

[0052] Figure 6 is a comparison diagram of the experimental results of the method of the embodiment and related methods in the embodiment of the application;

[0053] Figure 7 is a structure diagram of a cross-modal target tracking system based on multi-teacher joint guidance according to an embodiment of the application. DETAILED DESCRIPTION

[0054] In order to make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0055] As shown in Figure 1 The first embodiment of the present application proposes a cross-modal target tracking method based on multi-teacher joint guidance, which comprises the following steps:

[0056] S10, training the first teacher network and the second teacher network using the RGB modal data sample and the NIR modal data sample respectively, to obtain the first teacher feature learning network and the second teacher feature learning network;

[0057] S20, training the student network using the mixed modal data sample containing the RGB modal data and the NIR modal data, and guiding the student network using the first teacher feature learning network or the second teacher feature learning network;

[0058] S30, using a triplet loss function, taking the feature extracted by the student network as an anchor, the feature extracted by the teacher feature learning network of the same modal as a positive sample, and the feature extracted by the teacher feature learning network of different modal as a negative sample, to constrain the feature of the student network, and obtaining a tracker for cross-modal target tracking.

[0059] Specifically, when knowledge distillation is performed, in the first stage, the teacher feature extraction network is first learned, and then the teacher feature extraction network is not updated when guiding the student network. In order to obtain a teacher feature extraction network that performs well on RGB data and NIR data respectively, first, the data is divided into two subsets according to the modal type, and then the corresponding student network is used to learn the parameters of the corresponding teacher backbone network. In the second stage, the student network is trained. When training the student network, the training set of the cross-modal data set is used, including RGB data and NIR data. During the training of the student network, the parameters of the teacher network remain unchanged, and according to the different input modal labels, the corresponding modal teacher is used to guide the student network, so that the student network masters the knowledge of tracking different modal images. Even without modal information in the test stage, effective tracking of different modal data can be realized, and the whole process is as follows: Figure 2In order to strengthen the alignment of the features of the student network and the corresponding teacher network, a triplet loss function is used to further constrain the features of the student network and consider the feature space distance between the features of the student network and the features of the two modal teacher networks.

[0060] In the feature extraction stage, the embodiment uses the knowledge distillation method to enhance the features without increasing the modules. The backbone network for tracking RGB and NIR is set as the teacher network, which has the same size as the student network. The knowledge distillation is performed only at the layer3 and layer4 layers of the backbone network for extracting features, that is, the local distillation method is used, and the network structure is small. In addition, the triplet loss is used to further constrain the features extracted by the tracker in the feature extraction stage, so as to further improve the tracking performance. In the cross-modal tracking task, there is a significant modal difference between the RGB modal data and the NIR modal data. Through experiments, it is proved that the embodiment does not reduce the difference between different modal data, but uses knowledge distillation to migrate the knowledge of the two teachers for tracking RGB modal data and NIR modal data to the tracker, so that the tracker learns to track two modal data, thereby improving the robustness of the tracker. The target tracking evaluation indexes SR, PR and NPR are all higher than those of other trackers with the same task, and better tracking effect is obtained.

[0061] In an embodiment, the first teacher network, the second teacher network and the student network all use a convolutional neural network based on a residual network ResNet-50.

[0062] It should be noted that the teacher network and the student network in the embodiment all use the same benchmark network without adding any modules to the benchmark network. The knowledge distillation and the triplet loss are used to improve the ability of the benchmark network to process different modal data features, and the parameter quantity and the calculation complexity of the network are reduced.

[0063] In an embodiment, the step S20 of training the student network using the mixed modal data samples containing the RGB modal data and the NIR modal data and guiding the student network using the first teacher feature learning network or the second teacher feature learning network includes the following steps.

[0064] S21, when training the student network using the RGB modal data in the mixed modal data samples, guiding the student network using the first teacher feature learning network;

[0065] S22, in the process of training the student network using the NIR modality data in the mixed modality data sample, the second teacher feature learning network is used to guide the student network;

[0066] S23, in the process of training the student network, the parameters of the first teacher feature learning network and the second teacher feature learning network remain unchanged.

[0067] In an embodiment, in knowledge distillation, the student network can master the same or even more knowledge as the teacher network through the transfer of knowledge. In cross-modal target tracking, knowledge distillation adopts a multi-teacher approach, designing teachers for RGB and NIR modalities to jointly guide the student network for tracking RGB and NIR data. At the same time, different teacher networks have been trained using corresponding modality data before training the student network, and have good ability to track each modality data. Therefore, when the student network learns from the teacher network, the more similar the features it obtains to the features extracted by the corresponding teacher network, the better, thereby improving the tracking ability of the student network. The entire distillation process is constrained by the following formula:

[0068]

[0069]

[0070] In the formula: L KD is the knowledge distillation loss; is the feature map extracted by the teacher feature learning network; is the feature map extracted by the student network; is the feature map extracted by the second teacher feature learning network; is the feature map extracted by the first teacher feature learning network; i is the row number of the feature map; j is the column number of the feature map; n is the width of the feature map.

[0071] In order to enable the tracker to obtain the ability to effectively track two modalities, the embodiment adopts the cross-modal knowledge distillation approach, and in the feature extraction stage, the knowledge of the two teacher networks of RGB and NIR modalities is transferred to the cross-modal target tracking student network.

[0072] Further, the student network has learned many knowledge of cross-modal object tracking through knowledge distillation. However, since the features of the student network are constrained with the features of the teacher network in the corresponding modality during distillation, the teacher in the other modality is not utilized. The embodiment further constrains the features: while improving the similarity of the features of the student network and the features of the teacher in the corresponding modality, the discriminability of the network to the modalities is improved to obtain more discriminative feature representation. Specifically, the distance between the features extracted by the student network and the features of the teacher network in the corresponding modality is less than the distance between the features and the teacher network in the different modality.

[0073] As shown in Figure 3 , the embodiment specifically uses a triplet loss in the field of face recognition, takes the features extracted by the student network as an anchor, the features extracted by the teacher network in the same modality as a positive sample, and the features extracted by the teacher network in the different modality as a negative sample, further constrains the features, and enhances the knowledge learned by the student network. The formula of the triplet loss function is as follows:

[0074] D S_R =||F S_R -F T_R || 2 -||F S_R -F T_N || 2

[0075] D S_N =||F S_N -F T_N || 2 -||F S_N -F T_R || 2

[0076] L DC = {max(D S_R +margin, 0), max(D S_N +margin, 0)}

[0077] In the formula, D S_R is the difference between the distance of the features of the student network and the distance of the features of the two teacher networks when the input is RGB data; D S_N is the difference between the distance of the features of the student network and the distance of the features of the two teacher networks when the input is NIR data; L DC is a distance constraint loss, the distance constraint loss value is max(D S_R +margin, 0) when the input is RGB data, and the distance constraint loss value is max(D S_N +margin, 0) when the input is NIR data; F S_Ris the feature extracted by the student network when the input is RGB data; F S_N is the feature extracted by the student network when the input is NIR data; F T_R is the feature extracted by the RGB teacher network when the input is RGB data; F T_N is the feature extracted by the NIR teacher network when the input is NIR data; F 2 is the L2 loss; margin is a constant, set to 0.01.

[0078] In order to learn more effective features, this embodiment introduces a triplet loss in the field of face recognition. When cross-modal knowledge distillation, if only the features extracted by the student network are constrained with the features of the corresponding modal teacher, the situation that the features extracted by the student network should be more similar to the features extracted by the corresponding modal teacher network and more different from the features extracted by the different modal teacher network is ignored. Therefore, using the triplet loss to pull the features extracted by the student network closer to the features extracted by the same teacher modal network while pulling the features extracted by the different teacher modal network further enhances the alignment of the features of the student network and the teacher network.

[0079] Further, the overall network framework of cross-modal target tracking based on knowledge distillation and triplet loss is as shown in Figure 4 , using the trained student network to guide the target tracking of the test frame.

[0080] In the field of computer vision tracking, visible light data and near-infrared data are selected as the two modalities of this embodiment. The tracking performance of the algorithm of the present application is evaluated by comparing it with some classic and most advanced trackers on a large-scale cross-modal target tracking dataset CMOTB to verify the effectiveness of the method proposed in the present application (see Figure 6 ). The success rate (SR), precision (PR) and normalized precision (NPR) in overall evaluation (OPE) are used as evaluation indexes for quantitative performance evaluation in this embodiment. The success rate is the percentage of successful tracking frames with an overlap greater than a threshold, and the success rate score is calculated by the area under the success rate curve. The precision is the percentage of all predicted frames less than the threshold value from the true value, which is set to 20 pixels in this embodiment. Considering that the accuracy is very sensitive to target size and resolution, the normalized precision (NPR) is obtained by normalizing the true value. From the results, it can be seen that the method proposed in this embodiment has obvious improvement in the field of computer vision tracking, and when the sequence switches modalities during tracking, it can still accurately track the target. The tracking effect is as shown in Figure 5 , Figure 5 , where the upper left (a) is the original RGB modal image, the upper right (b) is the original NIR modal image, the lower left (c) is the RGB modal tracking result, and the lower right (d) is the NIR modal tracking result.

[0081] In addition, such as Figure 7 As shown, another embodiment of the present invention proposes a cross-modal target tracking system based on multi-teacher joint guidance, the system comprising:

[0082] The first training module 10 is used to train the first teacher network and the second teacher network using RGB modal data samples and NIR modal data samples respectively, so as to obtain the first teacher feature learning network and the second teacher feature learning network.

[0083] The second training module 20 is used to train the student network using mixed modal data samples containing RGB modal data and NIR modal data, and to guide the student network using a first teacher feature learning network or a second teacher feature learning network.

[0084] The constraint module 30 is used to constrain the features of the student network by using a triplet loss function, taking the features extracted by the student network as anchors, the features extracted by the teacher feature learning network of the same modality as positive samples, and the features extracted by the teacher feature learning network of different modalities as negative samples, so as to obtain a tracker for cross-modal target tracking.

[0085] In one embodiment, the first teacher network, the second teacher network, and the student network all employ convolutional neural networks based on the ResNet-50 residual network.

[0086] In one embodiment, the second training module 20 is specifically used for:

[0087] When training the student network using RGB modal data from the mixed modal data samples, the first teacher feature learning network is used to guide the student network.

[0088] When training the student network using NIR modal data from the mixed modal data samples, the second teacher feature learning network is used to guide the student network.

[0089] During the training of the student network, the parameters of the first teacher feature learning network and the second teacher feature learning network remain unchanged.

[0090] In one embodiment, the second training module 20 uses the following formula to constrain the training of the student network:

[0091]

[0092]

[0093] In the formula: LKD is the knowledge distillation loss; is the feature map extracted by the teacher feature learning network; is the feature map extracted by the student network; is the feature map extracted by the second teacher feature learning network; is the feature map extracted by the first teacher feature learning network; i is the row index of the feature map; j is the column index of the feature map; n is the width of the feature map.

[0094] In an embodiment, the formula of the triplet loss function is represented as:

[0095] D S_R =||F S_R -F T_R || 2 -||F S_R -F T_N || 2

[0096] D S_N =||F S_N -F T_N || 2 -||F S_N -F T_R || 2

[0097] L DC = {max(D S_R +margin, 0), max(D S_N +margin, 0)}

[0098] In the formula: D S_R is the difference between the student network feature and the two teacher network features when the input is RGB data; D S_N is the difference between the student network feature and the two teacher network features when the input is NIR data; L DC is the distance constraint loss, when the input is RGB data, the distance constraint loss value is max(D S_R +margin, 0), when the input is NIR data, the distance constraint loss value is max(D S_N +margin, 0); F S_R is the feature extracted by the student network when the input is RGB data; F S_N is the feature extracted by the student network when the input is NIR data; F T_R is the feature extracted by the RGB teacher network when the input is RGB data; F T_N is the feature extracted by the NIR teacher network when the input is NIR data; || 2is the L2 loss; margin is a constant.

[0099] It should be noted that other embodiments of the cross-modal target tracking system based on multi-teacher joint guidance according to the present application or the implementation methods can refer to the above-mentioned method embodiments, which will not be repeated here.

[0100] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0101] In addition, the terms "first", "second" are only used for descriptive purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the present application, the meaning of "a plurality of" is at least two, for example, two, three, etc., unless otherwise specifically limited.

[0102] Although the embodiments of the present application have been shown and described above, it can be understood that the above-mentioned embodiments are exemplary and cannot be understood as limiting the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above-mentioned embodiments within the scope of the present application.

Claims

1. A cross-modal target tracking method based on multi-teacher collaborative guidance, characterized in that, The method includes: The first teacher network and the second teacher network were trained using RGB modal data samples and NIR modal data samples, respectively, to obtain the first teacher feature learning network and the second teacher feature learning network. The student network is trained using a mixed modal data sample containing RGB and NIR modal data, and guided by either a first teacher feature learning network or a second teacher feature learning network. Specifically, when training the student network using RGB modal data from the mixed modal data sample, the first teacher feature learning network guides the student network; when training the student network using NIR modal data from the mixed modal data sample, the second teacher feature learning network guides the student network. During the training of the student network, the parameters of the first and second teacher feature learning networks remain unchanged, and constraints are applied using the following formula: s.t. } In the formula: It is a loss due to knowledge distillation; These are feature maps extracted by the teacher feature learning network; These are feature maps extracted from the student network; It is a feature map extracted by the second teacher feature learning network; It is a feature map extracted by the first teacher feature learning network; It is the row number of the feature map; It is the column number of the feature map; It is the width of the feature map; By employing a triplet loss function, the features extracted by the student network are used as anchors, the features extracted by the teacher feature learning network of the same modality are used as positive samples, and the features extracted by the teacher feature learning network of different modalities are used as negative samples. The features of the student network are constrained to obtain a tracker for cross-modal target tracking.

2. The cross-modal target tracking method based on multi-teacher joint guidance as described in claim 1, characterized in that, The first teacher network, the second teacher network, and the student network all employ convolutional neural networks based on the ResNet-50 residual network.

3. The cross-modal target tracking method based on multi-teacher joint guidance as described in claim 1, characterized in that, The formula for the triplet loss function is as follows: In the formula: It is the difference between the distance between the student network features and the distance between the two teacher network features when the input is RGB data; It is the difference between the distance between the student network features and the distance between the two teacher network features when the input is NIR data; This is the distance constraint loss. When the input is RGB data, the distance constraint loss value is max(D). S_R +margin,0), when the input is NIR data, the distance constraint loss value is max(D S_N +margin,0); These are the features extracted by the student network when the input is RGB data; These are the features extracted by the student network when the input is NIR data; These are the features extracted by the RGB teacher network when the input is RGB data; These are the features extracted by the NIR teacher network when the input is NIR data; It is L2 loss; It is a constant.

4. A cross-modal target tracking system based on multi-teacher collaborative guidance, characterized in that, The system includes: The first training module is used to train the first teacher network and the second teacher network using RGB modal data samples and NIR modal data samples respectively, so as to obtain the first teacher feature learning network and the second teacher feature learning network. The second training module is used to train the student network using mixed modal data samples containing RGB modal data and NIR modal data, and to guide the student network using either the first teacher feature learning network or the second teacher feature learning network. The constraint module is used to constrain the features of the student network by using a triplet loss function, taking the features extracted by the student network as anchors, the features extracted by the teacher feature learning network of the same modality as positive samples, and the features extracted by the teacher feature learning network of different modalities as negative samples, so as to obtain a tracker for cross-modal target tracking. The second training module is specifically used for: When training the student network using RGB modal data from the mixed modal data samples, the first teacher feature learning network is used to guide the student network. When training the student network using NIR modal data from the mixed modal data samples, the second teacher feature learning network is used to guide the student network. During the training of the student network, the parameters of the first teacher feature learning network and the second teacher feature learning network remain unchanged; The second training module uses the following formula for constraint during the training of the student network: s.t. } In the formula: It is a loss due to knowledge distillation; These are feature maps extracted by the teacher feature learning network; These are feature maps extracted from the student network; It is a feature map extracted by the second teacher feature learning network; It is a feature map extracted by the first teacher feature learning network; It is the row number of the feature map; It is the column number of the feature map; It is the width of the feature map.

5. The cross-modal target tracking system based on multi-teacher joint guidance as described in claim 4, characterized in that, The first teacher network, the second teacher network, and the student network all employ convolutional neural networks based on the ResNet-50 residual network.

6. The cross-modal target tracking system based on multi-teacher joint guidance as described in claim 4, characterized in that, The formula for the triplet loss function is as follows: In the formula: It is the difference between the distance between the student network features and the distance between the two teacher network features when the input is RGB data; It is the difference between the distance between the student network features and the distance between the two teacher network features when the input is NIR data; This is the distance constraint loss. When the input is RGB data, the distance constraint loss value is max(D). S_R +margin,0), when the input is NIR data, the distance constraint loss value is max(D S_N +margin,0); These are the features extracted by the student network when the input is RGB data; These are the features extracted by the student network when the input is NIR data; These are the features extracted by the RGB teacher network when the input is RGB data; These are the features extracted by the NIR teacher network when the input is NIR data; It is L2 loss; It is a constant.

Images