A domain adaptation method for migrating a video model from a source domain to a target domain

Abstract

The present application belongs to the technical field of computer, and particularly relates to a domain adaptation method for migrating a video model from a source domain to a target domain. The present application comprises: connecting video samples from the source domain and samples from the target domain in a time dimension to generate mixed samples; a mixed sample domain discriminator discriminates the domain composition of the input samples, and enhances the ability of the backbone network to extract domain-independent features through adversarial learning; a mixed sample classifier classifies the input samples; in the training process, the loss function values of the mixed sample domain discriminator and the mixed sample classifier are used to dynamically measure the domain adaptation degree of different modalities in the video model, and the modality with a stronger domain adaptation degree is used as a teacher modality to perform knowledge distillation on the modalities with weaker domain adaptation degrees, so as to enhance the domain adaptation degree of the other modalities. The present application utilizes the characteristics of the time dimension information of the video model input samples, and significantly improves the domain adaptation performance of the video model.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, specifically relating to adaptive methods in the field of video models. Background Technology

[0002] In recent years, unsupervised domain adaptation (UDA) has attracted considerable research attention, aiming to transfer models trained on labeled source domain datasets to unlabeled target domains. UDA is valuable because it alleviates the need for extensive and costly data annotation for deep neural networks. Currently, UDA methods have made significant progress in image-based tasks (such as image classification, object detection, and semantic segmentation). However, UDA methods remain underexplored in video-based tasks. Unlike image data, video data contains multiple modalities (e.g., RGB and optical flow) and multiple dimensions (i.e., spatial and temporal dimensions), making domain adaptation in video more challenging.

[0003] Currently, most domain adaptation methods for video models are based on adversarial learning (AL). These methods add a domain discriminator after the backbone network for feature extraction to determine the domain from which the extracted features originate. The backbone network and the domain discriminator learn each other through adversarial training to extract more domain-independent features. AL-based methods are easy to implement, requiring only a few fully connected layers with gradient inversion layers, and have been widely used for image-based tasks. However, existing AL-based video model domain adaptation methods generally only match sample-level feature distributions, failing to fully utilize the temporal dimension of video data. Furthermore, methods using self-supervised learning (SSL) for cross-domain feature alignment have been extensively studied in recent years and have made significant progress. SSL-based methods rely on designing various self-supervised subtasks to enhance the robustness of feature extraction and perform feature alignment between different modalities, and then enhance the feature extraction and classification capabilities of the target domain by assigning pseudo-labels to the target domain data. However, the performance of SSL-based video model domain adaptation methods on the target domain heavily depends on the reliability of pseudo-label assignment.

[0004] Since video data contains multiple modalities, the correlations between different modalities can be leveraged to enhance the model's domain adaptation capabilities. Furthermore, some studies have found that the degree of domain drift between the source and target domains varies across different modalities, leading to differences in the domain adaptation levels of different modal sub-models. However, existing video model domain adaptation methods generally employ self-supervised auxiliary tasks to facilitate inter-modal interaction, treating different modalities equally. How to utilize the differences in domain adaptation levels among different modalities remains to be explored. Summary of the Invention

[0005] The purpose of this invention is to provide a domain adaptation method for transferring video models from a labeled source domain to an unlabeled target domain with a different distribution than the source domain, or simply a domain adaptation method for transferring video models from the source domain to the target domain, so as to effectively improve the performance of multimodal video models on unlabeled target domain data.

[0006] This invention combines adversarial learning with the multi-dimensional characteristics of video data and innovatively uses knowledge distillation to enhance multimodal domain adaptability. Unlike existing adversarial learning-based video model domain adaptation methods, this invention proposes using mixed samples for adversarial training. Samples from both the source and target domains are truncated and concatenated along the time dimension to form a mixed sample. This mixed sample, along with original samples from the labeled source and unlabeled target domains, serves as the model's training data. This invention extends existing domain adversarial neural networks by using mixed samples for adversarial learning to strengthen the model's domain adaptability. This invention also adds a classification task to determine whether a sample is a mixed sample. This enhances the model's feature extraction capabilities, and the loss value of the mixed sample classifier, along with the loss of the domain discriminator, can serve as a measure of the modality's domain adaptability. Furthermore, unlike existing multimodal domain adaptation methods, this invention innovatively uses knowledge distillation for intermodal interaction, and different modalities are no longer treated equally, better utilizing the differences in domain adaptability between different modalities. This invention proposes a dynamic modality distillation method. During training, the domain adaptability of each modality sub-model is dynamically measured. The modality with higher domain adaptability is selected as the teacher modality to perform knowledge distillation on other modalities with relatively weaker domain adaptability. The modalities guide and promote each other, ultimately enabling the multimodal model to learn more domain-independent feature extraction. Experiments verify that this method can effectively improve the performance of multimodal video models on unlabeled target domain data.

[0007] The domain adaptation method for transferring video models from the source domain to the target domain proposed in this invention is based on deep learning technology. The specific steps are as follows: (i) generation of mixed samples; (ii) construction of an adversarial mixed sample domain discriminator; (iii) construction of a mixed sample classifier; and (iv) dynamic modal distillation.

[0008] (a) Generation of pooled samples;

[0009] Video data contains an additional time dimension compared to image data, providing rich temporal information. To better utilize the time dimension of video data, this invention first proposes a method for generating hybrid samples from original samples.

[0010] First, the input video data is processed by image frame extraction, downsampling, and optical flow calculation to obtain original samples of video multimodal data of consistent size. The original samples from the source domain contain video category labels, while the original samples from the target domain do not contain video category labels and have different distributions from the source domain samples.

[0011] Then, each modality is proportionally truncated using an original sample from the source domain and an original sample from the target domain, and then concatenated along the time dimension to form a mixed sample. The size of the mixed sample is consistent with that of the original sample, and the mixed sample contains neighborhood information from both the source and target domains. Taking the RGB modality as an example, the original sample is an image frame obtained by extracting and downsampling the input video data, with dimensions of time × height × width × number of channels (t × h × w × c). The mixed sample is generated from an original sample from the source domain and an original sample from the target domain. r% and (1-r)% are truncated along the time dimension from the original sample from the source domain and the original sample from the target domain, respectively. The two truncated parts are then concatenated along the time dimension to obtain the mixed sample, with dimensions still being t × h × w × c. The neighborhood of this mixed sample is r% from the source domain and (1-r)% from the target domain.

[0012] (II) Construction of an adversarial mixed-sample domain discriminant;

[0013] The construction of the adversarial mixed-sample domain discriminator extends the domain adversarial neural network from image tasks to video tasks. It uses original samples from the source and target domains, as well as mixed samples generated from the original samples, for adversarial training on each submodality. The mixed-sample domain discriminator consists of a gradient inversion layer and a two-layer fully connected classifier. A mixed-sample domain discriminator D is added after the 3D convolutional feature extraction network F for each modality of the video model. For the m-th modality, the loss function of the domain discriminator includes two parts:

[0014] (1) Original sample neighborhood discrimination loss When the input is a raw sample from the source domain or the target domain, the domain discriminator determines which domain the sample comes from:

[0015]

[0016] Among them, y d It is a two-dimensional vector representing the domain label, when the input x is an original sample from the source domain x∈X. S At that time, y d =<1,0>, or when the input x is an original sample x∈X in the target domain. T At that time, y d =<0,1>, where σ is the softmax function; It is the original sample neighborhood discrimination loss corresponding to the neighborhood discriminator D, for the m-th modality;

[0017] (2) Mixed sample domain constitutes discriminative loss When input, it is a mixed sample generated from original samples from different domains. At that time, the domain discriminator determines the proportion of samples originating from the source domain and the target domain respectively:

[0018]

[0019] Where KL represents the KL divergence, Let r be the probability distribution of the source and target domains output by the domain discriminator D, where r is the proportion of the source domain in the mixed samples. A two-dimensional vector representing the labels of the mixed sample domain; It corresponds to a mixed sample The domain discriminator determines the proportion of samples originating from the source domain and the target domain.

[0020] Suppose the model has M modalities. The loss function of the mixed-sample neighborhood discriminator is the sum of the two-term losses for all modalities:

[0021]

[0022] (III) Construction of a hybrid sample classifier;

[0023] The hybrid sample classifier enhances the model's feature extraction capabilities, and its loss value, along with the loss of the domain discriminator, serves as a measure of the modality's domain adaptability. Each modality's hybrid sample classifier consists of two fully connected layers, appended to the 3D convolutional feature extraction network F of each modality in the video model. Its input includes original samples and hybrid samples, and its output is the classification result indicating whether a sample is an original sample or a hybrid sample.

[0024]

[0025] Among them, y x It is a two-dimensional vector representing the sample type label, when the input x is the original sample x∈{X} S ,X T When}, y d =<1,0>, or when the input is a mixed sample, y d =<0,1>;

[0026] Suppose the model has M modalities, and the loss function of the mixture classifier is the sum of the losses of all modalities:

[0027]

[0028] (iv) Dynamic modal distillation;

[0029] The dynamic modality distillation involves dynamically evaluating the domain adaptability of each sub-modality of the model during training, selecting the modality with stronger domain adaptability as the teacher modality to guide other modalities with weaker domain adaptability in learning domain-irrelevant knowledge. Specifically, the evaluation method involves summing the loss function values ​​of the mixed-sample domain discriminator and the mixed-sample classifier to obtain the teacher score.

[0030]

[0031] Teacher scores The larger the value, the stronger the domain adaptability of that mode;

[0032] Teacher scores for each modality were obtained. Next, select the current one. The modality with the largest value is taken as the teacher modality. Using the original samples as input data, knowledge distillation is performed on all other modalities, expressed as follows:

[0033]

[0034] Where t represents the teacher modality, Q(x) t P(x) represents the probability distribution obtained from teacher modal video classification. i ) represents the classification probability distribution of other student modalities, and KL represents the KL divergence.

[0035] In this invention, the basic video classification task consists of a 3D convolutional backbone network and an FCN video classifier C. The model has M modalities, and the loss function for the video classification task is:

[0036]

[0037] Where y is the video category label of the source domain, and σ is the softmax function.

[0038] The final loss function is:

[0039]

[0040] The specific operation flow of the domain adaptation method for video models proposed in this invention is as follows:

[0041] (1) First, the input video data is processed by image frame extraction and optical flow calculation, and then downsampled to obtain multimodal original samples of the same size. The downsampling method is to sample 16 frames at a fixed sampling frequency starting from a random position. The original samples from the source domain contain video category labels, while the original samples from the target domain do not contain video category labels and have a different distribution from the source domain samples.

[0042] (2) Using labeled source domain raw samples as input data, train a 3D convolutional backbone network and a video classifier for feature extraction, and compute... The backbone network and video classifier are first pre-trained for 3000 iterations on labeled source domains;

[0043] (3) Generate mixed samples using the original samples from the source domain and the target domain;

[0044] (4) Features are extracted from the original samples and the mixed samples using a 3D convolutional network. The predicted neighborhood composition is obtained by passing the neighborhood discriminator with a gradient inversion layer. For the original samples, the softmax function is used to calculate... KL divergence is used to calculate mixed samples. The domain discriminator is trained adversarially against the 3D convolutional backbone network; due to the gradient inversion layer, adversarial training can be achieved with only one backpropagation.

[0045] (5) Train the mixed sample classifier using the features of the original samples and the mixed samples, and calculate...

[0046] (6) Sum the three loss function values ​​obtained in steps (4) and (5) to obtain the teacher score for each modality. The stronger the domain adaptability of a modality, the more difficult it is for the domain discriminator and the mixed sample classifier to distinguish the domain composition of the samples, and the larger the teacher score value. Select the modality with the largest teacher score value as the teacher modality, perform knowledge distillation on other modalities, and calculate...

[0047] (7) Add up all the losses obtained in process (2)-(6) to get the final loss, backpropagate to calculate the gradient, and use the SGD (Standard Gradient Descent) optimization algorithm to update the network parameters;

[0048] (8) Repeat steps (1)-(7) until the network converges, and obtain a video model that can perform well in the target domain that is unlabeled and has a different distribution from the source domain.

[0049] In terms of method evaluation, the domain adaptation method of the video model proposed in this invention takes video classification as the basic task and uses the accuracy, which is common in classification tasks, as the evaluation index.

[0050] In summary, the innovation of this invention lies in:

[0051] 1. This invention proposes a hybrid sample formed by truncating and concatenating original samples from labeled source domains and unlabeled target domains along the time dimension. The hybrid sample contains domain information from both the source and target domains and can be used to enhance adversarial learning performance and assist in self-supervised learning tasks.

[0052] 2. This invention extends Domain Adversarial Networks from image tasks to video tasks, and proposes a new hybrid sample Domain Adversarial Network suitable for video deep learning models. Hybrid samples are used to enhance the adversarial strength, enabling deep models to learn more domain-independent feature representations.

[0053] 3. This invention proposes a novel dynamic modality extraction method. This method dynamically evaluates the domain adaptation of each modality during training, selects the modality with the strongest domain adaptation in the current iteration as the teacher modality, and guides other modalities to learn domain-independent knowledge through knowledge distillation. This invention is the first attempt to utilize knowledge distillation to solve the multimodal domain adaptation problem. Attached Figure Description

[0054] Figure 1 This is a schematic diagram of the hybrid sample generation method proposed in this invention. The two modules on the left of the diagram show the original samples from the source domain (green) and the target domain (blue), respectively. The original samples from the source domain contain labels, while the original samples from the target domain do not contain labels, and the distribution of the source and target domain samples is different (from different environments). The gray module on the right shows a hybrid sample formed by extracting 50% from each of the original samples from the source and target domains and concatenating them along the time dimension.

[0055] Figure 2 This is an overview diagram of the system of this invention. The three modules on the left show different types of input samples, and different colors are used in the diagram to indicate the flow of different types of input samples. Solid lines are used to mark the data flow of RGB modalities, and dashed lines are used to represent optical flow modalities. For each modality, all three types of input are used to train the adversarial mixed-sample domain discriminator and the mixed-sample classifier. Dynamic modality distillation as described in this invention is performed between modalities. In each iteration, a teacher score is calculated, which controls the direction of knowledge distillation so that the modality with higher domain adaptation guides other modalities. Raw samples from both the source and target domains can be used for dynamic modality distillation. Detailed Implementation

[0056] Step 1. Video Frame and Optical Flow Image Preprocessing. For the source video data, image frame extraction and optical flow calculation are performed first to obtain video data in two modalities: RGB and optical flow. For the video used in the training phase, one frame is sampled every four frames from a random position in the acquired video RGB and optical flow frames as input data for the network, with each sample consisting of t frames. For the video used in the testing phase, five positions are sampled from the acquired video as input data for the network, and the average of the prediction results of the five samples is taken as the final prediction result. The dimensions of each RGB and optical flow frame are time × height × width × number of channels (t × h × w × c), where t is 16, and h and w are 224.

[0057] Step 2. Pre-train the model using labeled data from the source domain. The video model uses the I3D video classification model as the backbone network, and is initially pre-trained for 3000 iterations using only source domain data. The pre-training uses the SGD (Standard Gradient Descent) optimization algorithm with a learning rate of 0.001.

[0058] Step 3. Generate a mixed sample. Randomly generate a source domain ratio r, extract r×t frames from the original source domain samples, and extract (1-r)×t frames from the original target domain samples. Connect the two extracted parts along the time dimension to obtain a mixed sample that is still t frames.

[0059] Step 4. Construct an adversarial mixed-sample neighborhood discriminator. The adversarial mixed-sample discriminator consists of a gradient inversion layer and a two-layer fully connected classifier. The input is a 1024-dimensional feature vector, the hidden layer dimension is 100, and the output is a 2-dimensional vector. A mixed-sample neighborhood discriminator is added after the 3D convolutional feature extraction network F for each modality of the video model. Features are extracted from the original samples and mixed samples through the 3D convolutional network, and then passed through the neighborhood discriminator with the gradient inversion layer to obtain a predicted neighborhood composition. For the original samples, the softmax function is used to calculate... KL divergence is used to calculate mixed samples. The domain discriminator is trained adversarially against a 3D convolutional backbone network. Thanks to the gradient inversion layer, adversarial training can be achieved with only one backpropagation.

[0060] Step 5. Construct a mixed-sample classifier. The mixed-sample classifier consists of two simple fully connected layers, with a 1024-dimensional feature vector as input, a 100-dimensional hidden layer, and a 2-dimensional vector as output. A mixed-sample classifier is added after the 3D convolutional feature extraction network F for each modality of the video model. The mixed-sample classifier is trained using features from the original samples and the mixed samples, and the calculation...

[0061] Step 6. Construct dynamic mode distillation. Sum the three loss function values ​​obtained in Steps 4 and 5 to obtain the teacher score for each mode. The stronger the mode's domain adaptability, the harder it is for the domain discriminator and mixed-sample classifier to distinguish the domain composition of the samples, resulting in a higher teacher score. Select the mode with the highest teacher score as the teacher mode and perform knowledge distillation on other modes, calculating... Knowledge distillation uses raw samples from both the source and target domains.

[0062] Step 7. Sum all the losses obtained in steps 2-6 to get the final loss, backpropagate to calculate the gradient, and use the SGD (Standard Gradient Descent) optimization algorithm to update the network parameters with the learning rate set to 0.0001.

[0063] Step 8. Repeat steps 1-7 for domain-adaptive training, iterating 16,000 times.

[0064] Step 9. Testing Phase. Using the test samples described in Step 1, samples are taken from 5 locations in the acquired video each time as input data for the network. The average of the prediction results of the 5 samples is taken as the final prediction result.

[0065] For the proposed domain adaptation method for video models, the classification accuracy (Top-1 Accuracy) is used as the evaluation metric to assess the model's performance in transferring between different domains on the EPIC-Kitchens dataset. Experimental results (Table 1) show that this invention can effectively improve the performance of multimodal video models on unlabeled target domain data.

[0066] Table 1. Quantitative Experiment Results (Top 1 Accuracy)

[0067] D1→D2 D2→D3 D3→D1 D2→D1 D2→D3 D3→D2 direct migration 42.0 46.5 44.3 42.5 46.5 56.3 Domain Adversarial Neural Networks 46.5 51.0 47.9 50.2 51.0 52.7 This invention 56.0 52.4 51.0 50.3 52.4 54.7 .

Claims

1. A domain adaptation method for transferring video models from a source domain to a target domain, based on deep learning technology, characterized in that: The specific steps include: (a) Generation of pooled samples; First, the input video data is processed by image frame extraction, downsampling, and optical flow calculation to obtain original samples of video multimodal data of the same size. The original samples from the source domain contain video category labels, while the original samples from the target domain do not contain video category labels and have different distributions from the source domain samples. Then, each modality is proportionally truncated from an original sample from the source domain and an original sample from the target domain, and then spliced ​​along the time dimension to form a mixed sample. The size of the mixed sample is consistent with that of the original sample. The mixed sample contains domain information from both the source and target domains. (II) Construction of an adversarial mixed-sample domain discriminant; The construction of the adversarial mixed-sample domain discriminator extends the domain adversarial neural network from image tasks to video tasks. It uses original samples from the source and target domains, as well as mixed samples generated from the original samples, for adversarial training on each submodality. The mixed-sample domain discriminator consists of a gradient inversion layer and a two-layer fully connected classifier. A mixed-sample domain discriminator D is added after the 3D convolutional feature extraction network F for each modality of the video model. For the m-th modality, the loss function of the domain discriminator includes two parts: (1) Original sample neighborhood discrimination loss When the input is a raw sample from the source domain or the target domain, the domain discriminator determines which domain the sample comes from: Among them, y d It is a two-dimensional vector representing the domain label, when the input x is an original sample from the source domain x∈X. S At that time, y d =<1,0>, or when the input x is an original sample x∈X in the target domain. T At that time, y d =<0,1>, where σ is the softmax function; It is the original sample neighborhood discrimination loss corresponding to the neighborhood discriminator D, for the m-th modality; (2) Mixed sample domain constitutes discriminative loss When input, it is a mixed sample generated from original samples from different domains. At that time, the domain discriminator determines the proportion of samples originating from the source domain and the target domain respectively: Where KL represents the KL divergence, Let r be the probability distribution of the source and target domains output by the domain discriminator D, where r is the proportion of the source domain in the mixed samples. A two-dimensional vector representing the labels of the mixed sample domain; It corresponds to a mixed sample The domain discriminator determines the proportion of samples originating from the source domain and the target domain. Suppose the model has M modalities. The loss function of the mixed-sample neighborhood discriminator is the sum of the two-term losses for all modalities: (III) Construction of a hybrid sample classifier; The mixed sample classifier for each modality consists of two fully connected layers, which are added after the 3D convolutional feature extraction network F for each modality of the video model. Its input includes the original samples and the mixed samples, and its output is the classification result of whether the sample is the original sample or the mixed sample. Among them, y x It is a two-dimensional vector representing the sample type label, when the input x is the original sample x∈{X} S ,X T }hour, y d =<1,0>, or when the input is a mixed sample, y d =<0,1>; Suppose the model has M modalities, and the loss function of the mixture classifier is the sum of the losses of all modalities: (iv) Dynamic modal distillation; The dynamic modality distillation involves dynamically evaluating the domain adaptability of each sub-modality of the model during training, selecting the modality with stronger domain adaptability as the teacher modality to guide other modalities with weaker domain adaptability in learning domain-irrelevant knowledge. The evaluation method involves summing the loss function values ​​of the mixed-sample domain discriminator and the mixed-sample classifier to obtain the teacher score. Teacher scores The larger the value, the stronger the domain adaptability of that mode; Teacher scores for each modality were obtained. Next, select the current one. The modality with the largest value is taken as the teacher modality. Using the original samples as input data, knowledge distillation is performed on all other modalities, expressed as follows: Where t represents the teacher modality, Q(x) t P(x) represents the probability distribution obtained from teacher modal video classification. i ) represents the classification probability distribution of other student modalities, and KL represents the KL divergence.

2. The domain adaptation method for transferring a video model from a source domain to a target domain according to claim 1, characterized in that, The basic video classification task consists of a 3D convolutional backbone network and an FCN video classifier C. Assume the model has M modalities, and the loss function for the video classification task is: Where y is the video category label of the source domain, and σ is the softmax function; The final loss function is:

3. The domain adaptation method for transferring a video model from a source domain to a target domain according to claim 1, characterized in that, In step (i), for the RGB mode, the original sample is an image frame obtained by extracting and downsampling the input video data, with dimensions of time × height × width × number of channels t × h × w × c; the mixed sample is generated by an original sample from the source domain and an original sample from the target domain. r% and (1-r)% are respectively truncated from the original sample from the source domain and the original sample from the target domain along the time dimension. The two truncated parts are connected along the time dimension to obtain the mixed sample, with dimensions still being t × h × w × c. The neighborhood of the mixed sample is composed of r% from the source domain and (1-r)% from the target domain.

4. The domain adaptation method for transferring a video model from a source domain to a target domain according to claim 1, 2, or 3, characterized in that, The specific operating procedure is as follows: (1) First, the input video data is processed by extracting image frames and calculating optical flow, and then downsampling is performed to obtain multimodal original samples of the same size. The downsampling method is to sample 16 frames at a fixed sampling frequency starting from a random position. The original samples from the source domain contain video category labels, while the original samples from the target domain do not contain video category labels and have a different distribution from the source domain samples. (2) Using labeled source domain raw samples as input data, train a 3D convolutional backbone network and a video classifier for feature extraction, and compute... The backbone network and video classifier are first pre-trained for 3000 iterations on labeled source domains; (3) Generate mixed samples using the original samples from the source domain and the target domain; (4) Features are extracted from the original samples and the mixed samples using a 3D convolutional network. The predicted neighborhood composition is obtained by passing the neighborhood discriminator with a gradient inversion layer. For the original samples, the softmax function is used to calculate... KL divergence is used to calculate mixed samples. The domain discriminator is trained adversarially against the 3D convolutional backbone network; due to the gradient inversion layer, adversarial training is achieved with only one backpropagation. (5) Train the mixed sample classifier using the features of the original samples and the mixed samples, and calculate... (6) Sum the three loss function values ​​obtained in processes (4) and (5) to obtain the teacher score for each modality. Select the modality with the highest teacher score as the teacher modality and perform knowledge distillation on other modalities. Calculate... (7) Add up all the losses obtained in process (2)-(6) to get the final loss, backpropagate to calculate the gradient, and use the SGD optimization algorithm to update the network parameters; (8) Repeat steps (1)-(7) until the network converges, and obtain a video model that can perform well in the target domain that is unlabeled and has a different distribution from the source domain.

Images