System and method for cross-modal knowledge migration without task related source data

Abstract

A cross-modal knowledge migration system for adapting one or more source model networks to one or more target model networks is provided. The system is configured to perform the steps of: providing a TI pairing data set by a source feature encoder of the one or more source model networks; extracting TI source features and TI source moments from the TI pairing data through a BN layer of the one or more source model networks; providing a TI pairing dataset and an unlabeled TR dataset through the one or more target model networks to extract TI target features and TR target moments; jointly training all feature encoders and hybrid weights of the one or more target model networks by matching the extracted TI target features and TR target moments with the TI source features and TI source moments; and forming a final target model network by combining the trained one or more target model networks.

Description

Technical Field

[0001] The present invention relates to a system and method for cross-modal knowledge transfer in the absence of task-related source data. Background Art

[0002] Cross-modal knowledge distillation (CMKD) methods aim to learn rich representations for a modality without a large amount of labeled data from a large labeled dataset of another modality. These methods have been used for various practical computer vision tasks such as action recognition and image recognition. Most of the works in this area are based on the assumption of access to task-related paired data across modalities. A recent work relaxes this assumption in the context of domain generalization, where one does not have access to task-related paired data on the target domain but has access to task-related paired data in the source domain. For example, these methods consider unified database access (UDA) across domains (where the target domain has unlabeled RGB-D pairs) instead of a single modality. All of the above works either exploit task-related paired data for cross-modal knowledge transfer or treat cross-modal paired data as one domain. There are also some works in zero-shot domain adaptation, which exploit external task-independent paired data but require access to the source data. To address the storage or privacy issues regarding the source data, a new work direction called hypothesis transfer learning (HTL) has recently emerged, where only the trained source model can be accessed but not the source data. In this context, people have explored adapting target domain data that has limited or no labels at all in the presence of either a single source - passive domain adaptation (SFDA) - or multiple source models - multiple passive domain adaptation (MSFDA). These approaches do not work well in cases where the unlabeled target set is from a different modality than the source. A new cross-modal knowledge transfer approach is needed that allows for different source and target modalities and can perform effective knowledge transfer without access to the task-related data used to train the source model. Summary of the invention

[0003] The present disclosure relates to systems and methods for cross-modal knowledge transfer without task-related source data.

[0004] Some embodiments of the present invention describe that cost-effective depth and infrared sensors are now a reality as alternatives to conventional RGB sensors, and their advantages over RGB in areas such as autonomous navigation and remote sensing are clearly understood. Therefore, it is crucial to build computer vision and deep learning systems for depth and infrared data. However, large labeled datasets for these modalities are still lacking. In this case, it is valuable to transfer knowledge from neural networks trained on large, well-labeled datasets in the source modality (RGB) to neural networks working on the target modality (depth, infrared, etc.). For reasons such as memory and privacy, the source data may not be accessible, and knowledge transfer needs to be performed using only the source model. We describe an effective solution, SOCKET: Passive Cross-Modal Knowledge Transfer, for this challenging task of transferring knowledge from one source modality to a different target modality without access to the source data. The framework uses paired task-independent data and reduces the modality gap by matching the mean and variance of the target features with the batch normalization statistics present in the source model. We show through extensive experiments that our method significantly outperforms (up to 12% in some cases) existing passive methods for classification tasks that do not consider modality gaps.

[0005] According to some embodiments of the present invention, a cross-modal knowledge transfer system for adapting one or more source model networks to one or more target model networks is provided. The cross-modal knowledge transfer system may include: a memory configured to store: a task-independent (TI) paired dataset, an unlabeled task-related (TR) dataset, one or more source model networks including a batch normalization (BN) layer, a feature encoder, a convolutional neural network layer (CNN layer) and a classifier, one or more target model networks including a BN layer, a feature encoder, a CNN layer and a classifier, and a computer-implemented cross-modal knowledge transfer method with instructions; at least one processor, the at least one processor being configured to execute the steps of the computer-implemented cross-modal knowledge transfer method according to the instructions, wherein the steps include: transferring the TI source paired dataset from the one or more source model networks to the target model networks by passing the TI source paired dataset through the one or more source model networks. Multiple source model networks extract TI source features and TR source moments, wherein the CNN layers and classifiers of the one or more source model networks are frozen; batches of TI target features and TR target moments are extracted from the one or more target model networks by passing a TI paired dataset and an unlabeled TR dataset through the one or more target model networks, wherein the classifiers of the one or more target model networks are frozen; a modality-agnostic function is calculated based on the extracted TR target features of the one or more target model networks; the feature encoders of the one or more target model networks are jointly trained by minimizing the calculated modality-agnostic loss; and a final target model network is generated by combining the trained one or more target model networks.

[0006] In addition, some embodiments of the present invention provide a computer-implemented cross-modal knowledge transfer method with instructions using at least one processor and at least one memory. In this case, the instructions include extracting TI source features and TR source moments from the one or more source model networks by passing the TI source paired dataset through the one or more source model networks, wherein the CNN layers and classifiers of the one or more source model networks are frozen; extracting batches of TI target features and TR target moments from one or more target model networks by passing the TI paired dataset and the unlabeled TR dataset through the one or more target model networks, wherein the classifiers of the one or more target model networks are frozen; calculating modality-agnostic losses based on the extracted TR target features of the one or more target model networks; jointly training the feature encoders of the one or more target model networks by minimizing the calculated modality-agnostic losses; and generating a final target model network by combining the trained one or more target model networks.

[0007] The accompanying drawings, which are included in this specification to provide a further understanding of the invention, illustrate embodiments of the invention and together with the description serve to explain the principle of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] [ Figure 1 ]

[0009] Figure 1 A schematic diagram of a SOCKET method according to an embodiment of the present invention is shown;

[0010] [ Figure 2 ]

[0011] Figure 2 shows the difference between the prior art and the problem setting used in SOCKET;

[0012] [ Figure 3 ]

[0013] Figure 3 shows a detailed SOCKET process according to an embodiment of the present invention;

[0014] [ Figure 4 ]

[0015] Figure 4 A complete description of the SOCKET method in algorithmic form according to an embodiment of the present invention is shown;

[0016] [ Figure 5 ]

[0017] Figure 5 shows the main features of the data set used in the experiments according to the embodiments of the present invention;

[0018] [ Figure 6 ]

[0019] Figure 6 Results on the SUN RGB-D dataset for the single-source cross-modal knowledge transfer task from RGB to depth modality without access to task-related source data for all domain pairs are shown according to an embodiment of the present invention;

[0020] [ Figure 7 ]

[0021] Figure 7 The results on the SUN RGB-D dataset for the multi-source cross-modal knowledge transfer task from RGB to depth modality without accessing the task-related source data according to an embodiment of the present invention are shown;

[0022] [ Figure 8 ]

[0023] Figure 8shows the results on the DIML dataset when SOCKET uses different task-independent datasets compared to non-adapted sources and SHOT according to an embodiment of the present invention;

[0024] [ Fig. 9 ]

[0025] Fig. 9 Results on the RGB-NIR dataset for a single-source cross-modal knowledge transfer task from RGB to NIR or vice versa are shown according to an embodiment of the present invention, in the absence of task-related source data;

[0026] [ Fig.10 ]

[0027] Fig.10 The difference in cross-modal and cross-domain knowledge transfer results for SUN RGB-D scene classification using SHOT according to an embodiment of the present invention is shown;

[0028] [ Fig.11 ]

[0029] Fig.11 The contribution of the new loss component proposed by us according to the embodiment of the present invention to the results is shown;

[0030] [ Fig. 12A ]

[0031] Fig. 12A The effect of the amount of TI data on SOCKET results according to an embodiment of the present invention is shown;

[0032] [ Fig. 12B ]

[0033] Fig. 12B shows the influence of regularization hyperparameters on SOCKET results according to an embodiment of the present invention; and

[0034] [ Fig.13 ]

[0035] Fig.13 A cross-modal knowledge transfer system according to some embodiments of the present invention is shown. DETAILED DESCRIPTION

[0036] Various embodiments of the present invention are described below with reference to the accompanying drawings. It should be noted that the drawings are not drawn to scale, and elements of similar structure or function are represented by similar reference numerals throughout the drawings. It should also be noted that the drawings are intended only to facilitate the description of specific embodiments of the present invention. The drawings are not intended to be an exhaustive description of the present invention or to limit the scope of the present invention. In addition, the aspects described in conjunction with a specific embodiment of the present invention are not necessarily limited to that embodiment and may be practiced in any other embodiment of the present invention.

[0037] Figure 1 A schematic diagram of a passive cross-modal knowledge transfer (SOCKET) method according to some embodiments of the present invention is shown. We describe the single / multi-source cross-modal knowledge transfer problem without using the data used to train the source model. To effectively transfer knowledge, we enforce consistency of cross-modal features on task-independent paired data in feature space and minimize the modality gap by matching the distribution of unlabeled task-related features and source features.

[0038] Figure 2 Related work is shown and compared with SOCKET. In the accompanying figure, we compare our work with existing problem settings in the literature for knowledge transfer across different domains and modalities. The competing settings described in the figure are: (1) UDA (Unsupervised Domain Adaptation), DT (Domain Transformation), (2) MSDA (Multi-Source Domain Adaptation), (3) SFDA (Passive Single-Source DA), (4) MSFDA (Passive Multi-Source DA), (5) CMKD (Cross-Modal Knowledge Distillation), (6) ZDDA (Zero-Point DA). It should be noted that only the system according to the present invention allows cross-modal knowledge transfer from multiple sources without any access to the relevant source training data of the unlabeled target datasets for the different modalities.

[0039] Figure 3 A cross-modal knowledge transfer system 1100 ( Fig.13The source model 1-n (program modules 1-n) is formed by one or more neural networks (as shown in FIG. 1 ). For example, the source model 310 includes a convolutional neural network layer (CNN layer) 311 that outputs an intermediate feature map 312, a batch normalization (BN) layer 313 in a feature encoder 314, and a classification layer (classifier). The output of the feature encoder 315 is sent to a fully connected classifier layer 316. In some cases, the source model (corresponding to the source modality) can be referred to as a source model network. Our framework can be divided into two parts: (i) Before knowledge transfer (upper side): The cross-modal knowledge transfer system activates the source feature encoder and BN layer of the source model, but freezes the other parts of the source model including CNN and classifier, and passes the task-independent (TI) source data through the activated BN layer and source feature encoder to extract TI source features and source moments. The term "frozen" means that the parameters in the frozen layer remain fixed and are not updated. Since the task-related (TR) source feature map is not available, the cross-modal knowledge transfer system extracts the stored moments of its distribution from the BN layer. (ii) During knowledge transfer (lower side): The cross-modal knowledge transfer system only freezes the classification layer (classifier) ​​and feeds the TI and unlabeled TR target data through the target model (corresponding to the target modality) to obtain batch-wise TI target features 324 and TR target moments 322, respectively, both of which use the hybrid weight ζ k 's are combined from all models. In some cases, the target model can be called the target model network.

[0040] The cross-modal knowledge transfer system matches (combines) them with the pre-extracted source features 321 using the TI feature matching loss 320 and matches (combines) them with the source moments 317 using the distribution matching loss 318. Both the TI feature matching loss and the distribution matching loss are modality-specific losses. The TR target features 323 are used to calculate the modality-agnostic loss (modality-agnostic loss function) 319. The combination of the modality-specific loss function and the modality-agnostic loss function is minimized to jointly train all feature encoder parameters as well as the hybrid weights ζ k 's. The final target model is the optimal linear combination of the updated source models (corresponding to the trained target model network).

[0041] Depth sensors like Kinect and RealSense, LIDAR for direct measurement of point clouds, or high-resolution infrared sensors such as those from FLIR allow for an expanded range of computer vision applications compared to using only visible wavelengths. Directly sensing depth can provide an approximate 3D picture of the scene, improving the performance of applications such as autonomous driving navigation, while sensing in infrared wavelengths can allow easier detection of pedestrians or better detection of objects in adverse atmospheric conditions like rain, fog and smoke. These are just a few examples.

[0042] Building computer vision applications using current straightforward supervised deep learning methods for modalities such as depth and infrared requires large amounts of diverse labeled data. However, no such large and diverse datasets exist for these modalities, and the cost of building such datasets can be prohibitively high. In this context, researchers have developed methods such as knowledge distillation to transfer knowledge from models trained on modalities such as RGB, where large amounts of labeled data are available, to modalities of interest such as depth.

[0043] Compared to prior work, we address a novel and challenging problem in the context of cross-modal knowledge transfer. We assume that we only have access to (a) a source model trained for a task of interest (TOI), and (b) unlabeled data in the target modality that we need to build a model for the same TOI. The key aspect is that the cross-modal knowledge transfer system requires no access to any data in the source modality for the TOI. Such a problem setting is important in cases where storage and privacy considerations do not allow sharing of training data from the source modality; only the trained model can be shared.

[0044] Some embodiments provide SOCKET: Passive Cross-Modal Knowledge Transfer as an efficient solution to the problem of bridging the gap between source and target modalities. To this end, (1) we show that using an external dataset of source-target modality pairs that is independent of TOI - which we call task-independent (TI) data - can help learn an effective target model by bringing the features of the two modalities closer. In addition to using TI data, we encourage the statistics of the features of the unlabeled target data - which are by definition task-related (TR) - to match the statistics of the source data available to us from the normalization layers present in the trained source model.

[0045] We provide important empirical evidence that modality transfer from a source modality (such as RGB) to a target modality (such as depth) can be more challenging than domain transfer from one RGB dataset to another. This suggests that the proposed framework is necessary to help minimize the modality gap and thus make knowledge transfer more effective. Based on the above ideas, we show that we can improve existing state-of-the-art methods, which are only designed for cross-domain settings in the same modality. We summarize the main features of this disclosure as follows:

[0046] 1. We propose a new problem for knowledge transfer from a model trained on a source modality to a different target modality without any access to task-related source data and when the target data is unlabeled.

[0047] 2. To bridge the gap between modalities, we propose a new framework, SOCKET, for cross-modal knowledge transfer without access to the source data by: (a) using an external task-independent paired dataset, and (b) by matching the moments obtained by the normalization layers in the source model with the moments computed on the unlabeled target data.

[0048] 3. Extensive experiments on multiple datasets—for both knowledge transfer from RGB to depth and from RGB to IR, and for both single-source and multi-source cases—show that SOCKET is useful in reducing the modality gap in the feature space and produces significantly better performance (up to 12% in some cases) compared to existing passive domain adaptation baselines that do not consider the modality difference between the source and target modalities.

[0049] 4. We also empirically show that for datasets of interest, the knowledge transfer problem between modalities such as RGB and depth is more difficult than domain transfer within the same modality (such as sensor change and viewpoint change).

[0050] (Problem Setup and Notation)

[0051] We address the problem of cross-modal knowledge transfer for passive data by designing a specialized loss function that helps reduce the gap between the features of the source and target modalities. We focus on classification tasks where both the source and target data belong to the same N classes. Let us consider that we have n source models of the same modality (e.g., RGB). We denote the trained source classifier as Among them, S k represents the kth source model, m S represents the modality for which the source model is trained. The source model is represented as They are trained models that transform images from the source modality distribution Mapped to a probability distribution over classes. is the data for training the k-th source model, n k is the number of training data points corresponding to the kth source. In our problem setting, when transferring knowledge to the target modality, the source data is not available for all sources.

[0052] We can also access an unlabeled dataset in the target modality Among them, n T is the number of target samples. Note that the target modality m TDifferent from the source modality. Traditional passive UDA methods try to mitigate domain shift by adapting the source model to the unlabeled target data belonging to the same modality. As we will show, directly applying these methods to the cross-modal setting leads to poor performance. Therefore, we propose to use two new losses as regularization terms to address this issue, which can minimize the modality gap between the source and target modalities. Our goal is to learn a target classifier It can adapt well to target distributions obtained from different sensor modalities (e.g. depth or NIR).

[0053] For training We employ the following approaches: (a) methods that achieve learned feature embeddings for the target modality that closely match the embeddings of the source modality, which we group under a modality-specific loss (modality-specific loss function) as it bridges the gap between two different modalities; and (b) modality-agnostic loss terms that only operate on the unlabeled target data and do not consider variations in the modality.

[0054] We divide each source model into two blocks—feature encoder and classifier. For the kth source model, we denote these blocks as f k and g k .function Map the input image to an η-dimensional feature vector, These features are mapped to a probability distribution over N classes, where the maximum value is considered the classifier prediction. Thus, we can write in, is a combination of functions. Since the classifier layer g k contains information about the distribution of the unseen k-th source domain, we freeze all g k 's, and by k 's are optimized to train target-specific feature encoders.

[0055] (Cross-modal feature alignment)

[0056] Traditional passive UDA methods use domain-specific but modality-independent losses, which do not help reduce the feature distance between the source and target modalities. To better capture the cross-modal relationship between source and target, we propose a framework called SOCKET to reduce the modality gap. To train a target model with a reduced modality gap We propose to use task-independent feature matching and task-dependent distribution matching as described below.

[0057] (Task-independent feature matching)

[0058] Capturing the mapping between two different modalities effectively requires a large amount of paired data from both modalities. For our tasks of interest, we do not have a task-related (TR) dataset on the source side that includes data related to the task of interest (TOI). Therefore, it is not possible to match the target modality and the source modality by directly using data from task-related classes. In this case, we propose to use task-independent (TI) paired data from both modalities to reduce the modality gap. TI data refers to a dataset that contains samples only belonging to classes that are completely separated from the TR class and can come from any external dataset. For modalities like RGB-depth and RGB-IR, we have access to a large amount of paired TI data available in public datasets. We denote the paired TI data as in, is the ith TI data point from the source modality, is its paired counterpart from the target modality, n TI is the total number of pairs. We use TI data to calculate our proposed loss as follows

[0059] Step 1: The cross-modal knowledge transfer system feeds the source modality images of the TI dataset through each source model to pre-compute the modality m S We denote the i-th TI source feature extracted from source j as

[0060]

[0061] Step 2: In the knowledge transfer phase, the cross-modal knowledge transfer system is fed with the target modality images of the TI dataset that are stimulated to match the corresponding pre-extracted source modality features. The cross-modal knowledge transfer system is configured to define the following parameters in the feature encoder for the target modality: Minimize to achieve this:

[0062]

[0063] (Distribution matching related to the task)

[0064] In task-independent feature matching, we match TI features of two modalities in feature space. Even though this captures some class-independent cross-modal mapping between source and target modalities, it does not have information about the TR class-conditional cross-modal mapping. By this term, we refer to the cross-modal relationship between source and target given the related classes. Assuming that the marginal distribution of source features across batches can be modeled as a Gaussian distribution, the feature statistics can be fully characterized by its mean and variance. We propose to match feature statistics across source and target to further narrow the modality gap.

[0065] It may seem that a certain amount of source data is required to estimate the batch-wise mean and variance of its feature maps, but the running average statistics stored in the traditional batch normalization (BN) layer are sufficient for our purposes. The BN layer normalizes the feature maps during the training process to mitigate covariate shift. Therefore, it is able to capture channel-wise feature statistics cumulatively over all batches, yielding a rough estimate of the expected mean and variance of the batch-wise feature maps at the end of training. Let us consider the convolutional layer with the lth convolutional layer. The corresponding BN layer has r l nodes, and there are b such layers for each source model. We then refer to the expected batch-wise mean and variance of the lth convolutional layer of the kth source model as and

[0066] Before the knowledge transfer phase begins, we pre-extract information about source feature statistics from all pre-trained source models. In the knowledge transfer phase, for each iteration, we compute the batch-wise mean and variance of feature maps of target data from all source models, according to the weights ζ i We linearly combine them and minimize the distance between this weighted combination and the weighted combination of the pre-computed source feature statistics. We calculate the loss It is given by:

[0067]

[0068] in, and is the lth convolutional layer with source j (which we call ) is the running mean and variance of the batch normalization layer corresponding to , and as well as represents the mean and variance of the target output from the same batch normalization layer during the knowledge transfer phase. Loss and Minimize the modality gap between source and target. We name the combination of these two losses as modality-specific loss Among them, λ TI and λ d is a regularization hyperparameter.

[0069] (Overall Optimization)

[0070] The two methods proposed above help to reduce the modality gap between source and target without accessing task-related source data. In addition to them, we also directly use unlabeled target data for knowledge transfer. Specifically, we perform information maximization as well as minimization of the self-supervised pseudo-label loss.

[0071] Information Maximization (IM): IM essentially performs the task of maximizing the mutual information between the distribution of target data and its labels predicted by the source model. This mutual information is a combination of the conditional entropy and marginal entropy of the target label distribution.

[0072] In addition, we calculate the conditional entropy as follows and the marginal entropy of diversity

[0073]

[0074] in, ζ k is the weight assigned to the kth source such that ζ k ≥0, as well as is the empirical label distribution. The mutual information is calculated as The maximization of The minimization of ) ensures that the target labels as predicted by the sources are more confident and diverse in nature.

[0075] Pseudo label loss: Maximization helps obtain labels that are more predictively confident and globally diverse. However, it does not prevent mislabeling (i.e., assigning the wrong label to the input), which can lead to confirmation bias. To address this issue, we adopt a cross-entropy loss based on self-supervised pseudo-labels. After calculating the pseudo-labels, we calculate the pseudo-label cross-entropy loss as follows

[0076]

[0077] in, is the pseudo label of the i-th target data point, and 1{.} is an indicator function that gives a value of 1 when the argument is true. Our final loss is a combination of the above two losses. We call this combination a modality-agnostic loss It is expressed as

[0078] We compute the overall objective function as the sum of the modality-agnostic loss and the modality-specific loss and use Figure 4 The algorithm shown optimizes the weights in the feature encoder by minimizing the following objective function (also called loss function).

[0079] The accompanying drawing shows a complete description of the SOCKET method in algorithmic form according to an embodiment of the present invention.

[0080]

[0081] (experiment)

[0082] We first describe the datasets, baselines, and experimental details we use. Next, we present results for single-source and multi-source cross-modal transfer, which show the effectiveness of our approach. We also experimentally demonstrate that passive cross-modality is a more difficult problem compared to cross-domain knowledge transfer. We end the experiments by analyzing different hyperparameters.

[0083] (Datasets, baselines, and experimental details)

[0084] Datasets: To demonstrate the effectiveness of our approach, we conduct extensive testing on publicly available cross-modal datasets. We show results on two RGB-D (RGB and depth) datasets - SUN RGB-D and DIML RGB+D, and an RGB-NIR scene (RGB and near-infrared) dataset. Figure 5 Main features of the dataset used in the experiments according to embodiments of the present invention are shown, which summarizes the statistics of the dataset.

[0085] SUN RGB-D: A scene understanding benchmark dataset containing 10335 RGB-D image pairs of indoor scenes. The dataset has images acquired from four different sensors, namely Kinect version 1 (kv1), Kinect version 2 (kv2), Intel RealSense, and ASUS Xtion. We regard these four sensors as four different domains. All images are distributed in 45 classes, of which 17 classes are common to all domains. We refer to these common classes as TR classes and the remaining 28 classes as TI classes. To train four source models (one model for each domain), we use RGB images from TR classes that are specific to a particular domain. We regard the TR depth images of each domain as the target modality dataset. Here our goal is to classify TR scenes by adapting the RGB source model to the unlabeled target data of the depth modality.

[0086] DIML RGB+D: This publicly available dataset consists of more than 200 indoor / outdoor scenes. Instead of the full dataset, we use a smaller sample dataset with 1500 / 500 RGB-D pairs for training / testing distributed across 18 scene classes. We split the training pairs into RGB and depth, treating them as source and target, respectively. We Figure 5 These images are further divided into TR images and TI images. Synchronized RGB-D frames have been captured using Kinect v2 and Zed stereo cameras.

[0087] RGB-NIR Scenes: This publicly available dataset consists of 477 images from 9 scene categories captured in RGB and near-infrared (NIR). The images are captured using separate exposures from a modified SLR camera using visible light and NIR. We treat 6 of the categories as TR and the remaining 3 as TI to perform single-source knowledge transfer on this dataset. We conduct two experiments on this dataset: adaptation from RGB to NIR and vice versa.

[0088] (Baseline Method)

[0089] The problem statement we focus on in this paper is new and has not been considered in previous literature. Therefore, there is no direct baseline for our approach. However, the closest related work is the passive cross-domain knowledge transfer method that operates in both single-source and multi-source cases. SHOT and DECISION are the pioneering and most famous works for single-source and multi-source SFDA, respectively, and we compare these two methods.

[0090] Different from SOCKET, both baselines do not adopt strategies to overcome modality differences and only use modality agnostic losses. to train the target model. Using scene classification as the task of interest, we show that SOCKET outperforms these baselines for cross-modal knowledge transfer without access to task-related source data.

[0091] (Network Architecture)

[0092] In our experiments, we adopt the well-known Resnet50 model pre-trained on ImageNet as the backbone architecture for training the source model. Following this architecture, we replace the last fully connected (FC) layer with a bottleneck layer containing 256 units, in which we add a batch normalization (BN) layer at the end of the FC layer. Task-specific FC layers with weight normalization are added at the end of the bottleneck layer.

[0093] (Conduct knowledge transfer)

[0094] Recall that we initialize the target model with the source weights and the classifier layers are frozen. In the case of multiple sources, the weights in the feature encoder and the source mixing weight parameters (ζ k 's) is the optimization parameter. For all the following experiments, λ pl is set to 0.3. The regularization parameter λ for the modality-specific loss TI and λ d, we set them equal. We empirically choose these parameters to balance them with the modality agnostic loss so that no loss component overwhelms the other components by a large margin. Empirically, we find that the range of (0.1, 0.5) works best. All values ​​in this range outperform the baseline and we report the best accuracy among them. For images from modalities other than RGB, i.e., depth and NIR, we repeat the single-channel images into three-channel images so that they can be fed through a feature encoder initialized from a source model trained on RGB images. For all our experiments, we use a batch size of 32. We run our method 3 times for all experiments in PyTorch with 3 random seeds and report its average accuracy.

[0095] (Results on the SUN RGB-D dataset)

[0096] Our approach is general enough to handle any number of sources, and we demonstrate both single-source and multi-source knowledge transfer. Figure 6 Results on the SUN RGB-D dataset for the single-source cross-modal knowledge transfer task from RGB to depth modality are shown for all domain pairs without access to task-related source data, according to an embodiment of the present invention. In the figure, we show single-source RGB to depth results for all four domains. Considering the unlabeled depth data of each domain as the target, we adapt these data using source models trained on RGB data from each of the four domains. The rows represent the RGB domains on which the source models are trained. The columns represent the knowledge transfer results on the depth domain for the three methods: “Unadapted” shows the results for the unadapted source, SHOT, and SOCKET. From Figure 6 It is easy to see that for the target domains Kinect V1, Kinect V2, Realsense, and Xtion, SOCKET consistently outperforms the baseline by a good margin of 6.7%, 4.5%, 2.3%, and 3.8%, respectively, demonstrating the effectiveness of SOCKET in the passive cross-modal setting. In some cases, SOCKET outperforms the baseline by a very large margin, up to 12.4% (Realsense-RGB to Kinect V1-depth) or 9.0% (Xtion-RGB to Xtion depth).

[0097] Figure 7 Results on the SUN RGB-D dataset for the multi-source cross-modal knowledge transfer task from RGB to depth modality are shown without accessing task-related source data according to an embodiment of the present invention.

[0098] In this case, the figure shows the results of dual-source RGB to depth adaptation. For the four domains, we obtained six dual-source combinations, each for adapting to depth data from all four domains. The columns represent the knowledge transfer results on domain-specific depth data for DECISION and SOCKET. We also see that in this case, on average SOCKET outperforms the baseline by a good margin for all four target domains. Following the trend in single-source adaptation, SOCKET shows some very good improvements for some individual cases such as (Kinect v1+Xtion)-RGB to Kinect v1 depth-improvement of 12.2%-and (Kinect v2+Realsense)-RGB to Kinect v2 depth-improvement of 10.4%.

[0099] (Results on the DIML RGB+D dataset)

[0100] For this dataset, we Figure 5 The dataset was adjusted and single-source adaptation experiments were conducted. Figure 8 Results on the DIML dataset when SOCKET uses a different task-independent dataset compared to unadapted sources and SHOT according to some embodiments of the present invention are shown. Figure 8 In , we use TI data from both DIML RGB+D and SUNRGB-D datasets in two separate columns, where the TI data for SUN RGB+D is the same as that used in the experiments related to the SUN RGB-D dataset. By doing so, we show that SOCKET can perform well even with TI data from completely different datasets, and we find that SOCKET has relative gains of 4.7% and 11.9% over the baseline for these two TI data settings, respectively.

[0101] (Results on RGB-NIR scene dataset)

[0102] We now show that SOCKET also outperforms the baseline when the modalities are RGB and NIR using the RGB-NIR dataset. We follow Figure 5 We performed experiments on both RGB to NIR and vice versa. Fig. 9Results on the RGB-NIR dataset for the task of single-source cross-modal knowledge transfer from RGB to NIR and vice versa are shown in the absence of task-related source data according to some embodiments of the present invention. Columns represent knowledge transfer results on the depth domain for three methods: "Unadapted" shows the results with an unadapted source, SHOT, and SOCKET. For RGB to NIR migration, SOCKET shows a 3.5% improvement, while for NIR to RGB migration, it shows a 0.5% improvement over competing methods.

[0103] (cross-modality vs cross-domain)

[0104] To show the importance of the new problem we consider, we compare the single-source knowledge transfer results on the SUNRGB-D dataset with respect to both modality change and domain shift. Fig.10 The difference of cross-modal and cross-domain knowledge transfer results for SUN RGB-D scene classification using SHOT according to some embodiments of the present invention is shown. We use SHOT as a passive UDA method for this experiment. All domain-specific source models are trained on RGB images. For domain conversion, the target is all RGB images of the remaining 3 domains, and we report their average. Domain conversion involves changes in sensor configuration, viewpoint, etc. For modality change, the target data is depth images from the same domain. The scenes are the same as in the RGB source, except that they are captured using a depth sensor. The figure clearly shows that when we transfer knowledge across modalities instead of across domains of the same modality, the accuracy drops by a large amount of 12.5%. This shows that cross-modal knowledge transfer is different from DA and a framework like SOCKET is needed to reduce the modality for effective cross-modal knowledge transfer.

[0105] (Ablation and sensitivity analysis)

[0106] (Contribution of loss component)

[0107] Fig.11 Figure 2 shows the impact of our proposed novel loss on accuracy according to some embodiments of the present invention. The first accuracy column (a) corresponds to single-source adaptation from RGB to depth on the Kinect V2 domain, while the second column (b) shows the multi-source RGB to depth adaptation results from Kinect V1+Xtion to Kinect V1 domain on the SUN RGB-D dataset. The first row includes only the model with a modality-agnostic loss The second and third rows show the results of our proposed modality-specific loss and In both cases, SOCKET outperforms the baseline. Including the combination of our two losses The last row of will get the best result. We only show in parentheses the difference between the two methods compared to using only the mode-agnostic loss. The precision gain achieved.

[0108] (Influence of the number of TI images)

[0109] Fig. 12A The effect of the number of TI data on SOCKET results for the SUN RGB-D dataset according to some embodiments of the present invention is shown. We perform knowledge transfer from Kinect v1 RGB to unlabeled depth data. We use six random TI classes and vary the number of TI images from 0 to 60 in steps of 20 per class. The accompanying figure clearly shows that increasing the samples of TI data per class leads to an improvement in scene classification accuracy for RGB to depth transfer on the SUN RGB-D dataset. In short, for a fixed number of TI classes, the more TI images per class, the better SOCKET performs.

[0110] (Influence of Regularization Parameters)

[0111] Fig. 12B The effect of regularization hyperparameters on test accuracy for our new loss proposed as part of SOCKET according to some embodiments of the invention is shown. We perform Kinect v1 and Kinect v2 RGB to Kinect v1 depth migration on the SUN RGB-D dataset. For values ​​between 0 and 0, we keep λ TI and λ d are equal to each other. Using a value of 0 is the same as using SHOT. From the graph we see that as the value of the parameter increases, the accuracy also increases until a certain point, and then it starts to decrease.

[0112] We identify the new and challenging problem of cross-modal knowledge transfer without access to task-relevant data from the source modality. To effectively transfer knowledge to a target modality where we only have unlabeled data, some embodiments of the present invention may provide a framework, SOCKET, which includes loss functions designed to help bridge the gap between the two modalities in feature space. Experimental results on both RGB to depth and RGB to NIR experiments show that SOCKET outperforms baselines designed for passive unsupervised domain adaptation that perform poorly under modality transfer.

[0113] According to some embodiments of the present invention, a cross-modal knowledge transfer system and a computer-implemented cross-modal knowledge transfer method can reduce the size of the stored memory, solve privacy issues about the source data, and shorten the training time period of the target model network. Therefore, the cross-modal knowledge transfer system and computer-implemented cross-modal knowledge transfer method of the present invention can improve the function of the computer system (processor) and reduce the energy consumption of the computer system.

[0114] According to some features of the invention, each source model network includes a BN (batch normalization) layer and receives a set of unannotated / unlabeled datasets in the target modality to be classified. In some cases, an adapted target model (trained target network) is used to perform a computer vision task on the target modality data.

[0115] For a set of “task-independent” datasets, each data point is a pair of images with corresponding source and target modality images, which are used to help the knowledge transfer process by reducing the source-target modality gap.

[0116] In a cross-modal knowledge transfer system, statistics from one or more batch normalization layers are matched with batch-wise statistics of features of unlabeled target modality data to aid the knowledge transfer process by reducing the source-target modality gap.

[0117] Furthermore, an adapted target model can be obtained by using the source model as initialization and adjusting the parameters of the model by minimizing one or more loss functions. In this case, the combination of loss functions can include entropy, pseudo-labeling, and diversity defined on the output of a neural network given target unlabeled data as input.

[0118] In some cases, the combination of loss functions can include feature distances between source and target modality images from task-independent data, which helps reduce the modality gap between source and target features.

[0119] Additionally, the combination of loss functions can include the difference between the feature statistics of the unlabeled target dataset and the source feature statistics obtained from the batch normalization layer of the source model.

[0120] In some cases, the source-target modality pairs may be RGB-depth, RGB-infrared, RGB-LIDAR point clouds, or vice versa, or other combinations of these modalities. The dataset may be in the form of images captured in a single snapshot or in the form of videos captured over a longer duration. The tasks to be performed on the target unlabeled dataset may be computer vision tasks such as image recognition, object recognition, and scene recognition.

[0121] Fig.13A cross-modal knowledge transfer system 1100 according to some embodiments of the present invention is shown. System 1100 may be a neural network module trained to provide device layout. Cross-modal knowledge transfer system 1100 includes a TI / TR interface circuit interface 150, at least one processor 120, storage 130, and memory 140. Storage 130 and memory 140 may be integrated on a circuit and may be referred to as a memory. Storage 130 includes a task-independent paired data set 131, an unlabeled target data set 132, a source model network (source model) 133, a target model network (target model) 134, and a cross-modal knowledge transfer program (computer-implemented cross-modal knowledge transfer method) 135. Interface 150 is configured to communicate between memory 140, storage 130, and at least one processor 120. In some cases, the interface 150 can receive the task-independent paired dataset 131, the unlabeled target dataset 132, the source model network (source model) 133, and the target model network (target model) 134 from a database device (server) 195 outside the system 100 through a communication network 190 including a wireless communication network, a wired network, the Internet, or a combination thereof.

[0122] The above-mentioned embodiments of the present invention can be implemented in any of a variety of ways. For example, hardware, software or a combination thereof can be used to implement the embodiments. When implemented in software, the software code can be executed on any suitable processor or processor set (whether provided in a single computer or distributed between multiple computers). Such a processor can be implemented as an integrated circuit with one or more processors in an integrated circuit assembly. However, the processor can be implemented using a circuit of any appropriate format.

[0123] In addition, embodiments of the present invention may be implemented as methods for which examples have been provided. The actions performed as part of the method may be ordered in any suitable manner. Thus, embodiments may be constructed in which the actions are performed in a sequence different from that shown, which may include performing some actions simultaneously, although shown as sequential actions in illustrative embodiments.

[0124] Furthermore, the use of ordinal terms such as "first", "second" and the like to modify claim elements in the claims does not itself imply any priority, precedence or sequence of one claim element relative to another claim element or the temporal order of the actions of performing the method, but merely serves as a label to distinguish one claim element with a certain name from another element with the same name (except for the use of ordinal terms) to distinguish the claim elements.

[0125] Although the present invention has been described by way of examples of preferred embodiments, it is to be understood that various other adaptations and modifications can be made within the spirit and scope of the invention.

[0126] Therefore, it is the object of the appended claims to cover all such changes and modifications as come within the true spirit and scope of the invention.

Claims

1. A cross-modal knowledge transfer system, the cross-modal knowledge transfer system is used to adapt one or more source model networks to one or more target model networks, the cross-modal knowledge transfer system comprising: A memory configured to store: a task-independent TI paired dataset, an unlabeled task-dependent TR dataset, one or more source model networks including a batch normalization BN layer, a feature encoder, a convolutional neural network layer CNN layer, and a classifier, one or more target model networks including a BN layer, a feature encoder, a CNN layer, and a classifier, and a computer-implemented cross-modal knowledge transfer method with instructions; At least one processor, the at least one processor being configured to execute the steps of the computer-implemented cross-modal knowledge transfer method according to the instructions, wherein the steps include: extracting TI source features and TR source moments from the one or more source model networks by passing the TI source pairing dataset through the one or more source model networks, wherein the CNN layers and the classifiers of the one or more source model networks are frozen; extracting batches of TI target features and TR target moments from the one or more target model networks by passing the TI paired dataset and the unlabeled TR dataset through the one or more target model networks, wherein the classifiers of the one or more target model networks are frozen; Calculating a modality-agnostic loss function based on the extracted TR target features of the one or more target model networks; Calculating a modality-specific loss function by calculating the distances between the extracted TI target features and the TR target moments and the TI source features and the TI source moments, respectively; jointly training the feature encoders and mixing weights of the one or more target model networks by minimizing the calculated modality-agnostic loss function and the modality-specific loss function; and A final target model network is generated by combining the one or more trained target model networks.

2. The cross-modal knowledge transfer system according to claim 1, further comprising a TI / TR dataset interface, wherein the TI / TR dataset interface is configured to receive the TI paired dataset and the unlabeled TR dataset through a communication network, wherein: The TI / TR dataset interface is configured to store the received TI paired dataset and the unlabeled TR dataset in the memory.

3. The cross-modal knowledge transfer system according to claim 1, wherein: The at least one processor also executes sending the generated model parameters of the final target model network to one or more other untrained target model networks through the network.

4. The cross-modal knowledge transfer system according to claim 1, wherein: The pairs of the task-independent TI dataset and the unlabeled task-related TR dataset are RGB images and depth images, RGB images and infrared images, or RGB images and LIDAR point clouds.

5. The cross-modal knowledge transfer system according to claim 4, wherein: The unlabeled task-related TR dataset is used for computer vision tasks including image recognition, object recognition, and scene recognition.

6. The cross-modal knowledge transfer system according to claim 1, wherein: The TI source features are extracted from the feature encoder of the one or more source model networks, wherein the TR source moments are extracted from the BN layer of the one or more source model networks, wherein the batched TI target features are extracted from the feature encoder of the one or more target model networks, wherein the TR target moments are extracted from the BN layer of the one or more target model networks.

7. The cross-modal knowledge transfer system according to claim 1, wherein: The batch-wise TI target features and the TR target moments extracted from the one or more target model networks are respectively combined using hybrid weights.

8. The cross-modal knowledge transfer system according to claim 1, wherein: The modality-agnostic loss function and the modality-specific loss function are used to bridge the gap between the one or more source model networks and the one or more target model networks in the feature space.

9. The cross-modal knowledge transfer system according to claim 8, wherein: The combination of modality-agnostic loss functions includes entropy, pseudo-labeling, and diversity, wherein the entropy, the pseudo-labeling, and the diversity are defined on the outputs of the one or more source model networks.

10. The cross-modal knowledge transfer system according to claim 8, wherein: The combination of the modality-specific loss functions includes the extracted feature distance between the TI source features and the TI target features and the distance between the TI source moments and the TR target moments.

11. A computer-implemented cross-modal knowledge transfer method, the computer-implemented cross-modal knowledge transfer method having instructions using at least one processor and at least one memory, wherein: The instructions include: extracting TI source features and TR source moments from one or more source model networks by passing the TI source pairing dataset through the one or more source model networks, wherein the CNN layers and classifiers of the one or more source model networks are frozen; extracting batches of TI target features and TR target moments from one or more target model networks by passing the TI paired dataset and the unlabeled TR dataset through the one or more target model networks, wherein classifiers of the one or more target model networks are frozen; Calculating a modality-agnostic loss function based on the extracted TR target features of the one or more target model networks; Calculating a modality-specific loss function by calculating the distances between the extracted TI target features and the TR target moments and the TI source features and the TI source moments, respectively; jointly training the feature encoders and mixing weights of the one or more target model networks by minimizing the calculated modality-agnostic loss function and the modality-specific loss function; and A final target model network is generated by combining the one or more trained target model networks.

12. The computer-implemented cross-modal knowledge transfer method according to claim 11, wherein the computer-implemented cross-modal knowledge transfer method further uses a TI / TR dataset interface to receive the TI paired dataset and the unlabeled TR dataset through a communication network, wherein: The TI / TR dataset interface is configured to store the received TI paired dataset and the unlabeled TR dataset into the at least one memory.

13. The computer-implemented cross-modal knowledge transfer method according to claim 11, wherein: The at least one processor also executes sending the generated model parameters of the final target model network to one or more other untrained target model networks through the network.

14. The computer-implemented cross-modal knowledge transfer method according to claim 11, wherein: The pairs of the task-independent TI dataset and the unlabeled task-related TR dataset are RGB images and depth images, RGB images and infrared images, or RGB images and LIDAR point clouds.

15. The computer-implemented cross-modal knowledge transfer method according to claim 14, wherein: The unlabeled task-related TR dataset is used for computer vision tasks including image recognition, object recognition, and scene recognition.

16. The computer-implemented cross-modal knowledge transfer method of claim 11, wherein: The TI source features are extracted from the feature encoder of the one or more source model networks, wherein the TR source moments are extracted from the BN layers of the one or more source model networks, wherein the batched TI target features are extracted from the feature encoder of the one or more target model networks, wherein the TR target moments are extracted from the BN layers of the one or more target model networks.

17. The computer-implemented cross-modal knowledge transfer method of claim 11, wherein: The batch-wise TI target features and the TR target moments extracted from the one or more target model networks are respectively combined using hybrid weights.

18. The computer-implemented cross-modal knowledge transfer method of claim 11, wherein: The modality-agnostic loss function and the modality-specific loss function are used to bridge the gap between the one or more source model networks and the one or more target model networks in the feature space.

19. The computer-implemented cross-modal knowledge transfer method of claim 18, wherein: The combination of modality-agnostic loss functions includes entropy, pseudo-labeling, and diversity, wherein the entropy, the pseudo-labeling, and the diversity are defined on the outputs of the one or more source model networks.

20. The computer-implemented cross-modal knowledge transfer method of claim 18, wherein: The combination of the modality-agnostic loss functions includes the feature distance between the extracted TI source features and the TI target features and the distance between the TI source moment and the TR target moment.

Images