Systems and methods for unsupervised domain adaptation with hybrid training

Abstract

Systems and methods for unsupervised domain adaptation with hybrid training. The systems and methods of domain adaptation involve a first domain and a second domain. A machine learning system is trained with first sensor data and first label data of the first domain. Second sensor data of the second domain is obtained. Second label data is generated via the machine learning system based on the second sensor data. Inter-domain sensor (label) data is generated by interpolating the first sensor (label) data of the first domain with respect to the second sensor (label) data of the second domain. The machine learning system is operable to generate inter-domain output data in response to the inter-domain sensor data. Inter-domain loss data is generated based on the inter-domain output data with respect to the inter-domain label data. Parameters of the machine learning system are updated in optimizing final loss data that includes at least the inter-domain loss data. After domain adaptation, the machine learning system operable in the first domain is adapted to generate current label data that identifies current sensor data of the second domain.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to machine learning systems. BACKGROUND

[0002] Deep learning systems often rely on rich data and large amount of human annotation. As such, they often face serious challenges during deployment in real-world scenarios when domain shift occurs and labels under new releases are lacking or unavailable. These challenges are related to unsupervised domain adaptation (UDA), which involves building a predictive model for an unannotated target domain using a related source domain with rich labels. In addressing these challenges, some works focus on adversarial methods that learn domain-invariant features. However, this work has been found to be insufficient in achieving desired target domain performance. Further, there are other works that introduce additional training constraints, such as cluster assumption. However, these other methods impose constraints on selected domains individually and independently, and face several performance deficiencies as they ignore important interactions between source and target domains. SUMMARY

[0003] The following is a summary of certain embodiments detailed below. The described aspects are presented merely to provide the reader with a brief overview of these particular embodiments and the description thereof is not intended to limit the scope of the disclosure. Indeed, the present disclosure can encompass a variety of aspects that can not be explicitly set forth below.

[0004] According to at least one aspect, a computer-implemented method for unsupervised domain adaptation involves a first domain and a second domain. The method includes obtaining a machine learning system trained with first sensor data and first label data of the first domain. The method includes obtaining second sensor data of the second domain. The method includes generating second label data via the machine learning system based on the second sensor data. The method includes generating inter-domain sensor data by interpolating the first sensor data of the first domain with respect to the second sensor data of the second domain. The method includes generating inter-domain label data by interpolating the first label data of the first domain with respect to the second label data of the second domain. The method includes generating inter-domain output data via the machine learning system based on the inter-domain sensor data and the inter-domain label data. The method includes generating inter-domain loss data based on the inter-domain output data of the corresponding inter-domain label data. The method includes updating parameters of the machine learning system when optimizing final loss data. The final loss data includes the inter-domain loss data.

[0005] According to at least one aspect, a system includes at least a memory system and a processing system. The memory system includes at least one non-transitory computer- readable medium. The memory system includes a domain adaptation application and a machine learning system. The processing system is operatively connected to the memory system. The processing system includes at least one processor configured to execute the domain adaptation application to implement a method including generating inter-domain sensor data by interpolating first sensor data of a first domain with respect to second sensor data of a second domain. The method includes generating inter-domain label data by interpolating first label data of the first domain with respect to second label data of the second domain. The method includes generating inter-domain output data via the machine learning system based on the inter-domain sensor data and the inter-domain label data. The method includes generating inter-domain loss data based on the inter-domain output data with respect to the inter-domain label data. The method includes updating parameters of the machine learning system while minimizing final loss data, the final loss data including at least the inter-domain loss data. The method includes providing the machine learning system for deployment. The machine learning system is adapted to generate current label data classifying current sensor data of the second domain.

[0006] According to at least one aspect, a non-transitory computer-readable medium includes computer-readable data of a domain adaptation application that, when executed by a processing system having at least one processor, is configured to cause the processing system to implement a method including generating inter-domain sensor data by interpolating first sensor data of a first domain with respect to second sensor data of a second domain. The method includes generating inter-domain label data by interpolating first label data of the first domain with respect to second label data of the second domain. The method includes generating inter-domain output data via the machine learning system based on the inter-domain sensor data and the inter-domain label data. The method includes generating inter-domain loss data based on the inter-domain output data with respect to the inter-domain label data. The method includes updating parameters of the machine learning system while minimizing final loss data, the final loss data including at least the inter-domain loss data. The method includes providing the machine learning system for deployment or adoption. The machine learning system is adapted to generate current label data classifying current sensor data of the second domain.

[0007] These and other features, aspects, and advantages of the present application are discussed in the following detailed description, taken in connection with the accompanying drawings, where like numerals designate like parts, and in which: BRIEF DESCRIPTION OF DRAWINGS

[0008] Figure 1 is a diagram of an example of a system for training a machine learning system according to example embodiments of the present disclosure.

[0009] Figure 2is a conceptual diagram of a framework for unsupervised domain adaptation with hybrid training according to example embodiments of the present disclosure.

[0010] Figure 3 is a diagram of a method for unsupervised domain adaptation with hybrid training according to example embodiments of the present disclosure. DETAILED DESCRIPTION

[0011] The embodiments described herein, and numerous obvious substitutions, modifications, and equivalents, can be implemented without departing from the spirit of the disclosed subject matter and may be within the scope of the disclosed subject matter. The embodiments discussed herein are intended to cover and encompass all possible combinations of elements and features, and are intended to include all possible alternatives, equivalents, and modifications to the disclosed subject matter, including overcurrent protection devices other than those specifically described and / or claimed. It will be apparent to those skilled in the art that substantial modifications and improvements may be made in the specific embodiments which are shown and described without departing from the scope of the disclosed subject matter. Accordingly, the disclosure is not to be limited based on the specific examples provided for illustrative purposes.

[0012] In general, the present disclosure recognizes that learning only domain-invariant representations cannot bridge the performance gap between two domains. Unlike other methods that aim to solve the UDA problem by independently addressing the domains, the embodiments disclosed herein provide a framework that incorporates both domains in the design of constraints. By enforcing at least one constraint across domains, the embodiments disclosed herein directly address the target domain performance. In this regard, through a simple yet effective lens of hybrid training, the embodiments demonstrate that introducing training constraints across domains can significantly improve model adaptation performance.

[0013] Figure 1 An example of a system 100 for providing unsupervised domain adaptation according to example embodiments is illustrated. The system 100 includes at least a processing system 110. The processing system 110 includes at least an electronic processor, a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a field-programmable gate array (FPGA), a dedicated integrated circuit (ASIC), any suitable processing technology, or any number and combination thereof. The processing system 110 is operable to provide functionality as described herein.

[0014] The system 100 includes at least a memory system 120 operatively connected to the processing system 110. In example embodiments, the memory system 120 includes at least one non-transitory computer-readable medium configured to store and provide access to various data to enable at least the processing system 110 to perform operations and functions as disclosed herein. In example embodiments, the memory system 120 includes a single device or multiple devices. The memory system 120 can include electrical, electronic, magnetic, optical semiconductor, electromagnetic, or any suitable storage technology operable with the system 100. For example, in example embodiments, the memory system 120 can include random access memory (RAM), read-only memory (ROM), flash memory, disk drives, memory cards, optical storage devices, magnetic storage devices, memory modules, any suitable type of memory device, or any combination thereof. The memory system 120 is local, remote, or a combination thereof (e.g., partially local and partially remote) with respect to the processing system 110 and / or other components of the system 100. For example, the memory system 120 can include at least a cloud-based storage system (e.g., a cloud-based database system) that is remote from the processing system 110 and / or other components of the system 100.

[0015] The memory system 120 includes at least a domain adaptation application 130, a machine learning system 140, training data 150, and other related data 160 stored thereon. The domain adaptation application 130 includes computer-readable data configured to train the machine learning system 140 to adapt from a first domain to a second domain in an efficient manner when executed by the processing system 110. The computer-readable data can include instructions, code, routines, various related data, any software technology, or any number and combination thereof. With respect to the framework 200 and the machine learning system 140, the domain adaptation application 130 can include computer-readable data configured to perform the operations of the framework 200 and the machine learning system 140. Figure 2 Figure 3 ​The method 300 further discusses example implementations of the domain adaptation application 114. Moreover, the machine learning system 140 includes a convolutional neural network (CNN), a long short-term memory (LSTM) network, a recurrent neural network (RNN), any suitable artificial neural network, or any number and combination thereof. Furthermore, the training data 150 includes a sufficient amount of sensor data of the first domain, label data of the first domain, sensor data of the second domain, label data of the second domain, various mixup formulations (e.g., inter-domain sensor data, inter-domain label data, intra-domain sensor data, intra-domain label data, consistency regularization term data, etc.), various loss data, various weight data, and various parameter data, as well as any related machine learning data that enables the system 100 to provide unsupervised domain adaptation training as described herein. Meanwhile, the other related data 160 provides various data (e.g., operating systems, etc.) that enables the system 100 to perform the functions as discussed herein.

[0016] In example embodiments, the system 100 is configured to include at least one sensor system 170. The sensor system 170 includes one or more sensors. For example, the sensor system 170 includes an image sensor, a camera, a radar sensor, a light detection and ranging (LIDAR) sensor, a thermal sensor, an ultrasonic sensor, an infrared sensor, a motion sensor, an audio sensor, an inertial measurement unit (IMU), any suitable sensor, or any combination thereof. The sensor system 110 is operable to communicate with one or more other components of the system 100 (e.g., the processing system 110 and the memory system 120). More specifically, for example, the processing system 110 is configured to obtain sensor data directly or indirectly from one or more sensors of the sensor system 170. Upon receiving the sensor data, the processing system 110 is configured to process the sensor data in conjunction with the domain adaptation application 130 and the machine learning system 140.

[0017] Furthermore, the system 100 includes other components that facilitate the training of the machine learning system 140. For example, as shown in Figure 1 The memory system 120 is also configured to store other related data 160 that relates to the operation of the system 100 in connection with one or more components (e.g., the sensor system 170, the I / O devices 180, and the other functional modules 190), among other things. Moreover, the system 100 is configured to include one or more I / O devices 180 (e.g., a display device, a keyboard device, a speaker device, etc.) that relate to the system 100. Furthermore, the system 100 includes other functional modules 190, such as any suitable hardware, software, or combination thereof, that assist or facilitate the functioning of the system 100. For example, the other functional modules 190 include communication technology that enables the components of the system 100 to communicate with each other as described herein, among other things. Figure 1The configuration discussed in the example allows system 100 to operate to train machine learning system 140 developed in a first domain (“source domain”) to adapt to a second domain (“target domain”).

[0018] Figure 2 The diagram illustrates aspects of domain-adaptive application 130, machine learning system 140 (e.g., classification model 220 and discriminator 250), and training data 150. More specifically, Figure 2 The illustration shows an inter-domain and intra-domain hybrid training (IIMT) framework 200 for unsupervised domain adaptation according to an example embodiment. In this example, the labeled source domains are denoted as sets. And unlabeled target domains are marked as sets. Furthermore, in this example, the one-hot label is defined by y i Labeling. Furthermore, the overall classification model 220 is labeled as the one parameterized by θ. More specifically, in Figure 2 In the text, classification model 220 is labeled as... The embedded encoder 230 and the one labeled as The composite of embedded classifier 240: In example embodiments, such as Figure 2 As shown, encoder 230 is shared by two domains (i.e., the source domain and the destination domain). Furthermore, as... Figure 2 As shown, the IIMT framework 200 includes a blending 210, which is a core component and is applied both across two domains (e.g., inter-domain) and within each domain (e.g., intra-domain for the source domain and intra-domain for the target domain). The IIMT framework 200 also includes a consistency regularization term to better facilitate inter-domain blending in the presence of relatively large domain differences. Furthermore, the IIMT framework 200 is configured to provide end-to-end training on its loss data, such as all blending training losses (e.g., L...). s L q L t and L z ) and domain adversarial loss (e.g., L d ).

[0019] Figure 3 An example of a method 300 for providing unsupervised domain adaptation according to an exemplary embodiment is illustrated. In this case, method 300 is executed via domain application 130 when executed by processing system 110. However, system 100 is not limited to implementing method 300, but may provide other methods to capture... Figure 2Other methods (e.g., different steps or different order of steps) can be implemented with the understanding of the functions and objectives of the IIMT framework 200. For example, the intra-domain level of the source domain’s mixing formulation, although referenced at step 306, can occur before step 304, at the same time as step 304, or whenever the source domain’s sensor data and label data are available for mixing operations. Moreover, other modifications to the method 300 can be made without departing from the spirit and scope of the IIMT framework 200, which is discussed below.

[0020] At step 302, the processing system 110 obtains training data. In an example embodiment, the training data includes annotated sensor data of a source domain. In this case, as shown in Figure 2 , the label data (y s ) represents the true and / or verified classification of the sensor data (x s ) of the source domain. Additionally, as shown in Figure 2 , the processing system 110 obtains sensor data (x t ) of a target domain. In this case, the target domain is related to the source domain. The sensor data of the target domain can be unannotated or mostly unannotated.

[0021] At step 304, the processing system 110 is configured to generate label data for the sensor data of the target domain. For example, in the case that the processing system obtains unannotated sensor data of the target domain, the processing system 110 is configured to generate and utilize inferred labels (q t ) as relatively weak supervision for the target domain. In generating these inferred labels (q t ), the processing system 110 is configured to integrate data augmentation and entropy reduction into the virtual label (inferred label) generation process. Specifically, the processing system 110 is configured to perform “K” task-dependent random augmentations (e.g., random cropping and flipping for images and random scaling for time series) for each data sample to obtain transformed samples , where K represents an integer value.

[0022] The processing system 110 is then configured to generate the target domain virtual labels q i by performing the calculations of equation (1) and equation (2).

[0023]

[0024]

[0025] In the above equations, T denotes the softmax temperature, and c is the class exponent. That is, the processing system 110 averages the class predictions (e.g., the virtual labels) over the K augmented inputs to construct and then further performing sharpening to form virtual labels When using a smaller value (e.g., T < 1) produces sharper predicted distributions and helps to reduce the conditional entropy. The processing system 110 is now able to use these virtual labels (q t ) as annotations for the sensor data (x t ) of the target domain.

[0026] At step 306, the processing system 110 is configured to perform a mixing operation for the intra-domain level for the source domain, the inter-domain level across both the source domain and the target domain, and the intra-domain level for the target domain. These mixing operations can be generated in any suitable order, simultaneously, or any combination thereof. The processing system 110 is configured to perform the mixing operations as discussed below. Further, as shown in Figure 2 , the processing system 110 also performs at least one mixing operation for the consistency regularization term, but this mixing operation includes the encoded representation from the encoder 230 and is thus discussed at step 310. With these mixing operations included, the processing system 110 encourages linear behavior of the machine learning system 140, as linear interpolation in the original data leads to linear interpolation in the predictions.

[0027] Intra-domain mixing operation for source domain

[0028] For the intra-domain mixing, the processing system 110 is configured to select pairs of samples from the source domain, e.g., and . The processing system 110 is configured to perform mixing to enforce linear consistency within the source domain by generating the following mixed formulation based on the selected pairs of source samples, as indicated by the following equation, where .

[0029]

[0030]

[0031] After performing this mixing for the source domain, the processing system 110 is configured to provide the mixed formulation of as training data for the machine learning system 140. In this case, represents the intermediate sensor data for the source domain and represents the intermediate label data for the source domain. In this regard, this intra-domain mixing formulation serves as a data augmentation strategy and is particularly useful for achieving UDA in prediction smoothness in cases where there are sudden changes in predictions around data samples. Further, the intra-domain mixing imposes strong linearity to enforce a separate local Lipschitz for this source domain.

[0032] Once the interpolated results (i.e., the mixed formulation) are obtained, the processing system 110 is configured to generate, via the machine learning system 140, label data that classifies the interpolated results as belonging to the source domain or the target domain. Because samples of the same domain follow similar distributions, there is no need to apply linearities at the feature level. Thus, the processing system 110 employs these label-level mixed operations for the source domain and determines its corresponding loss data via the following equation .

[0033]

[0034] Inter-domain mixing operation for source domain and target domain

[0035] For inter-domain mixing, the processing system 110 is configured to select pairs of samples from their corresponding batches, e.g., and . The processing system 110 is configured to perform mixing to enforce linear consistency across the source and target domains by generating the following mixed formulation based on the selected pairs of source and target samples, as indicated by the following equation, where .

[0036]

[0037]

[0038] After performing this mixing across the two domains, the processing system 110 is configured to provide the mixed formulation of as training data for the machine learning system 140. In this case, represents inter-domain sensor data resulting from performing interpolation across the two domains, and represents inter-domain virtual label data resulting from performing interpolation across the two domains, thus facing important interactions between the source and target domains.

[0039] Inter-domain mixing training serves as a key component in the IIMT framework 200. In the training of the classification model 220, inter-domain mixing provides interpolated labels to enforce linear predictive behavior of the classifier across domains. This mixed training with interpolated labels induces a simplified inductive bias compared to training with source labels alone, which can directly improve the generalization capability of the classifier 240 for the target domain. When generating, via the machine learning system 140, label data that classifies the inter-domain mixed formulation, e.g., , the processing system 110 is configured to generate loss data via the following equation .

[0040]

[0041] In Equation (8), B denotes the batch size, and H denotes the cross-entropy loss. Furthermore, in Equation (6) and Equation (7), the mixing weight parameter is chosen based on the following equation.

[0042]

[0043]

[0044] In Equation (9), Beta refers to a Beta distribution with shared shape parameter When is set closer to 1, there is a greater probability of choosing a median from the range [0; 1] as resulting in a higher level of interpolation between the two domains. Furthermore, due to Equation (10), therefore is always above 0.5 to ensure that the source domain is dominant. Similarly, mixing dominated by the target domain can be generated via a transformation of and in Equation (6) that forms With , the processing system 110 is configured to use a mean squared error (MSE) loss as it is more forgiving of false virtual labels in the target domain.

[0045] Intra-domain mixing operation for target domain

[0046] For intra-domain mixing, the processing system 110 is configured to select pairs of samples from the target domain, e.g., and . As mentioned previously, with respect to the target domain, the labels (e.g., and ) are inferred labels that are generated by the machine learning system 140 based on sensor data of the target domain and are thus predicted labels (i.e., not ground truth labels). The processing system 110 is configured to perform mixing to enforce linear consistency within the target domain by generating the following mixed formulation based on the selected pairs of target samples, as indicated by the following equation, where .

[0047]

[0048]

[0049] After performing this mixing for the target domain, the processing system 110 is configured to provide The hybrid formula is used as training data for the machine learning system 140. In this case, This represents intermediate sensor data in the target domain and This represents the intermediate virtual label data for the target domain. In this regard, the intra-domain blending formulation is used as a data augmentation strategy and is particularly useful for achieving UDA in predictive smoothness when sudden changes in prediction occur near the data samples. Furthermore, the intra-domain blending imposes strong linearity to implement a separate local Lipschitz for the target domain.

[0050] Once the interpolation result (i.e., hybrid formulation) is obtained, the processing system 110 is configured to generate label data via the machine learning system 140, which in turn affects the interpolation result. Classification is performed. Because samples within the same domain follow similar distributions, feature-level linearity does not need to be applied. Therefore, the processing system 110 employs a mixture of these label-level operations for the target domain and determines the corresponding loss data via the following equation. .

[0051]

[0052] At step 308, the processing system 110 is configured to employ standard domain adversarial training to reduce domain dissimilarity. In an example embodiment, this implementation is limited to the Domain Adversarial Neural Network (DANN) framework to focus on evaluating mixed linear constraints. Other embodiments may include other and / or more complex formulations besides the DANN framework. More specifically, as Figure 2 As shown, the shared embedding encoder 230 (e.g., generator) and discriminator 250 are trained under an adversarial objective, such that encoder 230 learns generative domain-invariant features. At this point, discriminator 250 is labeled as... The binary domain labels are annotated with 0 / 1. Using DANN, the processing system 110 determines the domain adversarial loss (L0) via the following equation. d ):

[0053] .

[0054] like Figure 2 As shown and indicated in equation (14), the processing system 110 trains the discriminator based on a mixed formulaic encoding representation between the source and target samples, rather than directly on the original samples themselves.

[0055] At step 310, the processing system 110 is configured to impose a consistency regularization term for the latent features to better facilitate the cross-domain mixed training. The consistency regularization term is particularly effective in the presence of relatively large domain discrepancies, as the linear constraints imposed by the cross-domain mixing can not be as effective. Specifically, when interpolating between heterogeneous raw inputs in equation (6), then there can be some difficulty in forcing the classification model 220 to produce corresponding interpolated predictions. At the same time, the joint training with domain adversarial loss for feature-level domain confusion (step 308) can increase this training difficulty. The processing system 110 therefore imposes a consistency regularization term to address these challenges. More specifically, as shown in Figure 2 , the processing system 110 generates mixed formulations for the consistency regularization term, and then trains the machine learning system 140 based on these mixed formulations, as discussed below.

[0056] For the consistency regularization term, the processing system 110 is configured to select pairs of samples from their corresponding batches and from the corresponding output data (e.g., encoded representations) of the encoder 230, e.g., and . The processing system 110 is configured to perform mixing to enforce linear consistency for the cross-domain mixing by generating the following mixed formulations based on the selected pairs of samples, as indicated by the following equations, where .

[0057]

[0058]

[0059] In equation (16), the processing system 110 denotes as the embedding space induced by and in defining the regularization term . Equation (15) is the same as equation (6) in that they both generate cross-domain sensor data (x st ), i.e., sensor data interpolated based on sensor data from the source domain and sensor data from the target domain. In this regard, the processing system 110 is configured to use the from the mixing performed at step 306. After performing these mixtures, the processing system 110 is configured to provide each mixed formulation of as training data for the machine learning system 140.

[0060] As shown in Figure 2 , the processing system 110 generates a mixed formulation for each mixed formulation At least the encoder 230 is trained, and a loss function is computed based on the training. In this example, the processing system 110 generates loss data for the consistency regularization term, which is denoted as .

[0061]

[0062] As indicated in equation (17), the consistency regularization term pushes the mixed features closer to the features of the mixed input through the MSE loss between the two vectors. In this way, the processing system 110 also imposes the linear constraint to be enforced at the feature level. The efficacy of this consistency regularization term is that when equation (17) is enforced and the training data passes through the shallow classifier 240, then the linearity in the model prediction is more easily satisfied. Moreover, when the processing system 110 imposes linear smoothness of the features across domains, then this action can support the domain confusion loss. Furthermore, similar to the processing of equation (6), and can also be switched in equation (16) to form As a modification, the machine learning system 140 is provided with as the training data.

[0063] At step 312, the processing system 110 is configured to determine the final loss data. More specifically, based on the foregoing components, the processing system 110 is configured to determine the final loss data, denoted by (L) and expressed by the following equation.

[0064]

[0065] Since only involves the virtual labels, then can be easily affected by the uncertainty in the target domain. In this regard, the processing system 110 is configured to set a linear schedule from 0 to a predetermined maximum value for during the training. According to initial experiments, this algorithm for determining and / or optimizing the final loss data is proven to be robust to other weighting parameters. More specifically, according to this algorithm, the processing system 110 is configured to search for only, while simply fixing all other weights to 1.

[0066] At step 314, the processing system 110 is configured to minimize the final loss data, as determined at step 312. In minimizing the final loss data, the processing system 110 is configured to update the parameters of the machine learning system 140. In updating the parameters and / or training with the updated parameters, the processing system 110 is configured to determine whether the machine learning system 140 meets a predetermined threshold criterion, and when the machine learning system 140 meets the predetermined threshold criterion, make the machine learning system 140 available for step 316.

[0067] In meeting the predetermined threshold criterion, the machine learning system 140 that was previously trained to operate in the first domain is now adapted to operate in the second domain via the framework 200 and the method 300. In this regard, the machine learning system 140 is configured to generate label data classifying sensor data of the first domain when received as input data, and also generate label data classifying sensor data of the second domain when received as input data. That is, the IIMT framework 200 and / or the method 300 are effective in further opening up an existing machine learning system 140 that has been trained with rich annotated training data of the first domain to expand to and operate in the second domain, even when annotations of training data of the second domain are lacking or unavailable, thereby providing a more robust and cost-effective technical solution to the processing domain shift.

[0068] At step 316, the processing system 110 is configured to provide the machine learning system 140 for deployment and / or adoption in any suitable application system. For example, the trained machine learning system 140 can be employed by the system 100, which can further include an application system that further uses or applies current label data generated by the trained machine learning system 140 in response to current sensor data. Alternatively, the trained machine learning system 140 can be deployed into another system (not shown) that includes at least some similar components as the system 100 (e.g., a sensor system, a processing system, etc.) along with an application system that applies current label data generated by the trained machine learning system 140 in response to current sensor data. Figure 1

[0069] ​The application system can include software technology, hardware technology, or any combination thereof. As a non-limiting example, the application system can involve gesture glove technology that includes sensors that provide sensor data to enable the machine learning system 140 to classify various gestures. In this example, the machine learning system 140 can be trained to classify gestures when performed by at least one individual and further adapted via the IIMT framework 200 and method 300 to classify the same gestures when performed by at least one other individual with some variation. As other examples, the application system can involve computer-controlled machines, robots, electronic home appliances, electronic power tools, health / medical technology, autonomous or driver-assisted vehicle technology, security technology, or any suitable technology that includes employing the machine learning system 140.

[0070] In general, various domains and applications can benefit from being able to extend the capabilities of a machine learning system that is operable in a first domain to further operate in a second domain related to the first domain. For example, as a non-limiting example, when there is a change from one sensor to another of the same type (e.g., upgrading one image sensor to another image sensor to enhance images), based on the change in the sensor, there can be some shift / variation in the sensor data that will be input into the machine learning system 140. Advantageously, the foregoing embodiments address these domain shifts / variation and ensure that the machine learning system 140 operates accurately in each domain and across both domains.

[0071] As discussed herein, embodiments include many advantageous features and benefits. For example, embodiments are advantageously configured to impose cross-domain training constraints on domain adaptation by mixing linearity of the lens. Further, to address potentially large domain differences, embodiments include a consistency regularization term at the feature level to facilitate inter-domain constraints. Further, the processing system 110 enables mixed training across domains via inferred labels (or virtual labels) of the target domain. Further, as both mixed training and domain adversarial training progress, the classification model 220 infers virtual labels with increasing accuracy. This process, when applied to the target domain, can be critical for directly improving generalization of the classifier 240.

[0072] Furthermore, embodiments are advantageous in providing the IIMT framework 200 for UDA, where all training constraints are unified under a hybrid formulation. Moreover, with the IIMT framework 200, embodiments incorporate both inter-domain and intra-domain hybrid training, and thus outperform state-of-the-art approaches in diverse application domains (e.g., image classification, human activity recognition, etc.). In this regard, embodiments disclosed herein address issues related to the differences between domains for the UDA problem. Furthermore, embodiments can be modified or enhanced based on the commonalities and relationships between the inter-domain hybrid linearity and the domain discriminators. Moreover, the teachings disclosed herein can extend to advancing and applying to advancing general time series analysis.

[0073] That is, the above description is intended to be illustrative, and not restrictive, and is provided in the context of a particular application and its requirements. Those skilled in the art will appreciate from the foregoing description that the application can be implemented in a variety of forms, and that the various embodiments can be implemented alone or in combination. Therefore, although the embodiments of the application have been described in connection with specific examples, it is to be understood that the application is not limited to the specific examples, and that there are other embodiments of the application that will be obvious to those of ordinary skill in the art in view of the foregoing description, and that the scope of the application should be determined not with reference to the above description but should be determined with reference to the appended claims, along with their full scope of equivalents. For example, components and functions can be separated or combined that were described as being separate or combined. Similarly, functions can be described as being performed by one component that was described as performing other functions, or by more than one component. Furthermore, not all of the features and aspects of the embodiments described can be necessary to practice the application. Thus, for example, the specific embodiments described above can not all be required to practice the application.

Claims

1. A computer-implemented method for unsupervised domain adaptation involving a first domain and a second domain, the method comprising: obtaining a machine learning system trained with first sensor data and first label data of the first domain; obtaining second sensor data of the second domain; generating second label data via the machine learning system based on the second sensor data; generating inter-domain sensor data by interpolating the first sensor data of the first domain with respect to the second sensor data of the second domain; generating inter-domain label data by interpolating the first label data of the first domain with respect to the second label data of the second domain; generating inter-domain output data based on the inter-domain sensor data and the inter-domain label data; generating inter-domain loss data based on the inter-domain output data with respect to the inter-domain label data; and updating parameters of the machine learning system while optimizing final loss data, the final loss data comprising the inter-domain loss data, wherein the first and second sensor data belong to image data.

2. The computer-implemented method of claim 1, further comprising: obtaining a first encoded representation of the first sensor data and a second encoded representation of the second sensor data; generating intermediate encoded representation data by interpolating the first encoded representation with respect to the second encoded representation; encoding the intermediate encoded representation data to generate a third encoded representation; and generating regularization loss data based on the third encoded representation with respect to the inter-domain sensor data, wherein the final loss data comprises the regularization loss data.

3. The computer-implemented method of claim 1, wherein the second label data is predicted label data generated by the machine learning system in response to the second sensor data.

4. The computer-implemented method of claim 1, further comprising: training a discriminator based on encoded representations of the inter-domain sensor data and the inter-domain label data; and generating domain adversarial loss data based on output data from the discriminator, wherein the final loss data comprises the domain adversarial loss data.

5. The computer-implemented method of claim 1, further comprising: generating intermediate sensor data by interpolating pairs of the first sensor data of the first domain; generating intermediate label data by interpolating pairs of the first label data of the first domain; generating first label predictions via the machine learning system based on the intermediate sensor data; and generating first domain loss data based on the intermediate label data with respect to the first label predictions, wherein the final loss data comprises the first domain loss data.

6. The computer-implemented method of claim 1, further comprising: generating intermediate sensor data by interpolating pairs of the second sensor data of the second domain; generating intermediate label data by interpolating pairs of the second label data of the first domain; generating second label predictions via the machine learning system based on the intermediate sensor data; and generating second domain loss data based on the intermediate label data with respect to the second label predictions, wherein the final loss data comprises the second domain loss data.

7. The computer-implemented method of claim 1, wherein the machine learning system is a convolutional neural network (CNN) or a long short-term memory (LSTM) network.

8. A system for domain adaptation, the system comprising: ​ ​ ​ A memory system comprising at least one non-transitory computer-readable medium storing a domain adaptation application and a machine learning system; a processing system operatively connected to the memory system, the processing system comprising at least one processor configured to execute the domain adaptation application to implement a method comprising: generating inter-domain sensor data by interpolating first sensor data of a first domain with respect to second sensor data of a second domain; generating inter-domain label data by interpolating first label data of the first domain with respect to second label data of the second domain; generating, via the machine learning system, inter-domain output data based on the inter-domain sensor data and the inter-domain label data; generating inter-domain loss data based on the inter-domain output data with respect to the inter-domain label data; updating parameters of the machine learning system while minimizing final loss data, the final loss data comprising at least the inter-domain loss data; and providing the machine learning system for deployment, the machine learning system adapted to generate current label data that classifies current sensor data of the second domain, wherein the first and second sensor data belong to image data.

9. The system of claim 8, wherein, the processing system, when executing the domain adaptation application, implements a method comprising: obtaining a first encoded representation of the first sensor data and a second encoded representation of the second sensor data; generating intermediate encoded representation data by interpolating the first encoded representation with respect to the second encoded representation; encoding the intermediate encoded representation data to generate a third encoded representation; and generating regularization loss data based on the third encoded representation with respect to the inter-domain sensor data, wherein the final loss data comprises the regularization loss data.

10. The system of claim 8, wherein: the machine learning system is trained to operate in the first domain; and the second label data is predicted label data generated by the machine learning system in response to the second sensor data.

11. The system of claim 8, wherein, the processing system, when executing the domain adaptation application, implements a method comprising: training a discriminator based on encoded representations of the inter-domain sensor data and encoded representations of the inter-domain label data; and generating domain adversarial loss data based on output data from the discriminator, wherein the final loss comprises the domain adversarial loss data.

12. The system of claim 8, wherein, the processing system, when executing the domain adaptation application, implements a method comprising: generating intermediate sensor data by interpolating pairs of first sensor data of the first domain; generating intermediate label data by interpolating pairs of first label data of the first domain; generating, via the machine learning system, a first label prediction based on the intermediate sensor data; and generating first domain loss data based on the intermediate label data with respect to the first label prediction, wherein the final loss data comprises the first domain loss data.

13. The system of claim 8, wherein, the processing system, when executing the domain adaptation application, implements a method comprising: generating intermediate sensor data by interpolating pairs of second sensor data of the second domain; generating intermediate label data by interpolating pairs of second label data of the first domain; generating, via the machine learning system, a second label prediction based on the intermediate sensor data; and generating second domain loss data based on the intermediate label data with respect to the second label prediction, wherein the final loss data comprises the second domain loss data. ​ 14. A non-transitory computer-readable medium comprising computer-readable data of a domain adaptation application, the computer-readable data of the domain adaptation application, when executed by a processing system having at least one processor, configured to cause the processing system to implement a method comprising: generating inter-domain sensor data by interpolating first sensor data of a first domain with respect to second sensor data of a second domain; generating inter-domain label data by interpolating first label data of the first domain with respect to second label data of the second domain; generating, via a machine learning system, inter-domain output data based on the inter-domain sensor data and the inter-domain label data; generating inter-domain loss data based on the inter-domain output data with respect to the inter-domain label data; updating parameters of the machine learning system while minimizing final loss data, the final loss data comprising at least the inter-domain loss data; and providing the machine learning system for deployment, the machine learning system adapted to generate current label data that classifies current sensor data of the second domain, wherein the first and second sensor data belong to image data.

15. The non-transitory computer-readable medium of claim 14, wherein the method further comprises: obtaining a first encoded representation of the first sensor data and a second encoded representation of the second sensor data; generating intermediate encoded representation data by interpolating the first encoded representation with respect to the second encoded representation; encoding the intermediate encoded representation data to generate a third encoded representation; and generating regularization loss data based on the third encoded representation with respect to the inter-domain sensor data; wherein the final loss data comprises the regularization loss data.

16. The non-transitory computer-readable medium of claim 14, wherein: the machine learning system is trained to operate in the first domain; and the second label data is predicted label data generated by the machine learning system in response to the second sensor data.

17. The non-transitory computer-readable medium of claim 14, wherein the method further comprises: training a discriminator based on encoded representations of the inter-domain sensor data and the inter-domain label data; and generating domain adversarial loss data based on output data from the discriminator, wherein the final loss data comprises the domain adversarial loss data.

18. The non-transitory computer-readable medium of claim 14, wherein the method further comprises: generating intermediate sensor data by interpolating pairs of first sensor data of the first domain; generating intermediate label data by interpolating pairs of first label data of the first domain; generating, via the machine learning system, a first label prediction based on the intermediate sensor data; and generating first domain loss data based on the intermediate label data with respect to the first label prediction, wherein the final loss data comprises the first domain loss data.

19. The non-transitory computer-readable medium of claim 14, wherein the method further comprises: generating intermediate sensor data by interpolating pairs of second sensor data of the second domain; generating intermediate label data by interpolating pairs of second label data of the first domain; generating, via the machine learning system, a second label prediction based on the intermediate sensor data; and ​ ​ ​ generating second domain loss data based on the intermediate label data predicted with respect to the second label, wherein the final loss includes the second domain loss data.

20. The non-transitory computer-readable medium of claim 14, wherein the machine learning system is a convolutional neural network (CNN) or a long short-term memory (LSTM) network.

Images