Method, apparatus, and storage medium for domain adaptation for neural networks

Abstract

To provide a domain adaptation method for a neural network, a device and a storage medium.SOLUTION: Mixed samples are acquired by mixing first and second samples from a source domain, and a third sample from a target domain, the first, second and third mixed samples belong to the same class, and a first loss function is constructed on the basis of an in-class distance between the samples in the source domain and the mixed samples, and an inter-class distance. The first loss function minimizes the weighted-averaged in-class distance, maximizes the weighted-averaged inter-class distance, constructs second loss functions with respect to a mixed domain on the basis of a sum of the first loss function and a cross entropy loss function, decides whether or not a neural network is converged by using the second loss function of the source domain and the second loss function of the mixed domain, and when the neural network is not converged, repeats the above steps.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to the field of deep neural networks, and more specifically, to supervised domain adaptation.

Background Art

[0002] In recent years, deep neural network models have made great progress in tasks such as computer vision and natural language processing. However, these advancements typically rely on large-scale labeled data such as ImageNet. When directly applying the model to a new environment, its performance will inevitably decrease significantly. This is because there is a shift, i.e., a domain shift, between the data in the new environment and the data used in model training. Usually, by constructing a target dataset with sufficiently diverse samples in the new environment and using this dataset to retrain or fine-tune the model, the impact of domain shift can be reduced. However, accurately labeling large-scale data is very costly and time-consuming.

[0003] As one solution to this problem, an unsupervised domain adaptation (UDA) method is used to transfer useful knowledge in the source domain corresponding to the old environment to the target domain corresponding to the new environment. The UDA method is very effective, but a large number of samples in the target domain without labels are required to train the target model. Therefore, a long training time and a lot of computing resources are needed. As another solution, the model is trained from scratch or the original model is fine-tuned using only a small number of labeled samples for each class in the target domain to transfer useful knowledge in the source domain to the target domain. Since the number of samples used is small, the above problems can be avoided and the model can quickly adapt to environmental changes. Also, in many cases, the number of target domain samples that can be obtained is very limited.

[0004] Conventionally, to solve the above problems, there are several methods to train a target model using only a small number of labeled target domain samples. CCSA (Unified Deep Supervised Domain Adaptation and Generalization) (see Reference 1) is the first method proposed to perform domain adaptation using only a small number of labeled target domain samples. This method realizes class alignment between the source domain and the target domain by minimizing the average intra-class distance of cross-domain sample pairs and maximizing the average inter-class distance of cross-domain sample pairs. Also, FADA (Few-Shot Adversarial Domain Adaptation) (see Reference 2) forms sample pairs with a small number of samples in the target domain and samples in the source domain, and realizes domain adaptation by performing domain adversarial training on the sample pairs. FADA first creates four groups of sample pairs, and then trains a "domain-class discriminator" to identify which group each sample pair belongs to. On the other hand, a feature extractor is trained so that it can extract features that make the "domain-class discriminator" unable to distinguish which group each sample pair belongs to. Through the adversarial training between the feature extractor and the "domain-class discriminator", the feature extractor can extract domain-invariant features and realize class alignment. d-SNE (Domain Adaptation Using Stochastic Neighborhood Embedding) (see Reference 3) is the method with the best current effect. This method realizes domain adaptation by minimizing the maximum intra-class distance of cross-domain sample pairs and maximizing the minimum inter-class distance of cross-domain sample pairs.

[0005] The above method has achieved certain effects, but there are still some drawbacks. First, without considering data augmentation, a small number of labeled data in the target domain are directly used. To increase the number of samples in the target domain, several data augmentation methods have been proposed, such as adding Gaussian noise, rotating the image, trimming, etc. to the image. Furthermore, in order to perform data augmentation, the method of input mixup (see reference 4) or manifold mixup may be used. However, all of these conventional data augmentation methods are data augmentations for a single domain and are not suitable for domain adaptation.

[0006] In addition, CCSA and d-SNE represent two extreme scenarios respectively. CCSA treats all intra-class distances and inter-class distances equally, while d-SNE only considers the maximum intra-class distance and the minimum inter-class distance. [References] Reference 1: Motiian S, Piccirilli M, Adjeroh D A , et al. Unified Deep Supervised Domain Adaptation and Generalization (CCSA) (ICCV 2017) Reference 2: Motiian S, Jones Q, Iranmanesh S M, et al. Few-Shot Adversarial Domain Adaptation (FADA) (NIPS 2017) Reference 3: Xu X, Zhou X, Venkatesan R, et al. d-SNE: Domain adaptation using stochastic neighborhood embedding (d-SNE) (CVPR 2019) Reference 4: Zhang H, Cisse M, Dauphin Y N, et al. mixup: BEYOND EMPIRICAL RISK MINIMIZATION (Input Mixup) (ICLR 2018)

SUMMARY OF THE INVENTION

PROBLEMS TO BE SOLVED BY THE INVENTION

[0007] The following briefly describes the present disclosure to basically understand the aspects of the present disclosure. Note that this brief summary is not an exhaustive summary of the present disclosure, does not intentionally identify the points or important parts of the present disclosure, does not intentionally limit the scope of the present disclosure, and is only for the purpose of briefly explaining the concepts in a simple form as a preamble to the more detailed description described later.

[0008] The present disclosure provides a method, an apparatus, and a storage medium for domain adaptation for neural networks.

MEANS FOR SOLVING THE PROBLEM

[0009] In one aspect of the present disclosure, a method for domain adaptation for a neural network includes a mixing step of mixing a first sample and a second sample from a source domain and a third sample from a target domain to obtain a mixed sample, where the first sample, the second sample, the third sample, and the mixed sample belong to the same class; a first construction step of constructing a first loss function based on the sum of the intra-class distance and the inter-class distance between the samples in the source domain and the mixed sample, where the first loss function minimizes the weighted average intra-class distance and maximizes the weighted average inter-class distance; a second construction step of constructing a second loss function for a mixed domain composed of the source domain and the mixed sample respectively based on the sum of the first loss function and the cross-entropy loss function; and a determination step of determining whether the neural network has converged using both the second loss function of the source domain and the second loss function of the mixed domain, and if the neural network has not converged, repeatedly executing the mixing step, the first construction step, and the second construction step.

[0010] Preferably, the method further includes a step of obtaining the mixed sample by adding the weighted value of the third sample to the weighted value of the difference between the first sample and the second sample.

[0011] Preferably, the method further includes a step of obtaining the mixed sample by adding the third sample to the weighted value of the average value of the first sample and the second sample.

[0012] Preferably, when the method obtains the mixed sample by adding the third sample to the weighted value of the average value of the first sample and the second sample, it further includes a step of setting an upper limit and a lower limit and constructing the mixed domain based on the mixed sample between the upper limit and the lower limit.

[0013] Preferably, the weights of the weighted average are calculated based on distances using a normalized exponential function.

[0014] Preferably, the first loss function is constructed to subtract the inter-class distance from the sum of a fixed tolerance value and the maximum intra-class distance, and the fixed tolerance value is used to remove inter-class distances whose lengths exceed the tolerance value.

[0015] Preferably, the method further includes updating the parameters of the neural network using the second loss function of the source domain and the second loss function of the mixed domain when the neural network has not converged.

[0016] Preferably, the second loss function is constructed based on the argmin function, in which a first weight is assigned to the first loss function and a second weight is assigned to the cross-entropy loss function, and the sum of the first weight and the second weight is 1.

[0017] In another aspect of the present disclosure, there is provided an apparatus for domain adaptation for a neural network, comprising: a mixing unit configured to obtain a mixed sample by mixing a first sample and a second sample from a source domain and a third sample from a target domain, wherein the first sample, the second sample, the third sample, and the mixed sample belong to the same class; a first constructing unit configured to construct a first loss function based on a sum of an intra-class distance and an inter-class distance between a sample in the source domain and the mixed sample, wherein the first loss function minimizes a weighted average of the intra-class distances and maximizes a weighted average of the inter-class distances; a second constructing unit configured to construct a second loss function for a mixed domain composed of the source domain and the mixed sample based on a sum of the first loss function and a cross-entropy loss function; and a determining unit configured to determine whether the neural network has converged by using both the second loss function of the source domain and the second loss function of the mixed domain. When the determining unit determines that the neural network has not converged, the mixing unit, the first constructing unit, and the second constructing unit repeatedly execute their respective processes.

[0018] In other aspects of the present disclosure, there are further provided corresponding computer program codes, computer-readable storage media, and computer program products.

[0019] According to the method and apparatus for domain adaptation for a neural network according to the present disclosure, target domain data can be augmented, and the performance of the model can be optimized.

[0020] The above and other advantages of the present disclosure will become more apparent by describing in detail the preferred embodiments of the present disclosure with reference to the drawings.

Brief Description of the Drawings

[0021] To understand the above and other advantages and features of the present disclosure, specific embodiments of the present disclosure will be described in detail below with reference to the drawings. The drawings and the following detailed description are included herein and form a part of this specification. Elements having the same function and structure are denoted by the same reference numerals. Note that these drawings are only for explaining typical examples of the present disclosure and do not limit the scope of the present disclosure.

Figure 1

Figure 2

Figure 3A

Figure 3B

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Embodiments for Carrying Out the Invention

[0022] Hereinafter, exemplary embodiments of the present disclosure will be described in detail with reference to the drawings. For convenience of explanation, not all features of the actual embodiments are shown in the specification. In actual implementation, specific embodiments may be modified to achieve the specific goals of the developer. For example, the embodiments may be modified according to system and business constraints. Also, although the development work is very complex and time-consuming, for those skilled in the art of this disclosure, this development work is merely an example work.

[0023] Note that, for clarity of the present disclosure, only the components of the apparatus and / or the processing steps closely related to the present disclosure are shown in the drawings, and details not related to the present disclosure are omitted.

[0024] As described above, there are still problems with conventional methods of domain adaptation such as CCSA, FADA, and d-SNE. To solve these problems, the present disclosure provides a new method for performing domain adaptation using only a small number of labeled target domain samples. First, since the conventional data mixing method is not ideal, the present disclosure provides a new domain mixing method for expanding target domain data. Second, the method of the present disclosure optimizes the model by minimizing the weighted average intra-class distance and maximizing the weighted average inter-class distance.

[0025] The following will describe a method 100 for domain adaptation for a neural network according to an embodiment of the present disclosure with reference to FIGS. 1 to 7.

[0026] As shown in FIG. 1, in step 101, a first sample and a second sample from the source domain are mixed with a third sample from the target domain to obtain a mixed sample. Here, the first sample, the second sample, the third sample, and the mixed sample belong to the same class.

[0027] Note that the conventional method has a problem that the number of training data in the target domain is not sufficient. If there is a method to increase the number of target domain samples or a method to generate samples similar to the target domain samples to increase the number of target domain samples, the performance of the model in the target domain can be improved. Also, all the conventional methods for increasing the number of samples in the target domain are data augmentations for a single domain and are not suitable for domain adaptation. In the technical problem to be solved by the present disclosure, since the number of samples in the target domain is very small, simply expanding the data of the target domain using the data of the target domain itself cannot achieve very good results. Therefore, the present disclosure proposes a method of transferring the diversity of samples in the source domain to the target domain using a domain mixing method.

[0028] FIG. 2 is a diagram schematically showing the basic concept of domain mixing. As shown in FIG. 2, in each training iteration, a set of mixed samples is generated using a group of source domain (D s ) samples and a group of target domain (D t ) samples as inputs. The domain composed of the mixed samples is referred to as the mixed domain (D m ). To generate the mixed samples, for each sample x t (x t ∈ D t ) in the group of target domain samples, two samples x s,1 , x s,2 (x s,1 , x s,2 ∈ D s , y s,1 = y s,2 = y t ) are randomly selected from the group of source domain samples. Here, y s,1 = y s,2 = y tOne sample from the target domain and two samples from the source domain have the same label, that is, they belong to the same class. Next, an input addition mixing operation or an input difference mixing operation is randomly executed on these three samples with a probability of 50% to generate a group of mixed samples.

[0029] FIG. 3A is a diagram schematically showing an input addition mixing operation (operation) according to one embodiment. In the input addition mixing operation, the above three samples are mixed as follows.

Number

Number

[0030] Here, x m ∈ D m , λ1 and λ2 are hyperparameters for controlling the mixing ratio, lower_bound is the lower limit of the value, and upper_bound is the upper limit of the value. Note that the specific values of λ1 and λ2, as well as lower_bound and upper_bound, may be set according to the design requirements.

[0031] Note that the input addition mixing operation shown in FIG. 3A can transfer some information from the source domain to the target domain.

[0032] To further increase the number of samples similar to the target domain samples, an input difference mixing operation according to another embodiment shown in FIG. 3B may be employed. In the input difference mixing operation, these three samples are mixed as follows.

Number

Number

[0033] Here, x m ∈ Dm λ3 is a hyperparameter for controlling differential increase and decrease. As can be seen through experiments, better results can be obtained by setting λ3 to 1. Note that the value of λ3 is not limited to 1 and may be set to various values according to design requirements.

[0034] Note that when adding the sum or difference between two source domain samples x s,1 and x s,2 to the target domain sample x t directly, since the values of some dimensions of the mixed sample exceed the value range [lower_bound, upper_bound], it is necessary to explicitly constrain the values of each dimension of the mixed sample within this value range using the clip function. That is, when the value of a specific dimension is greater than upper_bound, the value is made equal to upper_bound, and when the value of a specific dimension is less than lower_bound, the value is made equal to lower_bound. The value of upper_bound is usually 1, and the value of lower_bound may be 0 or -1, depending on the regularization method used. Note that the specific values of upper_bound and lower_bound are not limited to the above values and may be set to various values according to design requirements.

[0035] Note that adding the difference between two source domain samples to one target domain sample may also be regarded as applying noise to the source domain samples. However, unlike conventional Gaussian noise, the noise based on the difference between source domain samples contains information about the corresponding classes in the source domain. By adding this differential noise to the samples in the target domain, the diversity of the samples in the source domain can be transferred to the target domain. Therefore, more samples similar to those in the target domain can be generated, and by using these samples for training, the performance of the model in the target domain can be improved.

[0036] Next, in step 102, a first loss function is constructed based on the sum of the intra-class distance and the inter-class distance between the samples in the source domain and the mixed samples. The first loss function minimizes the weighted average intra-class distance and maximizes the weighted average inter-class distance.

[0037] As described above, as shown in the following equations (5) and (6), CCSA treats all intra-class distances and inter-class distances equally, while d-SNE only considers the maximum intra-class distance and the minimum inter-class distance.

Number

Number

[0038] Here, for each target domain sample x j with respect to, d(x, x j ) represents the distance between the source domain sample x i in the feature space. Here, (Outside 1) TIFF0007711536000007.tif13170, (Outside 2) TIFF0007711536000008.tif17170, k represents a specific sample class. For example, when there are a total of 10 classes, the value of k is 0 to 9. Here, (Outside 3) TIFF0007711536000009.tif13170 represents a set composed of samples belonging to the same class as the target domain sample in the source domain, (Outside 4) TIFF0007711536000010.tif13170 represents a set of samples that do not belong to the same class as the target domain sample in the source domain. When the source domain sample and the target domain sample belong to the same class, that is, y i = y j in the case of, di = d(xi , x j ) is called the within-class distance. When the source domain sample and the target domain sample do not belong to the same class, that is, yi ≠ y j , in the case of d i ’ = m - d(x i , x j ) is called the between-class distance, where m is a predefined tolerance value, for example, 1. d1 > d2 > … > d n1 Arrange the within-class distances so that d1 > d2 > … > d n2 ’, where n1 represents the number of samples in the source domain that belong to the same class as the target domain sample. Arrange the between-class distances so that d1’ > d2’ > … > d

[0039] CCSA and d-SNE represent two extreme scenarios respectively. In order to better consider the influence of each adjacent sample in the feature space, in step 102, as shown in the following formula (7), a first loss function is constructed by performing a distance-based weighted average on the within-class distance and the between-class distance.

Equation

[0040] Here, (External 5) TIFF0007711536000012.tif25170, (External 6) TIFF0007711536000013.tif15170, (External 7) TIFF0007711536000014.tif42170, (External 8) TIFF0007711536000015.tif15170, where the weight α i,j is calculated using the softmax function for all within-class distances, and the weight β i,jIt is calculated using the softmax function for all inter-class distances.

[0041] For convenience of explanation, the description of feature extraction is omitted in this specification. Therefore, X used in the loss function (7) does not represent the sample itself, but represents the features extracted from the sample.

[0042] Note that the soft d-SNE method according to this embodiment minimizes the large intra-class distance and maximizes the small inter-class distance so as to optimize the performance of the model. Also, since it may not be appropriate to use a fixed tolerance value m to exclude inter-class distances larger than the tolerance value, the soft d-SNE method according to this embodiment uses the maximum intra-class distance ( (Outer 9) TIFF0007711536000016.tif18170) to further adaptively adjust the tolerance value m.

[0043] FIG. 4 is a diagram schematically showing the differences between the soft d-SNE method according to this embodiment and the CCSA and d-SNE methods. As can be seen from FIG. 4, the soft d-SNE method according to this embodiment minimizes the weighted average intra-class distance and maximizes the weighted average inter-class distance in order to better consider each adjacent sample in the feature space.

[0044] Next, in step 103, based on the sum of the first loss function and the cross-entropy loss function, the second loss function is constructed for the mixed domain composed of the source domain and the mixed samples respectively. Specifically, in this embodiment, the second loss functions of the source domain and the mixed domain are shown in the following equations (8) and (9).

Equation

Equation

[0045] Here, (External 10) TIFF0007711536000019.tif16170 and (External 11) TIFF0007711536000020.tif16170 represent the cross-entropy losses of the source domain and the mixed domain, respectively. Note that the value of the weight α may be set according to the design requirements.

[0046] Finally, in step 104, it is determined whether the neural network has converged using both the second loss function of the source domain and the second loss function of the mixed domain. If the neural network has not converged, the above steps 101 to 103 are repeatedly executed.

[0047] Figure 5 schematically shows the training process of a neural network that combines domain mixing and class alignment. In Figure 5, (External 12) TIFF0007711536000021.tif14170 and (External 13) TIFF0007711536000022.tif16170 represent the feature extractors of the source domain D s and the mixed domain D m respectively. These two feature extractors transform samples from the source domain and the mixed domain into the feature space. The parameters of these two feature extractors are θ s and θ m respectively. Until the neural network converges, equations (8) and (9) are alternately used to iteratively optimize and adjust the parameters of the neural networks (feature extractors and classifiers) of the source domain and the mixed domain.

[0048] Note that the source domain and the mixed domain may use two different feature extractors respectively, or may share the same feature extractor.

[0049] Note that the loss function L Soft d-SNEIt can perform class alignment on the samples of the source domain and the mixed domain in the feature space and endow the classes with discriminability.

[0050] FIG. 6 is a schematic diagram of training a neural network using the domain adaptation method according to the present disclosure. Note that before starting the training of the neural network, it is necessary to initialize its parameters. For the sake of convenience of explanation, the initialization process is omitted in FIG. 6.

[0051] First, in step 601, after initialization, a set of data is sampled from the source domain dataset and the target dataset respectively. The data includes two parts: a sample (X) and its corresponding label (Y). Since only a small number of samples are included in the target dataset, this set of sampled target domain samples contains many duplicate samples.

[0052] Next, in step 602, mixed samples are generated using the source domain data and the target domain data. In this embodiment, the method shown in FIG. 3A or FIG. 3B may be used to generate the mixed samples.

[0053] Next, using the respective feature extractors and classifiers of the source domain and the mixed domain, steps 6031 and 6032 of feature extraction and steps 6041 and 6042 of classification are performed respectively based on the source domain samples, the mixed domain samples, and their corresponding labels, and the respective classification cross-entropy losses of the source domain and the mixed domain are calculated. Note that the calculation of feature extraction, classification, and classification cross-entropy loss for the source domain samples and the mixed domain samples is realized using a conventional neural network architecture.

[0054] Next, in step 605, the class alignment loss is calculated. In this embodiment, based on the loss function (7), using the features and corresponding labels of the source domain samples and the mixed domain samples extracted by the respective feature extractors of the source domain and the mixed domain, the class alignment loss L Soft d-SNE may be calculated.

[0055] Next, in step 606, it is determined whether the neural network has converged. In this embodiment, based on the loss functions (8) and (9), it may be determined whether the neural network has converged. If it has converged, the training is terminated.

[0056] If not, in steps 6071 and 6072, based on the weighted sum of the cross-entropy loss and the class alignment loss obtained in the previous iteration, the gradients of the source domain and the mixed domain are calculated respectively.

[0057] Next, in steps 6081 and 6082, using the gradient, the parameters of the classifiers of the source domain and the mixed domain are updated respectively, and in steps 6091 and 6092, using the gradient, the parameters of the feature extractors of the source domain and the mixed domain are updated respectively. After updating the parameters, the next round of training iteration is performed.

[0058] According to the method 100 according to the embodiment, the target domain data can be expanded, the weighted average intra-class distance can be minimized, and the weighted average inter-class distance can be maximized, so that the influence of each adjacent sample in the feature space can be better considered, and thus the model can be further optimized. Table 1 below shows the classification accuracy of training the model using various numbers of target domain samples in the MNIST→MNIST-M digit recognition domain adaptation task of the method 100 according to the embodiment. Each experiment was repeated 3 times using different random seeds.

Table 1

[0059] Figure 7 shows the results of Table 1 in graph form. As can be seen from Figure 7, the method 100 (Soft d-SNE + domain mixing) according to the embodiment has a significantly improved classification accuracy compared to the d-SNE method.

[0060] The above method may be fully realized by a computer-executable program, or may be partially or fully realized using hardware and / or firmware. When realized by hardware and / or firmware, or when a computer-executable program is loaded into a hardware device capable of executing the program, a correction device for the recognition result of an image block described later is realized. The following omits the above-described detailed content and explains the outline of these devices. Note that these devices can execute the above method, but the above method is not limited to adopting the constituent parts of the devices described later or being executed by the constituent parts.

[0061] FIG. 8 is a block diagram of a domain adaptation apparatus 800 for a neural network according to one embodiment of the present disclosure. The apparatus 800 includes a mixing unit 801, a first construction unit 802, a second construction unit 803, and a determination unit 804. The mixing unit 801 mixes a first sample and a second sample from a source domain and a third sample from a target domain to obtain a mixed sample. The first sample, the second sample, the third sample, and the mixed sample belong to the same class. The first construction unit 802 constructs a first loss function based on the sum of the intra-class distance and the inter-class distance between the samples in the source domain and the mixed sample. The first loss function minimizes the weighted average intra-class distance and maximizes the weighted average inter-class distance. The second construction unit 803 constructs a second loss function for the mixed domain composed of the source domain and the mixed sample respectively based on the sum of the first loss function and the cross-entropy loss function. The determination unit 804 determines whether the neural network has converged using both the second loss function of the source domain and the second loss function of the mixed domain. If it is determined that the neural network has not converged, the mixing unit 801, the first construction unit 802, and the second construction unit 803 repeatedly execute their respective processes.

[0062] The domain adaptation apparatus 800 for the neural network shown in FIG. 8 corresponds to the method 100 shown in FIG. 1. Therefore, the details of each part in the domain adaptation apparatus 800 for the neural network have already been described in detail in the description of the optimization method 100 for object prediction in FIG. 1, and the description thereof is omitted here.

[0063] The above-described processing and apparatus may be implemented by software and / or firmware. When implemented by software and / or firmware, a program configuring software for implementing the above method may be installed in a computer having a dedicated hardware configuration (e.g., the general-purpose personal computer 900 shown in FIG. 9) from a storage medium or a network, and the computer can execute various functions and the like when various programs are installed.

[0064] FIG. 9 is a block diagram showing an exemplary configuration of a general-purpose personal computer capable of realizing the method and / or apparatus according to an embodiment of the present disclosure. In FIG. 9, a central processing unit (CPU) 901 executes various processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage unit 908 into a random access memory (RAM) 903. The RAM 903 stores data necessary for the CPU 901 to execute various processes as needed. The CPU 901, ROM 902, and RAM 903 are connected to each other via a bus 904. An input / output interface 905 is also connected to the bus 904.

[0065] An input unit 906 (including a keyboard, a mouse, etc.), an output unit 907 (including a display, e.g., a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.), a storage unit 908 (including, e.g., a hard disk, etc.), and a communication unit 909 (including, e.g., a network interface card, e.g., a LAN card, a modem, etc.) are connected to the input / output interface 905. The communication unit 909 executes communication processing via a network, e.g., the Internet. Optionally, a driver 910 may be connected to the input / output interface 905. The removable medium 911 is, e.g., a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., and is set up in the driver 910 as needed, and a computer program read from it is installed in the storage unit 908 as needed.

[0066] When implementing the above processing by software, a program constituting the software is installed via a network, such as the Internet, or a storage medium, such as a removable medium 911.

[0067] Note that these storage media are not limited to the removable medium 911 shown in FIG. 9 that stores a program and provides the program to the user separately from the device. The removable medium 911 includes, for example, a magnetic disk (including a floppy disk (registered trademark)), an optical disk (including a compact disc-read only memory (CD-ROM) and a digital versatile disc (DVD)), a magneto-optical disk (mini disc (MD) (registered trademark)), and a semiconductor memory. Alternatively, the storage medium may be a ROM 902, a hard disk included in the storage unit 908, etc., which stores a program and is provided to the user together with the device including them.

[0068] The present disclosure further provides a corresponding computer program code, a computer program product in which machine-readable instruction codes are stored. When the instruction codes are read and executed by a machine, the method according to the above-described embodiment of the present disclosure can be executed.

[0069] Accordingly, the present disclosure further includes a storage medium in which a program product including machine-readable instruction codes is recorded. The storage medium includes, but is not limited to, a floppy disk, an optical disk, a magneto-optical disk, a memory card, a memory stick, etc.

[0070] Regarding the embodiments including the above-described respective examples, the following additional remarks are further disclosed. (Additional Remark 1) A method for domain adaptation for a neural network, comprising: A mixing step of obtaining a mixed sample by mixing a first sample and a second sample from a source domain and a third sample from a target domain, wherein the first sample, the second sample, the third sample, and the mixed sample belong to the same class; A first construction step of constructing a first loss function based on the sum of the intra-class distance and the inter-class distance between the samples in the source domain and the mixed sample, wherein the first loss function minimizes the weighted average intra-class distance and maximizes the weighted average inter-class distance; A second construction step of constructing a second loss function for each of the mixed domains composed of the source domain and the mixed sample based on the sum of the first loss function and the cross-entropy loss function; A determination step of determining whether the neural network has converged using both the second loss function of the source domain and the second loss function of the mixed domain, and if the neural network has not converged, repeatedly executing the mixing step, the first construction step, and the second construction step. A method comprising: (Appendix 2) The method according to claim 1 of Appendix 1, wherein the mixing step includes a step of adding the weighted value of the third sample to the weighted value of the difference between the first sample and the second sample. (Appendix 3) The method further includes a step of adding the third sample to the weighted value of the average value of the first sample and the second sample. The method according to Appendix 2. (Appendix 4) The method according to Appendix 3, wherein the mixing step further includes a step of setting an upper limit and a lower limit and constructing the mixed domain based on the mixed sample between the upper limit and the lower limit. (Appendix 5) The method according to Appendix 1, wherein the weights of the weighted average are calculated based on the distance using a normalized exponential function. (Appendix 6) The first loss function is constructed to subtract the inter-class distance from the sum of a fixed tolerance value and the maximum intra-class distance, The fixed tolerance value is the method described in Appendix 5, which is used to remove the inter-class distance whose length exceeds the tolerance value. (Appendix 7) The method according to any one of Appendices 1 to 6, wherein the distance is the Euclidean distance. (Appendix 8) When the neural network has not converged, further including the step of updating the parameters of the neural network using the second loss function of the source domain and the second loss function of the mixed domain, the method according to any one of Appendices 1 to 6. (Appendix 9) The method according to any one of Appendices 1 to 6, wherein the second loss function is constructed based on the argmin function. (Appendix 10) In the argmin function, a first weight is assigned to the first loss function and a second weight is assigned to the cross-entropy loss function, The method according to Appendix 9, wherein the sum of the first weight and the second weight is 1. (Appendix 11) An apparatus for domain adaptation for a neural network, A mixing unit that mixes a first sample and a second sample from a source domain and a third sample from a target domain to obtain a mixed sample, wherein the first sample, the second sample, the third sample, and the mixed sample belong to the same class, the mixing unit, A first construction unit that constructs a first loss function based on the sum of the intra-class distance and the inter-class distance between the sample in the source domain and the mixed sample, wherein the first loss function minimizes the weighted average intra-class distance and maximizes the weighted average inter-class distance, the first construction unit, A second construction unit that constructs a second loss function for each of the source domain and the mixed domain composed of the source domain and the mixed samples based on the sum of the first loss function and the cross-entropy loss function; A determination unit that determines whether or not the neural network has converged using both the second loss function of the source domain and the second loss function of the mixed domain; and An apparatus, wherein when the determination unit determines that the neural network has not converged, the mixing unit, the first construction unit, and the second construction unit repeatedly execute their respective processes. (Appendix 12) The apparatus according to Appendix 11, wherein the mixing unit adds the weighted value of the third sample to the weighted value of the difference between the first sample and the second sample. (Appendix 13) The apparatus according to Appendix 12, wherein the mixing unit adds the third sample to the weighted value of the average value of the first sample and the second sample. (Appendix 14) The apparatus according to Appendix 13, wherein the mixing unit sets an upper limit and a lower limit, and constructs the mixed domain based on the mixed samples between the upper limit and the lower limit. (Appendix 15) The apparatus according to Appendix 11, wherein the weight of the weighted average is calculated based on the distance using a normalized exponential function. (Appendix 16) The first loss function is constructed to subtract the inter-class distance from the sum of a fixed tolerance value and the maximum intra-class distance, The fixed tolerance value is used to remove the inter-class distance whose length exceeds the tolerance value. The apparatus according to Appendix 15. (Appendix 17) The apparatus according to any one of Appendices 11 to 16, wherein when the neural network has not converged, the apparatus updates the parameters of the neural network using the second loss function of the source domain and the second loss function of the mixed domain. (Appendix 18) The second loss function is the device according to any one of Appendices 11 to 16, which is constructed based on the argmin function. (Appendix 19) In the argmin function, a first weight is assigned to the first loss function, and a second weight is assigned to the cross-entropy loss function. The device according to Appendix 18, wherein the sum of the first weight and the second weight is 1. (Appendix 20) A computer-readable storage medium storing a program, when the program is executed by a processor, A mixing step of obtaining a mixed sample by mixing a first sample and a second sample from a source domain and a third sample from a target domain, wherein the first sample, the second sample, the third sample, and the mixed sample belong to the same class; the mixing step, A first construction step of constructing a first loss function based on the sum of the intra-class distance and the inter-class distance between the sample in the source domain and the mixed sample, wherein the first loss function minimizes the weighted average intra-class distance and maximizes the weighted average inter-class distance; the first construction step, A second construction step of respectively constructing a second loss function for a mixed domain composed of the source domain and the mixed sample based on the sum of the first loss function and the cross-entropy loss function; A determination step of determining whether the neural network has converged using both the second loss function of the source domain and the second loss function of the mixed domain, and if the neural network has not converged, repeatedly executing the mixing step, the first construction step, and the second construction step; a storage medium for executing.

[0071] Note that the terms "comprising", "having", or any other variation thereof are not exclusively limited to including, and a process, method, article, or apparatus that includes a series of elements includes not only these elements but also other elements not explicitly listed or elements inherent to this process, method, article, or apparatus. Also, unless further restricted, elements limited by the term "comprising one..." do not exclude the presence of other identical elements in the process, method, article, or apparatus that includes such an element.

[0072] The foregoing has described the preferred embodiments of the present disclosure with reference to the drawings, but the above embodiments and examples are illustrative and not restrictive. Those skilled in the art may make various modifications, improvements, and equivalent changes to the present disclosure within the spirit and scope of the claims. Making these modifications, improvements, or equivalent changes is included within the protection scope of the present disclosure.

Claims

**Claim 1** A method for domain adaptation for a neural network, executed by a computer, comprising: a mixing step of obtaining a mixed sample by mixing a first sample and a second sample from a source domain and a third sample from a target domain, wherein the first sample, the second sample, the third sample, and the mixed sample belong to the same class; a first construction step of constructing a first loss function based on the sum of the intra-class distance and the inter-class distance between the samples in the source domain and the mixed sample, and calculating a class alignment loss based on the first loss function, wherein the first loss function minimizes the weighted average intra-class distance and maximizes the weighted average inter-class distance; a second construction step of constructing a second loss function for each of a mixed domain composed of the source domain and the mixed sample based on the sum of the class alignment loss and a cross-entropy loss function; a method of repeatedly executing the mixing step, the first construction step, and the second construction step so as to update parameters of the neural network using the second loss function of the source domain and the second loss function of the mixed domain when both the second loss function of the source domain and the second loss function of the mixed domain have not converged. **Claim 2** The method according to claim 1, wherein the mixing step includes a step of adding a weighted value of the third sample to a weighted value of a difference between the first sample and the second sample. **Claim 3** The method according to claim 2, wherein the mixing step further includes a step of adding the third sample to a weighted value of an average value of the first sample and the second sample. The method according to claim 2. **Claim 4** The method according to claim 3, wherein the mixing step further includes a step of setting an upper limit and a lower limit and constructing the mixed domain based on the mixed sample between the upper limit and the lower limit. **Claim 5** The method according to claim 1, wherein the weights of the weighted average are calculated based on distances using a normalized exponential function. **Claim 6** The first loss function is constructed to subtract the inter-class distance from the sum of a fixed tolerance value and the maximum intra-class distance. The method according to claim 5, wherein the tolerance value of the fixation is used to remove the inter-class distance whose length exceeds the tolerance value.

7. The second loss function is constructed based on the argmin function, in the argmin function, a first weight is assigned to the first loss function, and a second weight is assigned to the cross-entropy loss function, The method according to any one of claims 1 to 6, wherein the sum of the first weight and the second weight is 1.

8. An apparatus for domain adaptation for a neural network, a mixing unit that obtains a mixed sample by mixing a first sample and a second sample from a source domain and a third sample from a target domain, wherein the first sample, the second sample, the third sample, and the mixed sample belong to the same class, the mixing unit; a first construction unit that constructs a first loss function based on the sum of the intra-class distance and the inter-class distance between the sample in the source domain and the mixed sample, and calculates the class alignment loss based on the first loss function, wherein the first loss function minimizes the weighted average intra-class distance and maximizes the weighted average inter-class distance, the first construction unit; a second construction unit that constructs second loss functions for the mixed domain composed of the source domain and the mixed sample respectively based on the sum of the class alignment loss and the cross-entropy loss function; The apparatus, wherein when both the second loss function of the source domain and the second loss function of the mixed domain have not converged, the mixing unit, the first construction unit, and the second construction unit repeatedly execute their respective processes so as to update the parameters of the neural network using the second loss function of the source domain and the second loss function of the mixed domain.

9. A computer-readable storage medium storing a program, wherein when the program is executed by a processor, a mixing step of obtaining a mixed sample by mixing a first sample and a second sample from a source domain and a third sample from a target domain, wherein the first sample, the second sample, the third sample, and the mixed sample belong to the same class, the mixing step; A first construction step of constructing a first loss function based on the sum of the intra-class distance and the inter-class distance between the samples in the source domain and the mixed samples, and calculating a class alignment loss based on the first loss function, wherein the first loss function minimizes the weighted average intra-class distance and maximizes the weighted average inter-class distance, the first construction step; A second construction step of constructing second loss functions for the mixed domain composed of the source domain and the mixed samples respectively based on the sum of the class alignment loss and the cross-entropy loss function, including a method for domain adaptation for a neural network; A storage medium that repeatedly executes the mixing step, the first construction step, and the second construction step so as to update the parameters of the neural network using the second loss function of the source domain and the second loss function of the mixed domain when both the second loss function of the source domain and the second loss function of the mixed domain have not converged.

Images