Information processing device, information processing method, and computer program

Abstract

Provided is an information processing device for performing model learning adaptive to a domain shift.　This information processing device comprises: an acquisition unit that acquires a labeled dataset composed of labeled data of a first domain, a first unlabeled dataset composed of unlabeled data of the first domain, and a second unlabeled dataset composed of unlabeled data of a second domain different from the first domain; and a training unit for training a model on the basis of a loss function calculated using the labeled dataset, the first unlabeled dataset, and the second unlabeled dataset.

Description

Information processing device, information processing method, and computer program

[0001] The technology disclosed in this specification (hereinafter referred to as "the present disclosure") relates to an information processing device, an information processing method, and a computer program that perform processing related to model learning.

[0002] In order to meet performance requirements in AI (Artificial Intelligence) development, supervised learning using a sufficient amount of training data is required. However, collecting real data and annotating the collected data requires a huge amount of time and cost. Furthermore, there is a known problem of the distribution (domain) changing between the training data of an AI system and the data used during operation. This change in domain is called a "domain shift," and the difference that occurs between domains is called a "domain gap." For example, a domain shift can occur due to differences in data collection equipment, settings, and environmental conditions between AI training and operation. There is a concern that the performance of the trained model in the domain after the shift will deteriorate.

[0003] For example, when using a deep-learned object detection model in the mobility field, if the equipment (vehicles, cameras, etc.), settings, and environmental conditions used during model training differ from those during operation when the trained model is used, a domain shift will occur, raising concerns that the performance of the object detection model will deteriorate during operation due to a domain gap.

[0004] The domain gap can be resolved by retraining the model in the post-shift domain. However, collecting data again and recreating a dataset with correct labels every time the collection equipment, settings, or environmental conditions change requires a huge amount of time and cost. Depending on the domain and learning task, retraining a model following a domain shift often requires highly specialized knowledge. In addition, there are many cases where datasets that have become unusable due to a domain shift are left lying dormant on servers, etc., and utilizing these accumulated datasets is also an important issue.

[0005] To reduce the costs of recollecting and annotating data due to a domain shift, a technique called "domain adaptation" is widely used. Domain adaptation generally overcomes the domain gap by utilizing a labeled dataset before the domain change and unlabeled data after the domain shift.

[0006] For example, a domain adaptation system has been proposed that classifies training data of a target domain by using a group of source models each trained to classify data of a source domain, generates pseudo labels indicating the category of the training data by integrating the classification results of the source models based on the respective weights of the source models for a given data category, trains a target model of the target domain based on the training data and the pseudo labels, and trains the target model to classify data from the target domain (see Patent Document 1).

[0007] U.S. Patent Application Publication No. 2023 / 0169389

[0008] Yu-Jhe Li et al. "Cross-Domain Adaptive Teacher for Object Detection" (2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR)

[0009] An object of the present disclosure is to provide an information processing device, an information processing method, and a computer program that perform processing for learning a model that adapts to a domain shift.

[0010] The present disclosure has been made in consideration of the above-mentioned problems, and a first aspect thereof is an information processing device including: an acquisition unit that acquires a labeled dataset consisting of labeled data of a first domain, a first unlabeled dataset consisting of unlabeled data of the first domain, and a second unlabeled dataset consisting of unlabeled data of a second domain different from the first domain; and a learning unit that performs model learning based on a loss function calculated using the labeled dataset, the first unlabeled dataset, and the second unlabeled dataset.

[0011] The learning unit learns the model based on a loss function calculated using labels of labeled data in the first domain, a loss function calculated using first pseudo labels generated from unlabeled data in the first domain, and a loss function calculated using second pseudo labels generated from unlabeled data in the second domain.

[0012] In the information processing device according to the first aspect, the first pseudo label is generated from unlabeled data of the first domain using a first teacher model, and the second pseudo label is generated from unlabeled data of the second domain using a second teacher model, and the first teacher model and the second teacher model may be used, which reflect learning parameters of the models according to an exponential moving average model.

[0013] A second aspect of the present disclosure is an information processing method including: an acquisition step of acquiring a labeled dataset consisting of labeled data of a first domain, a first unlabeled dataset consisting of unlabeled data of the first domain, and a second unlabeled dataset consisting of unlabeled data of a second domain different from the first domain; and a training step of training a model based on a loss function calculated using the labeled dataset, the first unlabeled dataset, and the second unlabeled dataset.

[0014] A third aspect of the present disclosure is a computer program written in a computer-readable format to cause a computer to function as: an acquisition unit that acquires a labeled dataset consisting of labeled data of a first domain, a first unlabeled dataset consisting of unlabeled data of the first domain, and a second unlabeled dataset consisting of unlabeled data of a second domain different from the first domain; and a learning unit that trains a model based on a loss function calculated using the labeled dataset, the first unlabeled dataset, and the second unlabeled dataset.

[0015] A computer program according to a third aspect of the present disclosure defines a computer program written in a computer-readable format to perform predetermined processing on a computer. The computer program can be provided to a computer capable of executing various program codes in a computer-readable format via a storage medium or communication medium, such as an optical disk, a magnetic disk, or a semiconductor memory, or a communication medium such as a network. By installing the computer program according to the third aspect of the present disclosure on a computer via any of these media, a cooperative effect is exerted on the computer, and the same effects as those of the information processing device according to the first aspect of the present disclosure can be obtained.

[0016] According to the present disclosure, it is possible to provide an information processing device, an information processing method, and a computer program that utilize accumulated data sets of a source domain to train a model adapted to a target domain at low cost.

[0017] It should be noted that the effects described in this specification are merely examples, and the effects brought about by the present disclosure are not limited to these. Furthermore, the present disclosure may also bring about additional effects in addition to the effects described above.

[0018] Further objects, features, and advantages of the present disclosure will become apparent from the following detailed description based on the following embodiments and the accompanying drawings.

[0019] FIG. 1 is a diagram showing an example functional configuration of a learning system to which the present disclosure is applied. FIG. 2 is a diagram showing the functional configuration of a learning system 200. FIG. 3 is a diagram showing the internal configuration of a pre-learning processing unit 200. FIG. 4 is a diagram showing a hardware configuration example of an information processing device 2000. FIG. 5 is a flowchart showing the steps of model learning performed in the learning system 100. FIG. 6 is a diagram showing the framework of an adaptive teacher. FIG. 7 is a diagram showing an example hardware configuration of an information processing device 2000.

[0020] Hereinafter, embodiments of the present disclosure will be described in the following order with reference to the drawings.

[0021] A. Overview A-1. Adaptive Teacher A-2. Problems A-3. Framework according to the present disclosure B. Examples B-1. Configuration example B-2. Operation example C. Configuration of information processing device

[0022] A. Overview A-1. Adaptive Teacher In domain adaptation, generally, labeled data from a domain before the change and unlabeled data after the shift are used to train a model adapted to the domain after the shift. In this specification, the domain before the shift is referred to as the "source domain," and the domain after the shift is referred to as the "target domain." The purpose of domain adaptation is to bridge the domain gap between the source domain and the target domain and improve the performance of a trained model in the target domain.

[0023] On the other hand, a teacher-student framework, in which a student model is monitored by pseudo-labels from a teacher model, is known as an effective semi-supervised learning method. Generally, a model whose learning parameters are updated during learning is called a "student model," while a model that only performs inference on a target with its learning parameters fixed and not updated is called a "teacher model." However, the teacher-student framework suffers from the problem of domain shift, which can cause many low-quality pseudo-labels to be generated from the teacher model, resulting in a degradation of the performance of the trained model (student model) across domains. Therefore, adaptive teaching (AT) has been proposed to reduce pseudo-label errors in domain adaptation (see Non-Patent Document 1). AT is a technology that uses labeled data from the source domain and unlabeled data from the target domain to train a model that is invariant across domains.

[0024] The framework of the adaptive teacher is shown in Figure 6. The illustrated AT framework 600 is intended to be applied to the task of object detection in image data and consists of two modules: a target domain-specific teacher model 610 and a student model 620 that is applied across the source and target domains (i.e., cross-domain).

[0025] The teacher model 610 has a feature extraction unit (Feature Encoder) 611 and a detection unit (Detector) 612. Similarly, the student model 620 has a feature extraction unit 621 and a detection unit 622. The AT framework 600 also includes a domain classification unit (Discriminator) 630 with a gradient reverse layer (GRL) 631 to align the data distribution across the domain in the student model 620.

[0026] A dataset consisting of unlabeled data in the target domain is input to the teacher model 610, while both the dataset consisting of unlabeled data in the target domain and the dataset consisting of labeled data in the source domain are input to the student model 620. Then, in the AT framework 600, model training is performed using two training streams: mutual training and adversarial training of the teacher model 610 and the student model 620.

[0027] First, pre-training is performed using labeled data from the source domain to initialize the feature extractor and detector. During the mutual training phase, the initialized feature extractor and detector are replicated in both the teacher model 610 and the student model 620. The teacher model 610 generates pseudo labels from unlabeled data from the target domain to train the student model 620. Meanwhile, updated training parameters for the student model 620 are reflected in the teacher model 610 according to an exponential moving average. By repeating this process, the pseudo labels for training the student model 620 are improved. Furthermore, the GRL 631 and the domain classifier 630 are used to adaptively train the student model 620 to be invariant across the source and target domains (i.e., to perform training so that domains cannot be distinguished and domain-independent features can be extracted). This reduces domain shift in the student model 620 and enables the teacher model 610 to generate higher-quality pseudo labels.

[0028] The general learning flow of the AT framework 600 is as follows.

[0029] (Step 1) Labeled data of the source domain is input to the student model 620, and unlabeled data of the target domain is input to the teacher model 610 and the student model 620 simultaneously.

[0030] (Step 2) On the student model 620 side, for each input of labeled data from the source domain and unlabeled data from the target domain, the feature extraction unit 621 outputs a feature map as an intermediate output, and the detection unit 622 outputs the final inference result.

[0031] (Step 3) The feature maps of the labeled data of the source domain and the unlabeled data of the target domain output as intermediate outputs are input to the domain classification unit 630 via the GRL 631. The domain classification unit 630 identifies the domain of the input data corresponding to each feature map.

[0032] (Step 4) Pseudo labels for unlabeled data in the target domain are generated from the inference results of the teacher model 610 and assigned as correct labels for the inference results of the student model 620 for the unlabeled data in the target domain.

[0033] (Step 5) A loss function L is calculated based on the correct label for the inference results of the student model 620 on the labeled data in the source domain. sup , the loss function L with pseudo-labels for the inference results of the student model 620 on unlabeled data in the target domain unsup , and a loss function L for the domain classification result of each feature map by the domain classification unit 630. dts Calculate each.

[0034] (Step 6) Based on the total loss function obtained by weighting and adding the loss functions calculated above, the learning parameters of the student model 620 are optimized and updated.

[0035] (Step 7) The learning parameters of the student model 620 are reflected in the teacher model 610 according to the exponential moving average (EMA).

[0036] The learning parameters of the teacher model 610 are θ t , the learning parameters of the student model 620 are θ s , and the shift coefficient is α, in step 7, the learning parameter θ of the teacher model 610 is calculated according to the following equation (1): t Here, the shift coefficient α is a value in the range of 0 to 1. The larger the shift coefficient α, the slower the update speed of the teacher model 610 becomes, and the smaller the shift coefficient α, the greater the contribution of the student model 620 becomes.

[0037] θ t =←αθ t+(1-α)θ s …(1)

[0038] A-2. Problems The AT framework 600 described in the above section A-1 has a problem in that the performance of domain adaptation that can be realized is restricted by the amount of labeled data on the source domain side.

[0039] In the AT framework 600, labeled data from the source domain and unlabeled data from the target domain are simultaneously input to a student model 620, and pseudo labels generated by a teacher model 610 are assigned to the unlabeled data from the target domain, thereby enabling model learning for the target domain as well. Here, the accuracy of the pseudo labels is highly dependent on the accuracy of supervised learning of the student model 620 using labeled data from the source domain. Therefore, in order to bridge the domain gap between the source domain and the target domain, it is necessary to ensure that the amount of labeled data from the source domain is sufficient relative to the amount of unlabeled data from the target domain.

[0040] On the other hand, to train a model that satisfies the performance requirements for the target domain through domain adaptation, it is necessary to secure a sufficient amount of unlabeled data for the target domain, because in order to improve model performance for the target domain, it is necessary to reflect as much target domain knowledge as possible in the model.

[0041] Therefore, in order to improve the performance of a model through domain adaptation, annotation work is required to add labels to data in the source domain or target domain, which poses a problem of increasing the cost of securing training data.In actual AI development processes, previously acquired training data is often used as source domain data, so the scope of application of domain adaptation is limited by the annotation resources on the source domain side.

[0042] A-3. Framework of the Present Disclosure The main purpose of the present disclosure is to improve the performance of domain adaptation and expand the scope of domain application without increasing the annotation cost of labeling additional data. The present disclosure can be implemented using, for example, a teacher-student framework, but its main feature is that it utilizes unlabeled data from the source domain.

[0043] 1 shows an example of the configuration of a domain adaptation (DA) framework to which the present disclosure is applied. The illustrated learning system 100 is composed of three modules: a first teacher model 110, a second teacher model 120, and a student model 130.

[0044] The first teacher model 110 includes a feature extractor 111 that extracts features of input data (e.g., a feature map of an input image) and a detector 112 that detects labels corresponding to the extracted features. Similarly, the second teacher model 120 includes a feature extractor 121 and a detector 122, and the student model 130 includes a feature extractor 131 and a detector 132. The learning system 100 further includes a domain discriminator 140 with a GRL 141 for adaptive learning that is consistent across source and target domains.

[0045] The first teacher model 110 generates first pseudo labels from unlabeled data of the source domain (more precisely, the first pseudo label assignment unit 113 assigns the estimation results by the first teacher model 110 to the first pseudo labels). The second teacher model 120 generates second pseudo labels from unlabeled data of the target domain (more precisely, the second pseudo label assignment unit 123 assigns the estimation results by the second teacher model 120 to the second pseudo labels). The first pseudo labels and the second pseudo labels function as pseudo-ground-truth labels for the results estimated by the student model 130 from the corresponding unlabeled data of the source domain and the unlabeled data of the target domain, respectively. Details of the first pseudo label assignment unit 113 and the second pseudo label assignment unit 123 will be described later.

[0046] Model learning is performed by simultaneously inputting three sets of data into the student model 130: labeled data from the source domain, unlabeled data from the target domain, and further unlabeled data from the source domain. In the student model 130, for these three inputs, the feature extraction unit 131 outputs intermediate output features, and the detection unit 132 outputs final inference results. The domain classification unit 140 classifies the corresponding domain of the input data based on each feature (however, the code has already been optimized by the GRL 141).

[0047] Next, the loss function L is calculated based on the correct label for the inference result of the student model 130 on the labeled data of the source domain. sup , the loss function L with the first pseudo label for the inference result of the student model 130 on the unlabeled data of the source domain uns , the loss function L with the second pseudo label for the inference result of the student model 130 on the unlabeled data of the target domain unt , a loss function L for the domain classification result of each feature by the domain classification unit 140 domains_t are calculated. Any optimization function can be used as the loss function depending on the desired task. For example, the loss function may be Cross-Entropy Loss or Focal Loss.

[0048] Then, each loss function L calculated as above sup , L uns , L unt , L domains_t The total loss function L is calculated by adding the weighted sum as shown in the following equation (2). total The learning parameters of the student model 130 are optimized and updated based on the above equation (2). Note that w1, w2, w3, and w4 in the following equation (2) are loss weights for the individual loss functions, and a large value is assigned to the loss weight of the loss function whose contribution is desired to be increased.

[0049] L total = w1L sup +w2L uns +w3L unt +w4L domains_t …(2)

[0050] The learning parameters of the student model 130 thus optimized and updated are reflected in the first teacher model 110 and the second teacher model 120 according to the exponential moving average (EMA). In this embodiment, the first exponential moving average model (source EMA: EMA_s) defining the first teacher model 110 and the second exponential moving average (target EMA: EMA_t) defining the second teacher model 120 are independent. Here, the learning parameter θ of the first teacher model 110 is t1 , the learning parameters θ of the second teacher model 120. t2 , and the learning parameters of the student model 130 are s , the learning parameters θ of the first teacher model 110 t1 The shift coefficient when updating is α1, and the learning parameter θ of the second teacher model 120 is t2 When the shift coefficient at the time of updating is α2, the learning parameters θ of the first teacher model 110 and the second teacher model are t1 , θ t2 The update equations for the second teacher model 120 are expressed as equations (3) and (4), respectively. Making the second teacher model 120 an exponential moving average model independent of the exponential moving average model of the first teacher model 110 corresponds to setting different values ​​for the moving coefficients α1 and α2.

[0051] θ t1 =←α1θ t1 + (1-α1) θ s …(3) θ t2 =←α2θ t2 + (1-α2) θ s …(4)

[0052] In general, the size of the domain gap between unlabeled data in the source domain and labeled data in the source domain is smaller than the domain gap between unlabeled data in a target domain different from the source domain and labeled data in the source domain. By processing the unlabeled data in the source domain using the first teacher model 110 independent of the second teacher model 120 for processing the unlabeled data in the target domain, it becomes possible to individually control the contribution of the unlabeled data in the source domain and the unlabeled data in the target domain to the learning of the student model 130.

[0053] The features of the learning system 100 and its learning flow according to the present disclosure will be summarized below.

[0054] In the learning system 100, the model is trained while assigning pseudo labels that function as correct labels not only to unlabeled data in the target domain but also to unlabeled data in the source domain, which leads to improved accuracy of the pseudo labels for the target domain.

[0055] By learning the unlabeled source domain using the first pseudo label for the unlabeled data in the source domain, it is possible to expand the common knowledge between the source domain and the target domain. In particular, in the early stage of learning when the learning accuracy for the target in a domain different from the source is not stable, the first pseudo label for the unlabeled data in the source domain acts as a function to efficiently bridge the domain gap between the source domain and the target domain.

[0056] In the learning system 100, the feature quantities intermediately output from the feature extraction unit 131 for the above three inputs are input to the domain classification unit 140, and the loss function L is calculated in a direction to fill the difference between the inputs. domains_tis optimized. In this case, GRL has a function of inverting the sign of the gradient at backward, and is used to replace processes that are optimized in different directions with processes that are optimized in a single direction. The learning system 100 is configured to input a new feature map of unlabeled data from the source domain into the domain classification unit 140, so that not only is the domain gap between the source domain and the target domain taken into consideration, but also the inherent domain gap that exists within the same source domain can be reflected in the learning. In particular, by taking the feature map of unlabeled data from the source domain into consideration in advance in the pre-learning phase before reflecting the pseudo labels in the learning, the accuracy of the pseudo labels in the learning process can be improved.

[0057] In the learning system 100, using unlabeled data from the source domain for domain adaptation also contributes to the learning stability of the entire domain adaptation. Common domain adaptation methods (e.g., see Section A-1 above) have issues with learning stability because they perform learning using data input from different domains or inaccurate pseudo labels. In contrast, in the learning system 100 to which the present disclosure is applied, adding unlabeled data from the source domain to domain adaptation learning makes it possible to incorporate features that complement the domain gap between the source domain and the target domain into learning, thereby improving domain complementation performance and the quality of pseudo labels. As a result, the burden of tuning hyperparameters according to the input data and model is reduced, enabling stable domain adaptation learning. In particular, in use cases where the amount of data in the source domain dataset cannot be secured sufficiently for the target domain, the contribution of a dataset consisting of unlabeled data from the source domain is significant, thereby easing constraints imposed by the amount of labeled data from the source domain.

[0058] The learning system 100 effectively utilizes source domain data, regardless of whether it is labeled or not. Furthermore, since unlabeled source domain data is used for domain adaptation learning without additional labeling, the time and cost of annotation work can be reduced.

[0059] B. Example Next, an example of the domain adaptation technology according to the present disclosure will be described.

[0060] B-1. Configuration Example Figure 2 shows the functional configuration of a learning system 100 according to one embodiment of the present disclosure. The illustrated learning system 100 is roughly divided into a pre-learning processing unit 200 and a domain adaptation learning unit 300. The learning system 100 applies domain adaptation according to the present disclosure to train a model with a wide range of domain applicability. The learning system 100 also includes a user interface (UI) unit 400 that allows a user to operate the learning system 100. The UI unit 400 is composed of various input and output devices for user operation.

[0061] The learning system 100 is actually implemented by linking together one or more information processing devices such as a personal computer (PC). The learning system 100 and the UI unit 400 may be configured as a single information processing device, or may be configured as separate information processing devices. Alternatively, the UI unit 400 may be an information terminal such as a PC or tablet that the user has at hand, and the learning system 100 may be a cloud service accessed via a network such as the Internet.

[0062] The learning system 100 and each unit included in the UI unit 400 will be described in detail below.

[0063] Pre-training Processing Unit: The pre-training processing unit 200 performs pre-training processing on the student model (pre-trained) 201 to enable pseudo-labeling of unlabeled data in the source domain and unlabeled data in the target domain. To avoid confusion, Fig. 2 shows a simplified representation of the pre-training processing unit 200. Fig. 3 shows the internal configuration of the pre-training processing unit 200 in more detail.

[0064] The student model 201 may be a model that has been pre-trained using any other dataset before the pre-training process. Furthermore, a learning model having any structure depending on the desired task can be used for the student model 201. For example, the student model 201 may be YOLO (You Only Look Once) used for object detection tasks, U-Net used for segmentation tasks, or the Vision Transformer used for various tasks.

[0065] The pre-learning processing unit 200 receives three datasets as inputs to the student model 201 from the respective dataset input units 401 to 403 of the UI unit 400: a dataset consisting of labeled data from the source domain, a dataset consisting of unlabeled data from the source domain, and a dataset consisting of unlabeled data from the target domain. Similar to a typical learning model, the student model 201 is composed of a feature extraction unit and a detection unit 202. For convenience, the feature extraction unit is not shown in FIG. 3 . It is assumed that the feature extraction unit is included within the block of the student model 201. The feature extraction unit of the student model 201 outputs each feature for the above three input data. Furthermore, the detection unit 202 inputs the feature of the labeled data from the source domain from the feature extraction unit of the student model 201 and outputs an inference result for the input feature. The loss function calculation unit 204 then calculates a loss function for the correct label of the output of the detection unit 202 (the inference result of the labeled data from the source domain by the student model 201), similar to ordinary supervised learning. The loss function here is used for pre-training the student model 201 so that it can correctly infer data from the source domain. The loss function calculation unit 204 can use any optimization function as the loss function depending on the desired task. For example, the loss function may be a cross-entropy loss or a focal loss.

[0066] Furthermore, each feature extracted from the above three pieces of input data by the feature extraction unit of the student model 201 is also input to the GRL-added domain classification unit 203. Then, the loss function calculation unit 204 calculates a loss function for the domain classification result of each feature by the domain classification unit 203. The loss function here is used for learning by the feature extraction unit of the student model 201 to acquire domain-invariant features (i.e., learning to enable extraction of domain-independent features that cannot distinguish between domains).

[0067] In the pre-learning processing unit 200, pre-learning is recursively repeated until a predetermined number of learning loops 206 required for pre-learning are completed. Then, the output unit 205 outputs the learning parameters of the student model 201 for which pre-learning has been completed to the subsequent domain adaptation learning unit 300.

[0068] Domain adaptation learning unit: The domain adaptation learning unit 300 performs domain adaptation processing to reflect knowledge of the target domain in the learning model pre-trained by the pre-training processing unit 200. The main feature of the domain adaptation learning unit 300 is that it performs domain adaptation learning using unlabeled data from the source domain in addition to labeled data from the source domain and unlabeled data from the target domain.

[0069] The domain adaptation learning unit 300 includes a student model 301 to be learned, a first teacher model 302 and a second teacher model 305, a first pseudo label assignment unit 304 and a second pseudo label assignment unit 307, a GRL-equipped domain classification unit 309, a loss function calculation unit 310, and an output unit 311. The student model 301 is the learning target of domain adaptation learning. The first teacher model 302 provides pseudo-ground-truth labels for unlabeled data in the source domain, and the second teacher model 305 provides pseudo-ground-truth labels for unlabeled data in the target domain.

[0070] Learning model: The student model 301 has learning parameters obtained as the output result of the pre-learning process in the pre-learning processing unit 200. The first teacher model 302 and the second teacher model 305 are defined as models taking the first exponential moving average (EMA_s) and the second exponential moving average (EMA_t) of the learning parameters of the student model 301, respectively, as shown in the above equations (3) and (4).

[0071] As described above, by making the second teacher model 305 a second exponential moving average (EMA_t) model independent of the first exponential moving average (EMA_s) model of the first teacher model 302, it becomes possible to individually control the contribution of each of the inputs of unlabeled data in the source domain and unlabeled data in the target domain to the learning of the student model 301. This is because it is expected that the magnitude of the domain gap between the unlabeled data and labeled data in the source domain will be smaller than the domain gap between the unlabeled data in the target domain, which is different from the source, and the labeled data in the source domain.

[0072] Similar to a general learning model, the student model 301 includes a feature extraction unit and a detection unit 308, the first teacher model 302 includes a feature extraction unit and a detection unit 303, and the second teacher model 305 includes a feature extraction unit and a detection unit 306. In Fig. 2, the illustration of each feature extraction unit is omitted, and the student model 301, the first teacher model 302, and the second teacher model are each considered to be included in the block 305.

[0073] Three pieces of data, labeled data of the source domain, unlabeled data of the target domain, and unlabeled data of the source domain, are simultaneously input to the student model 301. Each feature extracted from these three pieces of input data by the feature extraction unit of the student model 301 is input to the detection unit 308. The detection unit 308 outputs detection results for each of the three feature amounts. In addition, each feature extracted from the three pieces of input data by the feature extraction unit of the student model 301 is also input to the domain classification unit 309 with GRL, which classifies the input data into the corresponding domain based on each feature amount.

[0074] A dataset consisting of unlabeled data in the source domain is passed as input to the first teacher model 302. Then, the inference result of the first teacher model 302 obtained by the detection unit 303 is input to the first pseudo label assignment unit 304. Also, a dataset consisting of unlabeled data in the target domain is passed as input to the second teacher model 305. Then, the inference results obtained by the detection units 303 and 306 are input to the first pseudo label assignment unit 304 and second pseudo label assignment unit 307, respectively.

[0075] Pseudo Label Assignment Unit: The first pseudo label assignment unit 304 selects samples to be incorporated as first pseudo labels for domain adaptation learning of the student model 301 from the inference results for unlabeled data in the source domain sequentially output from the first teacher model 302. Similarly, the second pseudo label assignment unit 307 selects samples to be incorporated as second pseudo labels for domain adaptation learning of the student model 301 from the inference results for unlabeled data in the target domain sequentially output from the second teacher model 305. Therefore, the loss function calculation unit 310 at the subsequent stage calculates the loss function L only for those samples for which the first pseudo labels and second pseudo labels have been selected from the inference results for the unlabeled data in the source domain and the unlabeled data in the target domain sequentially output from the student model 301. uns , loss function L unt is calculated.

[0076] The first pseudo label assignment unit 304 and the second pseudo label assignment unit 307 may use any method for selecting samples. For example, samples whose detection scores output together with the inference results exceed a predetermined threshold may be selected. In this specification, the threshold set in the first pseudo label assignment unit 304 is referred to as a "source threshold (δ s ) and the threshold set in the second pseudo label assigning unit 307 is set as a “target threshold δ t Since the domains are the same, there is a concern that the first pseudo label for the source domain will be of higher quality, making it difficult to adopt the second pseudo label for the target domain. Therefore, the "source threshold (δs ) and "target threshold (δ t The adoption of the first pseudo label and the second pseudo label may be adjusted by setting "(1)".

[0077] The source threshold and the target threshold may be fixed thresholds specified by a user, or each threshold may be dynamically adjusted using a learning model. An example of a learning model used in the latter method is a Gaussian Mixture Model (GMM). The GMM has a function of probabilistically expressing which Gaussian distribution a certain data point belongs to within a Gaussian mixture distribution (or a function of classifying a data group, each of which is composed of multiple feature quantities, into multiple clusters). Using the GMM can solve the problem of pseudo-labels being biased toward one domain.

[0078] Loss Function Calculation Unit: The first pseudo label selected by the first pseudo label assignment unit 304 functions as a correct label for the inference result of the student model 301 on the unlabeled data of the source domain. The second pseudo label selected by the second pseudo label assignment unit 307 functions as a correct label for the inference result of the student model 301 on the unlabeled data of the target domain.

[0079] The loss function calculation unit 310 calculates a loss function L for the first pseudo label of the inference result of the unlabeled data of the source domain by the student model 301. uns , a loss function L for the second pseudo-label of the inference result of the student model 301 on unlabeled data in the target domain unt , a loss function L for the correct label of the inference result of labeled data in the source domain by the student model 301 sup are calculated, respectively. Different data conversion processes are used for the input to the student model 301 and the input to the first teacher model 302 and second teacher model 305, and the system is designed to be able to learn consistent features. Any optimization function can be used as the loss function depending on the desired task. For example, the loss function may be Cross-Entropy Loss or Focal Loss.

[0080] Furthermore, the GRL-attached domain classification unit 309 classifies each feature extracted from the three pieces of input data by the feature extraction unit of the student model 301 into a domain. The loss function calculation unit 310 calculates a loss function L for the domain classification result of each feature. domains_t Calculate.

[0081] Then, the loss function calculation unit 310 calculates each loss function L sup , L uns , L unt , L domains_t The total loss function L obtained by weighting and adding the above equation (2) total Only the learning parameters of the student model 301 are optimized and updated based on the learning parameters of the first teacher model 302. Furthermore, the learning parameters of the first teacher model 302 are updated using a first exponential moving average (EMA_s) of the learning parameters of the student model 301, and the learning parameters of the second teacher model 305 are updated using a second exponential moving average (EMA_t) of the learning parameters of the student model 301. As described above, by making EMA_s and EMA_t independent exponential moving average models, it is possible to individually control the contribution of each data input to the learning of the student model 301.

[0082] The domain adaptation learning unit 300 recursively repeats learning until a predetermined number of learning loops 312 required for domain adaptation are completed. Then, the output unit 311 outputs the learning parameters of the model for which domain adaptation learning has been completed to an external device (e.g., an information terminal that uses the trained model) outside the learning system 100.

[0083] The UI unit 400 is composed of various input and output devices for interaction between the user and the learning system 100. The user is an entity that interacts with the entire learning system 100, and provides the UI unit 400 with various datasets for domain adaptive learning and other inputs.

[0084] The UI unit 400 includes dataset input units 401 to 403 that input a dataset consisting of labeled data from the source domain, a dataset consisting of unlabeled data from the source domain, and a dataset consisting of unlabeled data from the target domain to the learning system 100. Each of the dataset input units 401 to 403 acquires a dataset with a directory name or file name specified by the user, for example, from an external storage device or a data server, and passes it to the student models 201 and 301, the first teacher model 302, and the second teacher model 305, respectively.

[0085] Each data set input to the UI unit 400 may be a visual image of any domain (source domain or target domain). For example, the input data may be an image of a natural object captured by a camera system (not shown). As another example, the input data may be an image of a biological object captured by a medical imaging system (not shown). The UI unit 400 may have an input port for capturing image data from the camera system or medical imaging system.

[0086] The UI unit 400 includes a display 406 having a display screen. The display 406 may be a display device capable of interactive input, such as a display screen with a touch panel. The learning system 100 receives intermediate outputs during learning (e.g., features extracted from input data by the feature extraction units of the student model 301, the first teacher model 302, and the second teacher model 305) and the performance of the model obtained as the final result of domain adaptation learning output from the output unit 312 from the domain adaptation learning unit 300, and visually presents these to the user via the display screen of the display 406. The learning system 100 may also receive from the domain adaptation learning unit 300 first pseudo labels generated by the first pseudo label assignment unit 304 for a dataset consisting of unlabeled data in the source domain and second pseudo labels generated by the second pseudo label assignment unit 307 for a dataset consisting of unlabeled data in the target domain, and visually present the appearance of the pseudo labels to the user via the display screen of the display 406. Therefore, the user can visually check the processing operations and final output of the learning system 100, such as intermediate outputs during learning, the performance of the final domain adaptation model, and the state of pseudo-labels, via the display screen of the display 406.

[0087] The UI unit 400 includes a control information input unit 405 that inputs information for controlling the internal variables of the learning system 100. The control information input unit 405 may be an input device such as a keyboard, or an input device that allows interactive input, such as a mouse or a touch panel integrated with the display screen of the display 406. The control information input unit 405 may also be a voice input device with a voice recognition function.

[0088] The control information input unit 405 outputs the input information to the learning system 100 via the components (not shown) that connect to it. The learning system 100 controls the performance of the generative model based on the information input from the control information input unit 405. The internal variables of the learning system 100 include, for example, hyperparameters of the learning model, such as the following:

[0089] Loss weights: w1, w2, w3, w4 Source threshold: δ s ・Target threshold: δ t ・First exponential moving average: EMA_s ・Second exponential moving average: EMA_t

[0090] The loss weights w1, w2, w3, and w4 are parameters used to control the loss function for each input of labeled data in the source domain, unlabeled data in the source domain, and unlabeled data in the target domain, and their respective contributions to the domain classification result.

[0091] Source threshold δ s is a variable used in the first pseudo label assignment unit 304 to select a first pseudo label as a learning sample from the detection result of the detection unit 303. Also, the target threshold δ t is a variable used in the second pseudo label assignment unit 307 to select a second pseudo label as a learning sample from the detection result of the detection unit 306. Generally, the accuracy of the first pseudo label for unlabeled data in the same source domain as the labeled data is higher than the accuracy of the second pseudo label for unlabeled data in a target domain that is different from the labeled data. If the first pseudo label and the second pseudo label are sampled using the same criteria, knowledge about the target domain will not be reflected in the learning model. Therefore, by configuring the system so that sampling can be controlled separately for unlabeled data in the source domain and unlabeled data in the target domain, the domain adaptation performance of the learning model is improved.

[0092] The first exponential moving average EMA_s is a moving coefficient used when defining the first teacher model 302 as an exponential moving average model of the student model 301. The second exponential moving average EMA_t is a moving coefficient used when defining the second teacher model 305 as an exponential moving average model of the student model 301. Generally, the larger the moving exponential average coefficient, the longer the average weight is maintained, making it more robust against noise accompanying pseudo labels. In this embodiment, moving coefficients are defined independently for two inputs with different domain gaps, namely, unlabeled data in the target domain and unlabeled data in the source domain, and the system is designed so that the contribution of each to the student model 301 (the speed at which it is reflected in learning) can be individually controlled.

[0093] Note that the above mainly illustrates hyperparameters closely related to domain adaptation according to the present disclosure. The control information input unit 405 may also receive input of hyperparameters other than those described above (not directly related to domain adaptation) and other internal variables.

[0094] FIG. 4 shows an example of a screen configuration for the control information input unit 405 to input information about the internal variables of the learning system 100.

[0095] In the left half of the screen shown in FIG. 4 , "Input: Target (unlabeled)," "Input: Source (labeled)," and "Input: Source (unlabeled)," denoted by reference numerals 413, 411, and 412 from top to bottom, are UI components corresponding to the dataset input sections 403, 401, and 402, respectively. Using the UI components 411, 412, and 413, the user can input directory names or file names from which to acquire a dataset consisting of labeled data from the source domain, a dataset consisting of unlabeled data from the source domain, and a dataset consisting of unlabeled data from the target domain, respectively. Each of the UI components 411, 412, and 413 may be, for example, a text box or a pull-down menu.

[0096] It is mandatory to specify a dataset consisting of labeled data from the source domain and a dataset consisting of unlabeled data from the target domain in the UI components "Input: Source (labeled)" and "Input: Target (unlabeled)" denoted by reference numerals 411 and 413, respectively. On the other hand, it is optional to specify a dataset consisting of unlabeled data from the source domain in the UI component "Input: Source (unlabeled)" denoted by reference numeral 412. When a dataset consisting of unlabeled data from the source domain is specified in the UI component 412 of “Input: Source (unlabeled)”, the domain adaptive learning unit 300 performs domain adaptive learning with the unlabeled data from the source domain added (i.e., according to the present disclosure); however, when such a dataset is not specified, domain adaptive learning without using the unlabeled data from the source domain (for example, the adaptive teaching described in section A-1 above) is performed.

[0097] The right half of the screen shown in FIG. 4 shows the thresholds (source threshold δ ) used by the first pseudo label assignment unit 304 and the second pseudo label assignment unit 307 to select training samples. s and the target threshold δ t ) setting (Threshold for pseudo-labeling). This area has two check boxes, "Use GMM" and "Tune by hand", respectively indicated by reference numerals 421 and 422 (radio buttons may be used instead of check boxes).

[0098] When the “Use GMM” checkbox 421 is checked, the source threshold δ used by the first pseudo label assignment unit 304 is set. s and the target threshold δ used by the second pseudo label assigner 307. t is automatically set using GMM. When the "Tune by hand" checkbox 422 is checked, the user can set the source threshold δ by moving the slider indicated by the reference numeral 423 in the horizontal direction. s and the target threshold δt can be manually adjusted. The left and right ends of the slider 423 are assigned to the source domain and the target domain, respectively. If the user feels that the pseudo labels are biased toward the source domain, the user can adjust the pseudo labels so that they are biased toward the target domain by moving the slider 423 to the right toward the target domain.

[0099] Although not shown in the screen shown in FIG. 4, the screen may further include UI components for setting other hyperparameters (such as the loss weights (w1, w2, w3, and w4), the first exponential moving average (EMA_s), and the second exponential moving average (EMA_t)). The screen may also include UI components for setting hyperparameters other than those described above (such as the number of iterations of the learning loops 206 and 312).

[0100] The user interface unit 400, which inputs and outputs information as described above, can be implemented using a small information terminal such as a smartphone or tablet. In addition, the learning model generated by the learning system 100 performing domain adaptation learning based on information input from the user interface unit 400 can be obtained by a small information terminal such as a smartphone or tablet via the output unit 311 and used on this type of information terminal.

[0101] B-2. Operational Example Fig. 5 shows, in the form of a flowchart, the steps of model learning performed in the learning system 100 shown in Fig. 2 and Fig. 3. The model learning process performed in the learning system 100 will be described below with reference to Fig. 5.

[0102] First, the pre-learning processing unit 200 executes pre-learning processing of the student model 201 (step S501).

[0103] The student model 201 may be a model pre-trained using any other dataset. The pre-training processing unit 200 pre-trains the student model 201 using three datasets: a dataset consisting of labeled data from the source domain, a dataset consisting of unlabeled data from the source domain, and a dataset consisting of unlabeled data from the target domain. That is, the pre-training processing unit 200 trains the student model 201 so that it can make correct inference using the dataset consisting of labeled data from the source domain. The pre-training processing unit 200 also trains the student model 201 so that it can extract domain-independent features using the three datasets: a dataset consisting of labeled data from the source domain, a dataset consisting of unlabeled data from the source domain, and a dataset consisting of unlabeled data from the target domain.

[0104] The pre-learning process in step S501 is repeatedly performed until a predetermined number of iterations is reached (No in step S502). After the pre-learning process is completed the predetermined number of iterations (Yes in step S502), the domain adaptive learning process is subsequently performed.

[0105] In the domain adaptation learning process, first, a process of generating a first pseudo label that functions as a correct label for unlabeled data in the source domain (step S503) and a process of generating a second pseudo label that functions as a correct label for unlabeled data in the target domain (step S504) are performed.

[0106] In step S503, a dataset consisting of unlabeled data in the source domain is passed as input to the first teacher model 302, and the inference results for each unlabeled data are input to the first pseudo label assignment unit 304. The first pseudo label assignment unit 304 assigns the detection scores of the inference results sequentially output from the first teacher model 302 to the first pseudo label assignment unit 304. s The samples with more than 1 are selected and taken as the first pseudo label.

[0107] In step S504, a dataset consisting of unlabeled data of the target domain is passed as input to the second teacher model 305, and the inference results for each unlabeled data are input to the second pseudo label assignment unit 307. The second pseudo label assignment unit 307 assigns a detection score among the inference results for the unlabeled data of the target domain sequentially output from the second teacher model 305 to the second pseudo label assignment unit 307 when the detection score is greater than or equal to the target threshold δ t Samples exceeding 1 are selected and taken as the second pseudo label.

[0108] The first pseudo label generation process (step S503) and the second pseudo label generation process (step S504) may be performed in the order described above, in the reverse order, or simultaneously in parallel.

[0109] Next, the loss function calculation unit 310 executes a calculation process of the loss function (step S505).

[0110] Three pieces of data, namely, labeled data in the source domain, unlabeled data in the source domain, and unlabeled data in the target domain, are simultaneously input to the student model 301. For each of the three pieces of input data, the student model 301 outputs intermediate output features and a final inference result.

[0111] The loss function calculation unit 310 calculates a loss function L between the correct label and the inference result of the labeled data in the source domain by the student model 301. sup , the loss function L with the first pseudo label for the inference result of the unlabeled data in the source domain by the student model 301 uns , the loss function L with the second pseudo label for the inference result of the student model 301 on unlabeled data in the target domain unt , a loss function L for the domain classification result of each feature by the domain classification unit 309 domains_t However, among the inference results for the unlabeled data of the source domain and the unlabeled data of the target domain sequentially output from the student model 301, the loss function L is calculated only for those selected by the first pseudo label assignment unit 304 and the second pseudo label assignment unit 307, respectively.uns , loss function L unt is calculated.

[0112] Then, each loss function L calculated by the loss function calculation unit 310 sup , L uns , L unt , L domains_t The total loss function L obtained by weighting and adding the above equation (2) total The learning parameters of the student model 301 are optimized and updated based on the learning parameters of the student model 301. Furthermore, the optimized and updated learning parameters of the student model 301 are reflected in the first teacher model 302 according to the first exponential moving average model (EMA_s), and are also reflected in the second teacher model 120 according to the second exponential moving average model (EMA_t).

[0113] The domain adaptive learning process consisting of steps S503 to S505 is repeatedly performed until a predetermined number of iterations is reached (No in step S506). Then, when the domain adaptive learning process is completed the predetermined number of iterations (Yes in step S506), the output unit 311 outputs the learning parameters of the model after the learning is completed (step S507), and this process ends.

[0114] C. Configuration of Information Processing Device In this section C, a configuration of an information processing device that can be used for processing related to domain adaptive learning according to the present disclosure will be described.

[0115] 7 shows an example of the hardware configuration of an information processing device 2000. This information processing device 2000 includes a CPU (Central Processing Unit) 2001, a ROM (Read Only Memory) 2002, a RAM (Random Access Memory) 2003, a host bus 2004, a bridge 2005, an expansion bus 2006, an interface unit 2007, an input unit 2008, an output unit 2009, a storage unit 2010, a drive 2011, and a communication unit 2013. The information processing device 2000 is configured, for example, by an information terminal such as a personal computer, a tablet, or a smartphone.

[0116] The CPU 2001 controls the overall operation of the information processing device 2000 in accordance with various programs. When performing processing with a high computational load (for example, processing related to learning a neural network model) on the information processing device 2000, it is desirable that the CPU 2001 be a multi-core CPU (for example, Apple M1 Max, etc.), or that the information processing device 2000 further be equipped with a multi-core processor (for example, NVIDIA R6000, etc.) such as a GPU (Graphics Processing Unit) or GPGPU (General-purpose computing on graphics processing units) in addition to the CPU 2001. However, hereinafter, for convenience, these will be collectively referred to simply as the CPU 2001.

[0117] The ROM 2002 stores in a nonvolatile manner programs (such as a basic input / output system) and calculation parameters used by the CPU 2001. The RAM 2003 is used to load programs to be executed by the CPU 2001 and to temporarily store parameters such as working data that change as appropriate during program execution. Programs loaded into the RAM 2003 and executed by the CPU 2001 include, for example, various application programs and an operating system (OS).

[0118] The CPU 2001, ROM 2002, and RAM 2003 are interconnected by a host bus 2004, which includes a CPU bus and the like. The CPU 2001 executes various application programs in an execution environment provided by an OS through the cooperative operation of the ROM 2002 and RAM 2003, thereby realizing various functions and services. If the information processing device 2000 is a personal computer, the OS may be, for example, Microsoft Windows (registered trademark), Unix (registered trademark), or a successor OS. Furthermore, application programs executed on the information processing device 2000 include, for example, a program that executes the domain adaptive learning process according to the present disclosure. At least some of the application programs may be computer programs provided as libraries.

[0119] The host bus 2004 is connected to an expansion bus 2006 via a bridge 2005. The expansion bus 2006 is, for example, a PCI (Peripheral Component Interconnect) bus or PCI Express, and the bridge 2005 is based on the PCI standard. However, the information processing device 2000 does not need to be configured so that the circuit components are separated by the host bus 2004, bridge 2005, and expansion bus 2006, and may be implemented so that almost all circuit components are interconnected by a single bus (not shown).

[0120] The interface unit 2007 connects peripheral devices such as an input unit 2008, an output unit 2009, a storage unit 2010, a drive 2011, and a communication unit 2013 in accordance with the standards of the expansion bus 2006. However, not all of the peripheral devices shown in Fig. 7 are necessarily required, and the information processing device 2000 may further include peripheral devices not shown. Furthermore, the peripheral devices may be built into the main body of the information processing device 2000, or some of the peripheral devices may be externally connected to the main body of the information processing device 2000.

[0121] The input unit 2008 is composed of an input control circuit that generates an input signal based on input from a user and outputs it to the CPU 2001. When the information processing device 2000 is a personal computer, the input unit 2008 may include a keyboard, a mouse, a touch panel, and may further include a camera and a microphone. The output unit 2009 includes display devices such as a liquid crystal display (LCD) device, an organic electroluminescence (EL) display device, and an LED (light emitting diode), as well as an audio output device such as a speaker. The input unit 2008 and the output unit 2009 are used to input hyperparameters required for executing the domain adaptive learning process according to the present disclosure, and to present intermediate and final learning results.

[0122] The storage unit 2010 stores files such as programs (applications, OS, etc.) executed by the CPU 2001 and various data. The storage unit 2010 is configured with a large-capacity storage device such as an SSD (Solid State Drive) or an HDD (Hard Disk Drive), but may also include an external storage device.

[0123] The removable storage medium 2012 is a storage medium configured as a cartridge, such as a microSD card. The drive 2011 performs read and write operations on the loaded removable storage medium 113. The drive 2011 outputs data read from the removable storage medium 2012 to the RAM 2003 or the storage unit 2010, and writes data on the RAM 2003 or the storage unit 2010 to the removable storage medium 2012.

[0124] The communication unit 2013 is a device that performs wireless communication such as Wi-Fi (registered trademark), Bluetooth (registered trademark), or cellular communication networks such as 4G and 5G. The communication unit 2013 may also include terminals such as a USB (Universal Serial Bus) or HDMI (registered trademark) (High-Definition Multimedia Interface), and may further include a function for performing HDMI (registered trademark) communication with USB devices such as scanners and printers, displays, etc. Programs executed on the information processing device 2000 are installed externally, for example, through the communication unit 2013. Data used in the implementation of intervention effect calculations according to the present disclosure is acquired from an external database, for example, through the communication unit 2013.

[0125] The present disclosure has been described in detail above with reference to specific embodiments. However, the present disclosure should not be construed as being limited to the above-described embodiments, and it is obvious that those skilled in the art can modify or substitute the embodiments without departing from the spirit of the present disclosure. Furthermore, the effects described in this specification are merely examples, and the effects brought about by the present disclosure are not limited thereto, and additional effects not described in this specification may exist.

[0126] The domain adaptation technique of the present disclosure, which utilizes unlabeled data from the source domain, can be applied to training AI models that perform image data processing tasks such as object detection, but the gist of the present disclosure is not limited thereto and can also be applied to training models that perform various other tasks. The present disclosure can be applied to training models that handle various modalities such as audio, video, and text.

[0127] A learning model generated using the domain adaptation technique according to the present disclosure can be suitably applied to various technical and industrial fields where domain shifts (especially domain shifts with large domain gaps) can occur, and can handle multiple use cases where the domain shifts. For example, a learning model generated using the domain adaptation technique according to the present disclosure can be used as an object detection model that detects desired objects from camera images in in-vehicle camera systems in the mobility field and in medical imaging systems installed in surgical assistance robots in the medical field.

[0128] In short, the present disclosure has been described in the form of examples, and the contents of the specification should not be interpreted as limiting. To determine the gist of the present disclosure, the claims should be taken into consideration.

[0129] The series of processes described in this specification can be executed by hardware, software, or a configuration that combines hardware and software. When executing processes by software, a program recording a processing sequence related to realizing the present disclosure is installed in memory in a computer incorporated in dedicated hardware and executed. It is also possible to install the program in a general-purpose computer capable of executing various processes and execute the processes related to realizing the present disclosure.

[0130] The program can be stored in advance on a recording medium installed in the computer, such as a HDD, SSD, or ROM. Alternatively, the program can be temporarily or permanently stored on a removable recording medium such as a flexible disk, CD-ROM (Compact Disc Read Only Memory), MO (Magneto Optical) disk, DVD (Digital Versatile Disc), BD (Blu-Ray Disc (registered trademark)), magnetic disk, or USB (Universal Serial Bus) memory. Using such a removable recording medium, a program related to the realization of the present disclosure can be provided as so-called package software.

[0131] The program may also be transferred wirelessly or via a wire from a download site to a computer via a network such as a wide area network (WAN) typified by cellular, a local area network (LAN), the Internet, etc. The computer can receive the program transferred in this manner and install it in a large-capacity storage device such as an HDD or SSD within the computer.

[0132] The present disclosure may also be configured as follows.

[0133] (1) An information processing device comprising: an acquisition unit that acquires a labeled dataset consisting of labeled data of a first domain, a first unlabeled dataset consisting of unlabeled data of the first domain, and a second unlabeled dataset consisting of unlabeled data of a second domain different from the first domain; and a learning unit that performs model learning based on a loss function calculated using the labeled dataset, the first unlabeled dataset, and the second unlabeled dataset.

[0134] (2) The information processing device described in (1) above, wherein the learning unit learns the model based on a loss function calculated using labels of labeled data in the first domain, a loss function calculated using first pseudo labels generated from unlabeled data in the first domain, and a loss function calculated using second pseudo labels generated from unlabeled data in the second domain.

[0135] (3) The information processing device described in (2) above, wherein the first pseudo labels are generated from unlabeled data of the first domain using a first teacher model, and the second pseudo labels are generated from unlabeled data of the second domain using a second teacher model.

[0136] (4) The information processing device according to (3), wherein the first teacher model and the second teacher model reflect learning parameters of the models according to an exponential moving average model.

[0137] (5) The information processing device described in (4) above, wherein the first teacher model and the second teacher model are defined according to mutually independent exponential moving average models based on the magnitude of a domain gap between the labeled dataset of the first domain and the unlabeled dataset of the first domain and the domain gap between the labeled dataset of the first domain and the unlabeled dataset of the second domain.

[0138] (6) The information processing device described in any one of (3) to (5) above, further comprising a first pseudo label assignment unit that selects samples from the labels generated by the first teacher model to be incorporated as training data for the model, and a second pseudo label assignment unit that selects samples from the labels generated by the second teacher model to be incorporated as training data for the model.

[0139] (7) The information processing device described in (6) above, wherein the first pseudo label assignment unit selects samples whose detection scores of the inference results of the first teacher model exceed a first threshold, and the second pseudo label assignment unit selects samples whose detection scores of the inference results of the second teacher model exceed a second threshold.

[0140] (8) The information processing device according to (7), wherein the first threshold and the second threshold are fixed thresholds specified by a user or dynamic thresholds specified using a predetermined learning model.

[0141] (9) The information processing device according to any one of (1) to (8), further comprising a domain classifier that classifies the domain of original input data from features intermediately output from the model, wherein the learning unit further uses a loss function calculated from the domain classification results of the domain classifier for each feature intermediately output from the model when labeled data of the first domain, unlabeled data of the first domain, and unlabeled data of the second domain are input, to learn the model.

[0142] (10) The information processing device according to any one of (1) to (9), further comprising a user interface unit that accepts information input, wherein the learning unit performs a learning process for the model based on the information input to the user interface unit.

[0143] (10-1) The information processing device according to (10), wherein the user interface unit accepts input of hyperparameters of the model.

[0144] (11) The information processing device according to (10), wherein the user interface unit accepts input of the labeled dataset, the first unlabeled dataset, and the second unlabeled dataset.

[0145] (12) The information processing device described in any one of (10) or (11) above, wherein the user interface unit includes a presentation unit that presents at least one of the learning results by the learning unit, the first pseudo label generated by the first teacher model, and the second pseudo label generated by the second teacher model.

[0146] (13) The information processing device according to any one of (10) to (12), wherein the user interface unit further receives an input of a weighting value to be assigned to each of a plurality of loss functions used in training the model.

[0147] (13-1) The information processing device according to (13), wherein the learning unit controls the learning process of the model based on a total loss function obtained by weighting and adding the plurality of loss functions using weighting values ​​input to the user interface unit.

[0148] (14) The information processing device described in any one of (10) to (13), wherein the user interface unit further accepts input of a first threshold for selecting samples to be incorporated as training data for the model from among pseudo labels generated from each unlabeled data in the first unlabeled dataset, and a second threshold for selecting samples to be incorporated as training data for the model from among pseudo labels generated from each unlabeled data in the second unlabeled dataset.

[0149] (14-1) The information processing device described in (14) above, wherein the learning unit selects a first pseudo label to be used in training the model from among the estimation results of a first teacher model for each unlabeled data in a first unlabeled dataset based on the first threshold, and selects a second pseudo label to be used in training the model from among the estimation results of a second teacher model for each unlabeled data in a second unlabeled dataset based on the second threshold.

[0150] (15) The information processing device described in any one of (10) to (14), wherein the user interface unit further accepts input of a first moving coefficient that defines an exponential moving average model for reflecting the learning parameters of the model in a first teacher model, and a second moving coefficient that defines an exponential moving average model for reflecting the learning parameters of the model in a second teacher model; and the learning unit generates a first pseudo label from each unlabeled data of the first unlabeled dataset using the first teacher model based on the first moving coefficient, and generates a second pseudo label from each unlabeled data of the second unlabeled dataset using the second teacher model based on the second moving coefficient.

[0151] (16) An information processing method comprising: an acquisition step of acquiring a labeled dataset consisting of labeled data of a first domain, a first unlabeled dataset consisting of unlabeled data of the first domain, and a second unlabeled dataset consisting of unlabeled data of a second domain different from the first domain; and a training step of training a model based on a loss function calculated using the labeled dataset, the first unlabeled dataset, and the second unlabeled dataset.

[0152] (17) A computer program written in a computer-readable format to cause a computer to function as: an acquisition unit that acquires a labeled dataset consisting of labeled data of a first domain, a first unlabeled dataset consisting of unlabeled data of the first domain, and a second unlabeled dataset consisting of unlabeled data of a second domain different from the first domain; and a learning unit that trains a model based on a loss function calculated using the labeled dataset, the first unlabeled dataset, and the second unlabeled dataset.

[0153] DESCRIPTION OF SYMBOLS 100...Learning system, 110...First teacher model 111...Feature extraction unit, 112...Detection unit, 113...First pseudo label assignment unit 120...Second teacher model, 121...Feature extraction unit, 122...Detection unit 123...Second pseudo label assignment unit, 130...Student model 131...Feature extraction unit, 132...Detection unit, 140...Domain classification unit 141...GRL 200...Pre-learning processing unit, 201...Student model (during pre-learning) 202...Detection unit, 203...Domain classification unit (with GRL) 204...Loss function calculation unit, 205...Output unit 300...Domain adaptation learning unit, 301...Student model 302...Second teacher model, 303...Detection unit 304...First pseudo label assignment unit, 305...Second teacher model 306...detection unit, 307...second pseudo label assignment unit, 308...detection unit 309...domain classification unit with GRL, 310...loss function calculation unit 311...output unit, 400...user interface unit 401-403...dataset input unit, 405...control information input unit 406...display 600...AT framework, 610...teacher model 611...feature extraction unit, 612...detection unit, 620...student model 621...feature extraction unit, 622...detection unit, 630...domain classification unit 631...GRL 2000...information processing device, 2001...CPU, 2002...ROM 2003...RAM, 2004...host bus, 2005...bridge 2006...expansion bus, 2007...interface unit 2008...input unit, 2009...output unit, 2010...storage unit 2011...drive, 2012...removable recording medium, 2013...communication unit

Claims

1. An information processing apparatus comprising: an acquisition unit that acquires a labeled data set consisting of labeled data of a first domain, a first unlabeled data set consisting of unlabeled data of the first domain, and a second unlabeled data set consisting of unlabeled data of a second domain different from the first domain; and a learning unit that performs learning of a model based on a loss function calculated using the labeled data set, the first unlabeled data set, and the second unlabeled data set.

2. The information processing apparatus according to claim 1, wherein the learning unit performs learning of the model based on a loss function calculated using the label of the labeled data of the first domain, a loss function calculated using a first pseudo-label generated from the unlabeled data of the first domain, and a loss function calculated using a second pseudo-label generated from the unlabeled data of the second domain.

3. The information processing apparatus according to claim 2, wherein the first pseudo-label is generated from the unlabeled data of the first domain using a first teacher model, and the second pseudo-label is generated from the unlabeled data of the second domain using a second teacher model.

4. The information processing apparatus according to claim 3, wherein the first teacher model and the second teacher model that reflect the learning parameters of the model according to an exponential moving average model are used.

5. The information processing apparatus according to claim 4, wherein the first teacher model and the second teacher model are respectively defined according to independent exponential moving average models based on the magnitude of the domain gap between the labeled data set of the first domain and the unlabeled data set of the first domain, and the magnitude of the domain gap between the labeled data set of the first domain and the unlabeled data set of the second domain.

6. The information processing apparatus according to claim 3, further comprising: a first pseudo-label assignment unit that selects samples to be incorporated as learning data of the model from among the labels generated by the first teacher model; and a second pseudo-label assignment unit that selects samples to be incorporated as learning data of the model from among the labels generated by the second teacher model.

7. The first pseudo-label assignment unit selects samples whose detection scores of the inference results of the first teacher model exceed a first threshold, and the second pseudo-label assignment unit selects samples whose detection scores of the inference results of the second teacher model exceed a second threshold. The information processing apparatus according to claim 6.

8. The first threshold and the second threshold are each a fixed threshold specified by a user or a dynamic threshold specified using a predetermined learning model. The information processing apparatus according to claim 7.

9. The apparatus further includes a domain classifier that classifies the domain of the original input data from the feature amounts output intermediate from the model. The learning unit further uses a loss function calculated from the domain classification results of the domain classifier for each feature amount output intermediate from the model when the labeled data of the first domain, the unlabeled data of the first domain, and the unlabeled data of the second domain are respectively input, to perform learning of the model. The information processing apparatus according to claim 1.

10. The apparatus further includes a user interface unit that receives information input. The learning unit performs learning processing of the model based on the information input to the user interface unit. The information processing apparatus according to claim 1.

11. The user interface unit receives input of the labeled data set, the first unlabeled data set, and the second unlabeled data set. The information processing apparatus according to claim 10.

12. The user interface unit includes a presentation unit that presents at least one of the learning result by the learning unit, the first pseudo-label generated by the first teacher model, and the second pseudo-label generated by the second teacher model. The information processing apparatus according to claim 10.

13. The user interface unit further receives input of a weighting value given to each of a plurality of loss functions used for learning of the model. The information processing apparatus according to claim 10.

14. The information processing apparatus according to claim 10, wherein the user interface unit further receives an input of a first threshold value for selecting samples to be incorporated as learning data of the model from among the pseudo labels generated from each label-free data of the first label-free data set, and a fourth threshold value for selecting samples to be incorporated as learning data of the model from among the pseudo labels generated from each label-free data of the second label-free data set.

15. The information processing apparatus according to claim 10, wherein the user interface unit further receives an input of a first moving coefficient for defining an exponential moving average model for reflecting the learning parameters of the model on a first teacher model, and a second moving coefficient for defining an exponential moving average model for reflecting the learning parameters of the model on a second teacher model, and the learning unit generates a first pseudo label from each label-free data of the first label-free data set using the first teacher model based on the first moving coefficient, and generates a second pseudo label from each label-free data of the second label-free data set using the second teacher model based on the second moving coefficient.

16. An information processing method comprising: an acquisition step of acquiring a labeled data set composed of labeled data of a first domain, a first label-free data set composed of label-free data of the first domain, and a second label-free data set composed of label-free data of a second domain different from the first domain; and a learning step of performing learning of a model based on a loss function calculated using the labeled data set, the first label-free data set, and the second label-free data set.

17. A computer program described in a computer-readable format so as to cause a computer to function as: an acquisition unit that acquires a labeled data set composed of labeled data of a first domain, a first label-free data set composed of label-free data of the first domain, and a second label-free data set composed of label-free data of a second domain different from the first domain; and a learning unit that performs learning of a model based on a loss function calculated using the labeled data set, the first label-free data set, and the second label-free data set.

Images