Method and apparatus for multi-target domain adaptation

Abstract

A method and an apparatus for multi-target domain adaptation are disclosed. According to one aspect of the present disclosure, the provided computer-implemented method for multi-target domain adaptation of a model that performs an image-based task comprises the steps of: integrating a plurality of different target domains into a target superdomain; masking at least some images in the target superdomain so as to generate one or more masked target images; and updating parameters of the model on the basis of the prediction of the model for the target superdomain and the masked target images.

Description

Method and device for multi-target domain adaptation

[0001] The present disclosure relates to multi-target domain adaptation of artificial intelligence models.

[0002] The content described below merely provides background information related to the present embodiment and does not constitute prior art.

[0003] Unsupervised Domain Adaptation (UDA) is considered an important technique to alleviate the high demand for data annotation. The goal of UDA is to minimize performance degradation when a model trained in a labeled source domain is applied to an unlabeled target domain. For example, the source domain may contain labeled synthetic data automatically generated by a simulator, while the target domain may contain unlabeled data collected in the real world. In another example, the source domain may contain manually labeled real-world data, while the target domain may contain unlabeled real-world data collected in environments (e.g., country, weather, and / or time zone) different from the source domain.

[0004] Most UDA techniques are designed for single-target domain adaptation (STDA). However, in real-world applications, models must handle multiple domains. For example, in autonomous driving scenarios, models must handle diverse domains due to changes in weather, driving location, and lighting conditions.

[0005] To address these challenges, multi-target domain adaptation (MTDA) has been introduced, allowing a single model to simultaneously adapt from a source domain to multiple unlabeled target domains. MTDA trains multiple STDA models, each adapting to a specific target domain, and distills the knowledge from these STDA models into an MTDA model. While this indirect adaptation approach yields excellent performance, it suffers from limitations, such as the requirement to train multiple STDA models individually and the inability to account for contextual variations across multiple target domains.

[0006] The main purpose of the present disclosure is to provide a method and apparatus that enable seamless adaptation from a source domain to all target domains by directly training a single model using the target superdomain.

[0007] The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the description below.

[0008] According to one aspect of the present disclosure, a computer-implemented method for multi-target domain adaptation of a model performing an image-based task is provided, the method comprising: integrating a plurality of different target domains into a target superdomain, wherein the target superdomain includes a plurality of original target images each belonging to one of the plurality of target domains, one or more first interpolated target images generated based on image-level domain interpolation between the plurality of target domains, and one or more second interpolated target images generated based on region-level domain interpolation between the plurality of target domains; masking at least some images of the target superdomain to generate one or more masked target images; and updating parameters of the model based on predictions of the model for the target superdomain and the masked target images.

[0009] According to another aspect of the present disclosure, there is provided an apparatus for multi-target domain adaptation, comprising: at least one memory storing instructions; and at least one processor, wherein the at least one processor executes the instructions to integrate a plurality of different target domains into a target superdomain, wherein the target superdomain includes a plurality of original target images each belonging to one of the plurality of target domains, one or more first interpolated target images generated based on image-level domain interpolation between the plurality of target domains, and one or more second interpolated target images generated based on region-level domain interpolation between the plurality of target domains; masking at least some images of the target superdomain to generate one or more masked target images; and updating parameters of a model based on predictions of the model for the target superdomain and the masked target image.

[0010] According to another aspect of the present disclosure, there is provided a device comprising at least one processor and at least one memory storing instructions that, when executed by the at least one processor, cause the at least one processor to implement an artificial intelligence model that performs an image-based task, wherein the artificial intelligence model comprises a target superdomain that integrates a plurality of different target domains, wherein the target superdomain includes a plurality of original target images each belonging to one of the plurality of target domains, one or more first interpolated target images generated based on image-level domain interpolation between the plurality of target domains, and one or more second interpolated target images generated based on region-level domain interpolation between the plurality of target domains, and a multi-target domain adaptation model adapted for the plurality of target domains based on one or more masked target images generated by masking at least some images of the target superdomain.

[0011] According to an embodiment of the present disclosure, by directly training a single model using a target superdomain, it has the effect of enabling smooth adaptation from the source domain to all target domains.

[0012] According to an embodiment of the present disclosure, the gap between target domains can be resolved and the generalization performance of the network can be improved.

[0013] According to the embodiment of the present disclosure, there is no need to change the structure of the model for domain adaptation, so that it can be applied non-invasively to a desired solution.

[0014] The effects of the present disclosure are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description below.

[0015] FIG. 1 is a block diagram schematically illustrating a multi-target domain adaptation device according to one embodiment of the present disclosure.

[0016] FIG. 2 is an exemplary diagram referenced to explain the operation of a domain integration module and a domain consistency module according to one embodiment of the present disclosure.

[0017] FIG. 3 is an exemplary diagram referenced to explain the operation of a domain adaptation module according to one embodiment of the present disclosure.

[0018] FIG. 4 is an exemplary diagram referenced to explain the operation of a domain alignment module according to one embodiment of the present disclosure.

[0019] FIG. 5 is a flowchart schematically illustrating a multi-target domain adaptation method according to one embodiment of the present disclosure.

[0020] FIG. 6 is a block diagram schematically illustrating an exemplary computing device that can be used to implement a method or device according to the present disclosure.

[0021] Hereinafter, some embodiments of the present disclosure will be described in detail using exemplary drawings. When designating components in each drawing, it should be noted that, where possible, identical components are given the same reference numerals, even if they appear in different drawings. Furthermore, when describing the present disclosure, detailed descriptions of related known structures or functions will be omitted if they are deemed to obscure the gist of the present disclosure.

[0022] In describing components of embodiments according to the present disclosure, symbols such as first, second, i), ii), a), b) may be used. These symbols are only for distinguishing the components from other components, and the nature, order, or sequence of the components are not limited by the symbols. When a part in the specification is said to "include" or "have" a component, this does not mean that other components are excluded, but rather that other components may be included, unless explicitly stated otherwise.

[0023] The detailed description set forth below, together with the accompanying drawings, is intended to explain exemplary embodiments of the present disclosure and is not intended to represent the only embodiments in which the present disclosure may be practiced.

[0024] In this disclosure, we propose a Mask Mix Multi-Target Domain Adaptation (M3TDA) framework that can directly adapt a single model to multi-target domains.

[0025] FIG. 1 is a block diagram schematically illustrating a multi-target domain adaptation device according to one embodiment of the present disclosure.

[0026] As illustrated in FIG. 1, a multi-target domain adaptation (MTDA) device (10) according to an embodiment of the present disclosure may include all or part of a domain congregation module (100), a domain consistency module (120), a domain adaptation module (140), and a domain alignment module (160). Not all blocks illustrated in FIG. 1 are essential components, and some blocks included in other embodiments may be added, changed, or deleted. Meanwhile, the components illustrated in FIG. 1 represent functionally distinct elements, and at least one of the components may be implemented in a form in which they are integrated with each other in an actual physical environment.

[0027] FIG. 2 is an exemplary diagram referenced to explain the operation of a domain integration module and a domain consistency module according to one embodiment of the present disclosure.

[0028] The domain consolidation module (100) consolidates multiple unlabeled target domains into a single target superdomain. Integration between domains may include naive domain consolidation and / or mix-based domain consolidation.

[0029] Simple domain consolidation is a method of merging all target domains into a target superdomain without considering inter-target domain gaps.

[0030] Hybrid-based domain integration is a method of creating one or more intermediate domains by interpolating target domains to bridge the gap between target domains.

[0031] The gap between target domains may include a style gap due to differences in visual appearance (e.g., color, texture, and / or pattern) and a contextual gap due to differences in scene context. To resolve the style gap and the contextual gap, the domain integration module (100) may generate an intermediate domain at the style level and an intermediate domain at the domain level, respectively.

[0032] The domain integration module (100) can generate an intermediate domain at the style level using image-level domain interpolation. The domain integration module (100) can generate an intermediate domain that blends the styles of different target domains based on a style transfer technique. For example, the domain integration module (100) can generate a style-transferred target image by transferring the style of one target domain to an image of another target domain. As a style transfer technique, a method based on Generative Adversarial Networks (GAN), a Fourier transform, and / or a photometric method can be applied.

[0033] The domain integration module (100) can generate an intermediate domain at the region level using region-level domain interpolation. The domain integration module (100) can generate an intermediate domain by mixing some regions of images of different target domains based on a mixing strategy. For example, the domain integration module (100) can cut out some regions from an image of one target domain and then paste them into an image of another target domain. As mixing strategies, CutMix, Fmix, and / or CowMix can be applied.

[0034] The domain consistency module (120) generates masked images for use in training to enhance consistency in model predictions. The domain consistency module (120) can generate masked target images by element-multiplying a mask randomly sampled from a uniform distribution with some images (e.g., a mixed target image) within the target superdomain.

[0035] FIG. 3 is an exemplary diagram referenced to explain the operation of a domain adaptation module according to one embodiment of the present disclosure.

[0036] The domain adaptation module (140) directly adapts the model (30) to the multi-target domain by using images of the target superdomain (e.g., original target images, style-transferred target images, and mixed target images included in each target domain) and masked images.

[0037] Referring to FIG. 3, the domain adaptation module (140) can train the model (30) using five losses: source loss (300), target loss (310), image loss (320), region loss (330), and masking loss (340).

[0038] The source loss (300) represents the loss between the model's (30) prediction for an image of the source domain (hereinafter, the source image) and the label of the source image. For example, if the task of the model (30) is a semantic segmentation task, the pixel-wise cross-entropy between the segmentation map predicted from the source image and the label can be used as the source loss (300).

[0039] Target loss (310), image loss (320), region loss (330), and masking loss (340) are calculated based on the prediction of the model (30) for the target images. The domain adaptation module (140) can generate pseudo-labels for unlabeled target images using the teacher model (32). The parameters of the teacher model (32) can be set based on the exponential moving average (EMA) of the parameters of the model (30). The domain adaptation module (140) can also apply a predetermined weight to the pseudo-labels in consideration of the reliability or quality of the pseudo-labels.

[0040] The target loss (310) represents the loss between the prediction of the model (30) for the original target image and the pseudo label for it. The pseudo label for the target loss (310) can be calculated based on the prediction of the teacher model (32) for the original target image. For example, the domain adaptation module (140) can input the same target image to the model (30) and the teacher model (32), respectively, and calculate the target loss (310) based on the outputs of the models (30 and 32). Depending on the implementation example, the target loss can also be calculated based on the prediction of the model (30) for the target image augmented by the source image and the pseudo label generated from the non-augmented target image.

[0041] Image loss (320) represents the loss between the prediction of the model (30) for the style-transferred target image (hereinafter, style-transferred segmentation map) and the pseudo-label. The pseudo-label for the image loss (320) can be generated based on the prediction of the teacher model (32) for the original target image before style transfer is applied.

[0042] The domain loss (330) represents the loss between the prediction of the model (30) for the mixed target image (hereinafter, the mixed segmentation map) and the pseudo label. The pseudo label for the domain loss (330) can be generated based on the prediction of the teacher model (32) for two or more original target images used to generate the mixed target image. When the original target images are mixed using a predetermined mask (e.g., a rectangular mask, a mask with a cowhide pattern, a mask randomly sampled in the frequency domain, etc.), the domain adaptation module (140) can apply the same mask to the predictions of the teacher model (32) for the original target image to generate the pseudo label.

[0043] The masking loss (340) represents the loss between the prediction of the model (30) for the masked target image (hereinafter, the masked segmentation map) and the pseudo label. The pseudo label for the masking loss (340) can be generated based on the prediction of the teacher model (32) for the unmasked target image. For example, the pseudo label applied to the mixed target image used to generate the masked target image can also be applied to the masked target image. Accordingly, the model (30) is trained to predict the segmentation map of the entire image including the masked portion. Due to the mixing-based domain integration described above in FIG. 2, the model (30) can be induced to learn how to infer the prediction result of the masked portion from the features of the original target image and the mixed target image. Since the masked target image contains only limited information about the target image before being masked, the prediction task for it is more difficult than that for the target image before being masked. In the present disclosure, by utilizing a masked image, a model (30) can be trained to predict the correct result using the remaining part without access to the entire image.

[0044] The total loss used for adaptation of the model (30) can be expressed as in mathematical expression 1. The domain adaptation module (140) can update the parameters of the model (30) in a direction in which the total loss is minimized.

[0045]

[0046] Here, is the source loss, is the target loss for the image of the i-th target domain, is the image loss for the image transferred from the i-th target domain to the style of the j-th target domain, is the region loss for the mixed image between the i-th target domain and the j-th target domain, is the masking loss for the masking image generated from the mixed image between the i-th target domain and the j-th target domain. Also, λ A , λ I , λ R , λ M are the weights applied to target loss, image loss, region loss, and masking loss, respectively.

[0047] FIG. 4 is an exemplary diagram referenced to explain the operation of a domain alignment module according to one embodiment of the present disclosure.

[0048] The domain alignment module (160) performs entropy-based domain alignment to minimize the intra-domain gap.

[0049] Target images collected in the real world have diverse distributions due to the influence of different countries, weather conditions, and circadian variations. This can lead to significant differences in prediction performance even when using the same model.

[0050] The domain alignment module (160) utilizes the entropy map of the model output to measure the uncertainty in the prediction of the target superdomain. The entropy map can be generated based on the pixel-wise entropies of the segmentation map.

[0051] Referring to FIG. 4, the domain alignment module (160) can reduce the gap between the source domain and the target superdomain (source-to-target superdomain gaps) while reducing the gap within the target superdomain by using a network including two pairs of generators and discriminators. For convenience of explanation, it is assumed below that all operations illustrated in FIG. 4 are performed by the domain alignment module (160), but some operations may also be performed by the domain adaptation module (140).

[0052] The network may include an inter generator (40), an inter discriminator (42), an intra generator (44), and an intra discriminator (46). The inter generator (40) and / or the intra generator (44) may correspond to the model (30) of FIG. 3.

[0053] The inter generator (40) generates a segmentation map from a source image and a target image. The inter discriminator (42) is trained to receive an entropy map generated from the segmentation map and predict the domain label of the image corresponding to the entropy map, while the inter generator (40) is trained to deceive the inter discriminator (42). The inter generator (40) and the inter discriminator (42) can be optimized to minimize segmentation loss (400) (e.g., cross entropy) and adversarial loss (420).

[0054] The domain sorting module (160) can divide the target domain into two subdomains, referred to as an easy subdomain and a hard subdomain, using an entropy map generated from target images. The domain sorting module (160) can calculate the average of each entropy map as a ranking score, and assign the target image to either the easy subdomain or the hard subdomain based on whether the ranking score of each target image falls within a preset ratio.

[0055] Because the target domain lacks labels, it is difficult to directly align the gaps between the easy subdomain and the hard subdomain. To address this issue, the domain alignment module (160) can generate pseudo labels based on the segmentation map predicted from the easy subdomain by the inter-generator (40). The pseudo labels can be, for example, converted into one-hot vectors for each entry in the segmentation map.

[0056] The intra generator (44) generates a segmentation map from the target image of the easy subdomain and the target image of the hard subdomain. To narrow the gap within the domain, the intra discriminator (46) is trained to receive an entropy map produced from the segmentation map and predict the split labels of the image corresponding to the entropy map, while the intra generator (44) is trained to deceive the intra discriminator (46). The intra generator (44) and the intra discriminator (46) can be optimized to minimize the segmentation loss (440) (e.g., cross entropy between the prediction and the pseudo label for the target image of the easy subdomain) and the adversarial loss (460).

[0057] FIG. 5 is a flowchart schematically illustrating a multi-target domain adaptation method according to one embodiment of the present disclosure.

[0058] The method illustrated in FIG. 5 can be implemented by having the functions of one or more components of the aforementioned MTDA device (10) executed by at least one computing device. The following description is given in terms of operations performed by the computing device.

[0059] A computing device integrates multiple different target domains into a target superdomain (S500). The target superdomain may include multiple original target images, each belonging to one of the multiple target domains.

[0060] The target superdomain may further include one or more first interpolated target images generated based on image-level domain interpolation between the plurality of target domains. The image-level domain interpolation may generate one or more intermediate domains that fill a style gap between the plurality of target domains. The image-level domain interpolation may include style transfer between the plurality of target domains. Each of the first interpolated images may be a first original target image of a first target domain, which is one of the plurality of target domains, transferred to a style of a second target domain that is distinct from the first target domain. The style transfer may be performed, for example, based on a GAN, a Fourier transform, a photometric method, or a combination thereof.

[0061] The target superdomain may further include one or more second interpolated target images generated based on region-level domain interpolation between the plurality of target domains. The region-level domain interpolation may generate one or more intermediate domains that fill a contextual gap between the plurality of target domains. The region-level domain interpolation may include blending between the plurality of target domains. Each of the second interpolated images may be a blend of a first original target image of a first target domain, which is one of the plurality of target domains, with a portion of a second original target image of a second target domain, which is distinct from the first target domain. The blending may include, for example, CutMix, Fmix, CowMix, or a combination thereof.

[0062] The computing device masks at least some images of the target superdomain to generate one or more masked target images (S520). The masked target images may be generated based on second interpolated target images. The computing device may generate the masked target images by element-multiplying at least some images with masks randomly sampled from a uniform distribution.

[0063] The computing device updates the model's parameters based on the model's predictions for the target superdomain and masked target images (S540). The model may be trained and / or adapted to perform image-based tasks. For example, the image-based tasks may include semantic segmentation tasks.

[0064] In step S540, the computing device may input original target images into a teacher model to generate pseudo labels for the first interpolated images, the second interpolated images, and the masked images. The teacher model may have parameters based on an exponential moving average of the parameters of the model. The parameters of the model may be updated in a direction that minimizes a weighted sum of source loss, target loss, image loss, region loss, and masking loss. The image loss may be a loss between the model's prediction and the pseudo label for each of the first interpolated images. The pseudo label for each of the first interpolated images may be generated from the teacher model's prediction for the first original target image used to generate the first interpolated image. The region loss may be a loss between the model's prediction and the pseudo label for each of the second interpolated images. The pseudo-label for each of the second interpolated images can be generated from a mixture of the predictions of the teacher model for each of the first original target image and the second original target image used to generate the second interpolated images. The masking loss can be a loss between the prediction of the model for each of the masked target images and the label. The pseudo-label for each of the masked target images can be generated based on region-level interpolation of the predictions of the teacher model for each of the first original target image and the second original target image corresponding to the second interpolated image used to generate the masked target image. Accordingly, the model can be trained to predict a segmentation map for the entire region of the unmasked second interpolated target images from the masked target images.

[0065] In step S540, the computing device may divide the target superdomain into a first subdomain and a second subdomain based on the prediction of the intra generator for the target superdomain. For example, the computing device may generate an entropy map for each of a plurality of target images included in the target superdomain from the prediction of the intra generator, assign a ranking score to each of the plurality of images based on the entropy map, and divide the target superdomain based on the ranking score. The computing device may pseudo-label the first subdomain using the prediction of the intra generator for the first subdomain. The computing device may train a model based on the prediction of the model for the first subdomain, and may train the model adversarially against the intra discriminator. The intra discriminator may be trained to distinguish between the prediction of the model for the first subdomain and the prediction of the model for the second subdomain, and the model may be trained to generate a prediction that can deceive the intra discriminator.

[0066] FIG. 6 is a block diagram schematically illustrating an exemplary computing device that can be used to implement a method or device according to the present disclosure.

[0067] The computing device (60) may include some or all of a memory (600), a processor (620), storage (640), an input / output interface (660), and a communication interface (680). The computing device (60) may structurally and / or functionally include at least a portion of the MTDA device (10). The computing device (60) may be a stationary computing device, such as a desktop computer or a server, as well as a mobile computing device, such as a laptop computer or a smart phone. The computing device (60) may be implemented with any specialized hardware accelerator capable of efficiently processing operations for an artificial intelligence model. For example, the computing device (60) may include a graphic processing unit (GPU), a Tensor Processing Unit (TPU), or a neural processing unit (NPU).

[0068] The memory (600) may store a program that causes the processor (620) to perform a method or operation according to various embodiments of the present disclosure. For example, the program may include a plurality of instructions executable by the processor (620), and the method illustrated in FIG. 5 may be performed by executing the plurality of instructions by the processor (620). The memory (600) may be a single memory or a plurality of memories. In this case, information required to perform the method or operation according to various embodiments of the present disclosure may be stored in a single memory or may be divided and stored in a plurality of memories. When the memory (600) is composed of a plurality of memories, the plurality of memories may be physically separated. The memory (600) may include at least one of a volatile memory and a nonvolatile memory. The volatile memory includes a static random access memory (SRAM) or a dynamic random access memory (DRAM), and the nonvolatile memory includes a flash memory.

[0069] The processor (620) may include at least one core capable of executing at least one instruction. The processor (620) may execute instructions stored in the memory (600). The processor (620) may be a single processor or multiple processors.

[0070] Storage (640) maintains stored data even when power supplied to the computing device (60) is cut off. For example, storage (640) may include non-volatile memory, or may include storage media such as magnetic tape, optical disk, or magnetic disk. A program stored in storage (640) may be loaded into memory (600) before being executed by processor (620). Storage (640) may store a file written in a programming language, and a program generated from the file by a compiler or the like may be loaded into memory (600). Storage (640) may store data to be processed by processor (620) and / or data processed by processor (620).

[0071] The input / output interface (660) may include input devices such as a keyboard, mouse, etc., and output devices such as a display device, printer, etc. A user may trigger execution of a program by the processor (620) and / or check the processing result of the processor (620) through the input / output interface (660).

[0072] The communication interface (680) may provide access to an external network. The computing device (60) may communicate with other devices via the communication interface (680).

[0073] Each component of the device or method according to the present invention may be implemented in hardware, software, or a combination of hardware and software. Furthermore, the functions of each component may be implemented in software, with a microprocessor executing the software functions corresponding to each component.

[0074] Various implementations of the systems and techniques described herein may be implemented as digital electronic circuits, integrated circuits, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations of one or more computer programs executable on a programmable system. The programmable system includes at least one programmable processor (which may be a special purpose processor or a general purpose processor) coupled to receive data and instructions from and transmit data and instructions to a storage system, at least one input device, and at least one output device. Computer programs (also known as programs, software, software applications, or code) include instructions for the programmable processor and are stored on a "computer-readable recording medium."

[0075] A computer-readable recording medium includes any type of recording device that stores data that can be read by a computer system. Such a computer-readable recording medium may be a non-volatile or non-transitory medium such as a ROM, CD-ROM, magnetic tape, floppy disk, memory card, hard disk, magneto-optical disk, storage device, and may further include a transitory medium such as a data transmission medium. Furthermore, the computer-readable recording medium may be distributed across network-connected computer systems, so that computer-readable code can be stored and executed in a distributed manner.

[0076] Although the flowchart / timing diagram of this specification describes each process as being executed sequentially, this is merely an illustrative description of the technical idea of ​​one embodiment of the present disclosure. In other words, a person of ordinary skill in the art to which one embodiment of the present disclosure belongs may modify and apply various modifications and variations by changing the order described in the flowchart / timing diagram without departing from the essential characteristics of one embodiment of the present disclosure, or by executing one or more of the processes in parallel. Therefore, the flowchart / timing diagram is not limited to a chronological order.

[0077] The above description is merely an example of the technical idea of ​​the present embodiment, and those skilled in the art will appreciate that various modifications and variations can be made without departing from the essential characteristics of the present embodiment. Therefore, the present embodiments are not intended to limit the technical idea of ​​the present embodiment, but rather to explain it, and the scope of the technical idea of ​​the present embodiment is not limited by these embodiments. The scope of protection of the present embodiment should be interpreted by the claims below, and all technical ideas within a scope equivalent thereto should be interpreted as being included in the scope of rights of the present embodiment.

[0078]

[0079] Statement regarding sponsored research or development

[0080] This invention is the result of research conducted with the support of the Korea Institute of Industrial Technology Evaluation and Planning with funds from the Ministry of Trade, Industry and Energy (Project Unique Number: 2001785100, Subproject Number: 20017826, Project Name: Development of 200 TOPS-class Scalable AI Accelerator Processor for Real-time Artificial Intelligence Inference, Contribution Ratio: 1 / 2), and with the support of the National IT Industry Promotion Agency with funds from the Ministry of Science and ICT (Project Unique Number: 1711193602, Subproject Number: 2020-0-01305-004, Project Name: Development of 2,000 TFLOPS-class Server Artificial Intelligence Deep Learning Processor and Module, Contribution Ratio: 1 / 2).

[0081]

[0082] CROSS-REFERENCE TO RELATED APPLICATION

[0083] This patent application claims priority to Korean Patent Application No. 10-2023-0133589, filed in Korea on October 6, 2023, and Korean Patent Application No. 10-2023-0176840, filed in Korea on December 7, 2023, the entire contents of which are incorporated herein by reference.

Claims

1. A computer implementation method for multi-target domain adaptation of a model performing image-based tasks, A process of integrating a plurality of different target domains into a target superdomain, wherein the target superdomain includes a plurality of original target images each belonging to one of the plurality of target domains, one or more first interpolated target images generated based on image-level domain interpolation between the plurality of target domains, and one or more second interpolated target images generated based on region-level domain interpolation between the plurality of target domains; A process of generating one or more masked target images by masking at least some images of the target superdomain; and A method comprising a process of updating parameters of the model based on predictions of the model for the target superdomain and the masked target images.

2. In paragraph 1, The above masked target images are generated based on the second interpolated target images, A method wherein the model is trained to predict a segmentation map for the entire region of the second interpolated target images that are not masked from the masked target images.

3. In paragraph 1, The above updating process is: A process of generating pseudo labels for the first interpolated images, the second interpolated images, and the masked target images based on predictions for the original target images of a teacher model having parameters based on an exponential moving average (EMA) of the parameters of the model. A method comprising:

4. In paragraph 3, A method wherein each of the first interpolated images is a first original target image of a first target domain, which is one of the plurality of target domains, transferred into a style of a second target domain that is distinct from the first target domain.

5. In paragraph 4, A method wherein the pseudo labels for the first interpolated images are generated based on the prediction of the teacher model for the first original target image.

6. In paragraph 3, A method wherein each of the second interpolated images is a mixture of a first original target image of a first target domain, which is one of the plurality of target domains, and a portion of a second original target image of a second target domain, which is distinct from the first target domain.

7. In paragraph 6, A method wherein the pseudo labels for the second interpolated images are generated based on region-level interpolation between the prediction of the teacher model for the first original target image and the prediction of the teacher model for the second original target image.

8. In paragraph 6, Each of the above masked target images is obtained by multiplying a predetermined mask by an unmasked second interpolated target image, which is one of the second interpolated target images, A method wherein the pseudo labels for the masked target images are generated based on region-level interpolation of the prediction of the teacher model for the first original target image and the prediction of the teacher model for the second original target image, corresponding to the unmasked second interpolated target image.

9. In paragraph 1, The above updating process is: A process of dividing the target superdomain into a first subdomain and a second subdomain based on the prediction of the inter generator for the target superdomain; A process of pseudo-labeling the first subdomain using the prediction of the inter generator of the first subdomain; and A process of training the model based on the prediction of the model for the first subdomain, and training the model adversarially against an intra discriminator, the intra discriminator being trained to distinguish between the prediction of the model for the first subdomain and the prediction of the model for the second subdomain. A method comprising:

10. In paragraph 9, The above separation process is, A process of generating an entropy map for each of a plurality of target images included in the target superdomain from the prediction of the above inter generator; A process of assigning a ranking score to each of the plurality of target images based on the entropy map; and A process of separating the target superdomain based on the above ranking score. A method comprising:

11. In paragraph 1, The above image-level domain interpolation is, A method for creating one or more intermediate domains that fill a style gap between the plurality of target domains.

12. In paragraph 1, A method wherein the image-level domain interpolation comprises style transfer between the plurality of target domains.

13. In paragraph 12, The above style transfer is performed based on Generative Adversarial Networks (GAN), Fourier transform, photometric methods, or a combination thereof.

14. In paragraph 1, The above domain-level domain interpolation is, A method for creating one or more intermediate domains that fill a contextual gap between the plurality of target domains.

15. In paragraph 1, A method wherein the above domain-level domain interpolation comprises mixing between the plurality of target domains.

16. In paragraph 15, The method wherein the above mixture comprises CutMix, Fmix, CowMix, or a combination thereof.

17. In paragraph 1, The above generating process is, A method comprising element-wise multiplying at least some of the images with masks randomly sampled from a uniform distribution.

18. In paragraph 1, A method wherein the image-based task includes a semantic segmentation task.

19. At least one memory for storing instructions; and at least one processor, At least one processor executes the instructions, Integrating a plurality of different target domains into a target superdomain, wherein the target superdomain includes a plurality of original target images each belonging to one of the plurality of target domains, one or more first interpolated target images generated based on image-level domain interpolation between the plurality of target domains, and one or more second interpolated target images generated based on region-level domain interpolation between the plurality of target domains; Masking at least some images of the target superdomain to generate one or more masked target images; A device configured to update parameters of the model based on predictions of the model for the target superdomain and the masked target images.

20. At least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the at least one processor to implement an artificial intelligence model that performs an image-based task, The artificial intelligence model is a device, wherein the target superdomain integrates a plurality of different target domains, the target superdomain includes a plurality of original target images each belonging to one of the plurality of target domains, one or more first interpolated target images generated based on image-level domain interpolation between the plurality of target domains, and one or more second interpolated target images generated based on region-level domain interpolation between the plurality of target domains, and a multi-target domain adaptation model adapted for the plurality of target domains based on one or more masked target images generated by masking at least some images of the target superdomain.

Images