Domain adaptation system and method

Abstract

To provide a domain adaptation system and a domain adaptation method for improving the accuracy of unsupervised domain adaptation by appropriately aligning a source instance and a target instance.SOLUTION: A method in a domain adaptation system 100 for adapting an object detection model trained on the basis of a source domain image to a target domain image includes: storing a plurality of source instances in a memory by category; selecting at least one source instance from among the plurality of source instances stored in the memory on the basis of a category of a target instance; aligning the selected source instance with the extracted target instance; and training the object detection model on the basis of a result of alignment.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a system for performing domain adaptation between a source domain and a target domain, and more particularly to a system for unsupervised domain adaptation to improve the accuracy of object detection in unlabeled target domain images. [Background technology]

[0002] Deep learning has attracted a great deal of attention in recent years. Deep learning has become a key component of many applications in computer vision. Object detection is considered one of the notable applications in deep learning and computer vision. There are numerous applications for the task of object detection, including object tracking, video surveillance, image captioning, and image segmentation.

[0003] In applications, the task of object detection involves recognizing and detecting various objects present in an image or video, labeling them, and classifying the objects. The main concept behind these processes is that every object has unique attributes. These attributes help distinguish one object from another.

[0004] Deep learning uses a multi-layer approach to extract high-level attributes from data, which object detection techniques use to localize and classify objects.

[0005] Deep learning-based object detectors can be broadly categorized into two classes: one-stage detectors and two-stage detectors. Two-stage detectors first obtain object region proposals through selective object search, and then classify each candidate. First-stage detectors directly predict category confidence and regress bounding boxes based on predefined anchors.

[0006] Although single-stage detectors are highly efficient, two-stage detectors are widely adopted to achieve higher performance. The good performance of these detectors heavily relies on large-scale training images containing bounding box annotations.

[0007] This limits generalization performance when faced with a new environment (target environment) where the object appearance, background, and even weather conditions differ significantly from those in the training images (source environment). Unfortunately, acquiring a sufficient number of annotated training images from a new environment is costly and time-consuming, which is a fatal drawback of existing approaches for real-world deployment.

[0008] For example, suppose an autonomous driving detection model trained on a dataset collected in Tokyo may not necessarily perform well on traffic images in Delhi due to variations in scene layout, weather conditions, etc. To mitigate this issue, training data should ideally be collected from a variety of environments to ensure all situations are already met.

[0009] However, collecting data from prototype autonomous vehicles driving in different cities, different weather conditions, and different seasons is difficult. Annotating all of this collected data is even more difficult, limiting the development of fully autonomous vehicles. Making more efficient use of training data is important not only for autonomous driving, but also for deep learning in general.

[0010] To overcome this problem, typical transfer learning approaches use a pre-trained feature extraction model trained on a large labeled dataset and then fine-tune the entire network on a task-specific dataset. However, collecting and annotating enough data to fine-tune the pre-trained network is still expensive and time-consuming.

[0011] Simulation environments, however, provide a gateway to the above problems. Simulators can provide nearly unlimited amounts of well-annotated data. Although simulation can bring great benefits, there will always be a reality gap between the simulated and real data, and simulators still lag behind in providing photorealistic rendering results.

[0012] One of the most common approaches to bridging the gap between simulation (i.e., source domain) and reality (i.e., target domain) is domain adaptation, where a data distribution in one domain is mapped to another domain in latent space by minimizing an explicitly defined domain disparity measure.

[0013] One widely investigated technique to address this problem is unsupervised domain adaptation (UDA), which attempts to transfer knowledge from a labeled source domain to an unlabeled target domain in an adversarial manner. Various methods have been developed along this line, and various algorithms have been proposed. hmm Align source and target domains using a mentation strategy hmm By training the model, we aim to learn invariant feature representations.

[0014] US-11106182-B2 (Patent Document 1) and Ganin, et al., "Domain adversarial training of neural networks," The Journal of Machine Learning Research, 2016 (Non-Patent Document 1) disclose a technique for achieving a domain-invariant feature space of a backbone neural network with the help of a neural network-based domain classifier.

[0015] For example, given a backbone feature extraction network F and a domain classifier network D, the task of network F is to minimize the task-specific loss (classification / segmentation / detection, etc.) and maximize the classifier loss, while the optimization of task network D is to minimize the domain classification loss.

[0016] Maximizing the classification loss by the network F is achieved with the help of a gradient reversal layer R, which is applied before the input to the domain classifier. The task of R is to act as an identity function during feedforward, and during backpropagation the gradient is multiplied by -1.

[0017] All methods utilize the strategy presented in [1] to adapt detector models using labeled source and unlabeled target domains and fall into the adversarial feature learning category.

[0018] The technology disclosed in Patent Document 1 uses adversarial feature learning in automatic speech recognition (ASR), and the technology disclosed in Non-Patent Document 1 uses adversarial feature learning in image classification tasks.

[0019] While the technique disclosed in [1] addresses UDA for image classification tasks, our task is to develop UDA techniques for object detection tasks. In the context of object detection, we typically present images of complex scenes in which the objects of interest, i.e., instances, occupy only a small region.

[0020] Therefore, it is crucial to consider the local nature of instance-level adaptation for detection tasks. In contrast, traditional domain adaptation methods typically consider the entire input image and ignore local regions of interest when attempting to fill the domain gap at the image level.

[0021] Recently, various domain adaptation methods have been developed for object detection tasks that attempt to learn invariant feature representations via source and target domain alignment using a variety of proposed alignment strategies. Specifically, they are based on tuning both image-level features from a backbone network and instance-level features extracted from object proposals using adversarial training.

[0022] Chen, et al., "Domain adaptive faster R-CNN for object detection in the wild," CVPR, 2018 (Non-Patent Document 2) is the first to formulate and address the problem of domain-adaptive object detection by aligning image-level and instance-level features. Specifically, given the UDA problem setting, this method utilizes the Faster R-CNN detection framework and applies adversarial learning to both image-level features extracted from the backbone and instance-level features extracted from the RCN network.

[0023] At the image level, we perform adversarial training by classifying backbone features into two domains using a domain classification loss. Similarly, at the instance level, a domain classification loss is used to classify instances obtained from a region proposal network (RPN) into two domains.

[0024] Xu, et al., “Cross-domain detection via graph-induced prototype alignment.”, CVPR, 2020 (Non-Patent Document 3), and Rezaeianaran, et al., “Seeking similarities over differences: Similarity-based domain alignment for adaptive object detection.”, ICCV, 2021 (Non-Patent Document 4) are recent domain adaptation methods for object detection that follow a two-stage approach for instance-level adaptation.

[0025] In the first step, these non-patent documents aggregate instances into a variable number of feature groups based on visual similarity (Non-Patent Document 4) or overlapping criteria (Non-Patent Document 3), and in the second step, adversarial training or contrastive learning (Non-Patent Document 3) is applied to align the mean embeddings of the groups extracted from the source and target domains based on categorical information. [Prior art documents] [Patent documents]

[0026] [Patent Document 1] U.S. Patent No. 11,106,182 [Non-patent literature]

[0027] [Non-Patent Document 1] Ganin, et al., “Domain adversarial training of neural networks.”, The Journal of Machine Learning Research, 2016 [Non-patent document 2] Chen, et al.,“Domain adaptive faster r-cnn for object detection in the wild.”,CVPR,2018 [Non-patent document 3] Xu, et al,“Cross-domain detection via graph-induced prototype alignment.”,CVPR,2020 [Non-patent document 4] Rezaeianaran, et al,“Seeking similarities over differences: Similarity-based domain alignment for adaptive object detection.”,ICCV,2021 Summary of the Invention [Problem to be solved by the invention]

[0028] However, the above-mentioned background techniques have the following drawbacks: Although it is usually assumed that the two domains share the same label space, it is not guaranteed that the samples extracted from the source and target domains cover the same class of objects in a mini-batch.

[0029] Second, even if the objects in the source and target samples belong to the same class, their scales and orientations may be very different, i.e., intra-class variance, which leads to the align- ment hmm It becomes difficult to find the best source and target instance pairs for each object. Due to these drawbacks, the reliability of object detection models on unlabeled target domain images has not yet been established.

[0030] The present invention appropriately aligns the source and target instances. hmm The objective of this study is to provide a system that can improve the accuracy of unsupervised domain adaptation by implementing the method. [Means for solving the problem]

[0031] To achieve the above object, the present invention provides a domain adaptation method for adapting an object detection model trained on a source domain image to a target domain image, the method comprising: Image features of the source domain image Source instance multiple The source instances are extracted from the target domain image, and the source instances are stored in a memory for each category. These are the image features of the target domain image. extracting a target instance and, based on a category of the target instance, selecting a plurality of source instances stored in the memory; Among them, those belonging to the same category as the target instance Source instance At least one and aligning the selected source instance to the extracted target instance. hmm and hmm The object detection model is trained based on the results of the analysis. [Effects of the Invention]

[0032] According to the present invention, the source instance and the target instance are appropriately aligned. hmm By implementing this method, it is possible to improve the accuracy of unsupervised domain adaptation and provide a domain adaptation system. [Brief explanation of the drawings]

[0033] [Figure 1] The domain adaptation system is comprised of a computer system, and FIG. 1 is a hardware block diagram of an embodiment of the computer system. [Figure 2] FIG. 1 is a block diagram illustrating the operation of a domain adaptation system. [Figure 3] FIG. 2 is a detailed block diagram of the architecture of the storage system of the source instance. [Figure 4]3 is an example of a graphical user interface (GUI) for using the system of FIG. 2. [Figure 5] FIG. 10 is an example block diagram for searching a storage area of ​​a source instance. [Figure 6] FIG. 10 is an example of a functional block diagram for extracting a target instance from a target image. DETAILED DESCRIPTION OF THE INVENTION

[0034] Next, we describe an embodiment of a domain adaptation system according to the present invention, which includes training a pre-trained object detection model for unsupervised domain adaptation based on a source domain dataset and a target domain dataset, such that the domain gap between a target domain (e.g., unlabeled traffic images of Delhi) and a source domain (e.g., labeled traffic images of Tokyo) is bridged.

[0035] The training results in a new set of parameters for the object detection model that are aligned to the domain gap between the datasets from the two domains. During the inference phase, given an input of a Delhi traffic image, the trained object detection model predicts bounding boxes with class labels for all objects in the image.

[0036] In unsupervised domain adaptation for object detection, we are given Ns labeled images for a source domain S (Equation 1 below), where y i s and B i s are the class label and bounding box coordinates, respectively.

[0037]

number

[0038] For the target domain T (Equation 2 below), N T Only unlabeled images are available. Both domains share the same label space, but their visual distributions are different. The domain adaptation system trains an object detection model to improve its object detection capabilities for the target domain despite the domain gap.

[0039]

number

[0040] The domain adaptation system comprises a computer system. A hardware block diagram of the computer system is shown in Figure 1. The domain adaptation system 100 includes one or more processors 701, volatile memory 703, non-volatile memory 704, a display device 713, input / output devices 712, cursor control devices 711 such as a keyboard and pointing device, a storage device 710 including a storage medium 708, and a signal generating device 707 communicatively coupled to an address / data bus 702.

[0041] Bus 702 is shown as an abstraction representing one or more physical buses, connected by appropriate bridges or controllers, and address / data buses configured to communicate information.

[0042] Thus, the address / data bus 702 can include a system bus, a Peripheral Component Interconnect (PCI) bus or PCI-Express bus, a HyperTransport or Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), an IIC (12C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus (also known as "Firewire").

[0043] The processor 701 is configured to process information and instructions and may include a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), and / or other hardware devices for processing data.

[0044] The domain adaptation system 100 is configured to utilize one or more data storage units and may include a volatile memory unit 703, such as random access memory RAM, static RAM, or dynamic RAM, coupled to the address / data bus 702. The volatile memory unit 703 is configured to store information and instructions for the processor 701.

[0045] The domain adaptation system 100 may further include a non-volatile memory 704, and the storage medium 708 (also referred to as a "machine-readable medium") is shown as a single medium. The terms "machine-readable medium" and "storage medium" are interpreted to include a single medium or multiple media that store one or more sets of instructions 709. The non-volatile memory 704 is configured to store static information and instructions for the processor 701. Furthermore, the computer system (domain adaptation system) 100 includes a main memory 705 that stores instructions (programs) 706. The storage medium 708 also stores instructions 709.

[0046] Input / output devices 712 are coupled to the address / data bus 702. The input / output devices are configured to provide information and command selections to the processor 701. The input devices may include alphanumeric keys and / or function keys. The computer system (domain adaptation system) 100 may include a cursor control device 711 programmed to send user input and / or commands to the processor 701. The display device 713 is set up to display video or graphics and may be a cathode ray tube (CRT), liquid crystal display (LCD), field emission display (FED), plasma display, or other display device capable of displaying video and / or graphic images.

[0047] Figure 2 shows a block diagram for explaining the operation of the domain adaptation system 100. Functional blocks among the multiple blocks are realized by the processor 701 executing a program. The same applies to functional descriptions in the following explanation. Note that functional blocks may be configured from hardware such as dedicated integrated circuits.

[0048] The domain adaptation system 100 according to the present invention appropriately aligns an instance of a source domain image (source instance) with an instance of a target domain image (target instance) in unsupervised domain adaptation. hmm This is the first example of using memory storage to record source instances by category in order to facilitate this. The source instance storage area is configured in the storage medium 708 of the storage device 710.

[0049] The storage area for source instances is a dynamic memory with multiple slots, and each slot records a source instance, which is classified by category. When the processor 701 starts domain adaptation, it creates a memory storage once and then repeatedly updates the storage at regular intervals.

[0050] The processor 701 checks the number of iterations (Iter) of learning for domain adaptation (102). Whenever Iter matches a predefined value (M), it starts the process of storing the source instance in the storage area; otherwise, it executes the steps following step 108.

[0051] When the processor 701 makes a positive decision in the block of step 102, it accesses the source domain dataset 103 and repeatedly passes the labeled source images of the source domain dataset 103 through the backbone 104 of the object detection model (first model) 112. The backbone 104 of the first model extracts instances from the given source images, and then classifies the features of the labeled instances into categories such as cars, people, trucks, etc., and stores them in the memory storage 710.

[0052] Figure 3 is a detailed block diagram of the architecture of the source instance storage system. The first model backbone (feature extractor) 104 extracts image features (instances) from the input source image (source domain dataset) 103, and pools instances 203 corresponding to annotated bounding boxes 201 using pooling 202. The pooled instances 203 then pass through a box (regression) head 204, which flattens the instances 203. Finally, the memory-flattened source instances 205 are stored by class (category) in different memory slots of a storage medium 708.

[0053] In Figure 2, processor 701 iterates the selection of mini-batches of source and target images from source domain dataset 103 and target domain dataset 107 if condition 102 is not met. Data augmentation 108 is applied to the selected mini-batch images, resulting in strongly and weakly augmented source images 109, a strongly augmented target image 110, and a weakly augmented target image 111.

[0054] A first model 112 (a pre-trained Faster R-CNN) predicts object bounding boxes and labels for the source image 109. The predicted results and annotations are then used to compute a supervised loss 113.

[0055] In the next step, a replica 112A of the first model (object detection model) 112 predicts object bounding boxes and labels for the input 110. An unsupervised loss 114 is calculated based on the prediction results of the first model 112 and the pseudo-labels 119.

[0056] Based on the input 111 (weakly enhanced target image), pseudo labels 119 are obtained by a second model 118. The network architecture of the second model 118 is the same as the first model 112, except that the weights of the second model 118 are iteratively updated at a fixed interval (N) 116 using an exponential moving average (EMA) 117.

[0057] Domain adaptation is performed at the image level with the help of a domain loss 115, and at the instance level by an attention-weighted contrastive loss 118A. The input to the domain loss 115 are the features of the first model 112 for the source and target domain images.

[0058] The processor 701 aligns the target instance to the most similar source instance from all source instances of the same category according to an attention-based guiding function (memory search approach), and the loss 118A is calculated from these target domain instances and source domain instances.

[0059] Unlike a mini-batch, where the source and target instances may not belong to the same category, the storage medium 708 can store the features of all source instances in advance, so that it is possible to not only obtain source instances from the same category as the target instances, but also obtain the most matching instance among them for alignment. hmm You can comment.

[0060] The processor 701 calculates gradients (W1, W2, W3, W4) based on each of the four losses described above, and combines the four gradients to continuously update the weights for the object detection model (first model) 112 at each learning process (626). The processor 701 sets the weights to the parameters of the object detection model when the number of iterations of the learning process reaches a specified number.

[0061] The domain adaptation system has three main components: a first model 112 containing Faster R-CNN, which takes a source image and generates bounding boxes and labels for all object instances present in the image.

[0062] The second component is a second model 118 that includes a replica 112A of the first model, which takes target images as input and generates pseudo-labels 119 that are used to train the first model 112 (112A). These two models 112, 118 are trained in a student-teacher setting, where the first model 112 updates its learned knowledge and feeds it back to the second model 118 via an exponential moving average (EMA).

[0063] The third component involves learning domain-invariant features at the image and instance level. At the image level, adversarial training is utilized to align a global image representation in a backbone network. hmm Promote ment.

[0064] At the instance level, the proposed memory-based aligner hmm Use the alignment system to align each object instance. hmm Therefore, the overall training goal of the present invention can be stated as follows:

[0065] ζ=ζ det +λ1ζ mut +λ2ζ img +λ3ζ inst where ζ det is the supervised training loss of the source dataset of the first model, ζ mut is the unsupervised training loss of the first model trained on the pseudo-labels generated by the second model for the target image, ζ img and ζ inst are the image-level and instance-level domain adaptation losses, respectively, and λ1, λ2, and λ3 are trade-off parameters.

[0066] Figure 4 shows a graphical user interface (GUI) for using the system of Figure 2. This GUI is used to upload a source domain dataset 801, a target dataset 802, and a pre-trained object detection model 803. A set of parameters for running the system include memory size 804, a confidence threshold 805, and a memory update frequency 806.

[0067] In the next step, domain adaptation is started by clicking Domain Adaptation 808. When domain adaptation stops after a predefined maximum number of iterations based on the model parameters 809 of the source domain, the GUI outputs updated model parameters 810 of the detection model trained to adapt to the target data distribution of the target domain.

[0068] 5 is an example of a functional block diagram for searching a source instance storage area (storage medium 708). This search system has two inputs: a source instance storage area (source memory) and a target box feature (category) 507. For the two inputs, the class-based source feature set search system 401 retrieves all source instances in the same category as the target instance from the source instance storage area 708.

[0069] Next, a vector dot product 402 is calculated between the target instance 507 and all source instances of the same category 403. The result of this operation is an attention value 404. A Softmax operation 405 is then applied to obtain an attention score 406. Based on the attention score, a source memory slot 408 is selected to store the top k (407) source instances to match the target instance.

[0070] The processor 701 then selects a slot for the source instance that is most similar to the target instance by Attention Weighted Contrastive Learning based on the top k (407) source instances (category class N) 408 and target domain (category) 409. The processor 701 calculates an Attention Weighted Contrastive Loss 118A based on the source instance that is most similar to the target instance and the target instance.

[0071] 6 is a functional block diagram for extracting features 513 of a target instance from a target image 110 in a target domain and calculating an unsupervised loss 114. First, a first model (feature extractor) 112 extracts features of the target image 110, and then these target features 502 are passed through a region proposal network (RPN) 503, which generates regions of interest on the target image 504.

[0072] Next, ROI pooling (first model) 505 is applied to the extracted features to obtain pooled target features 506. The pooled features 506 are passed through a box head (first model) 507, which flattens the features. A detection head (first model) 509 takes the smoothed features 508 as input and detects bounding boxes 511, which are filtered by 510. Box features 513 corresponding to these filtered boxes 511 are selected by a filter section 512.

[0073] Once the processor 701 has searched the top-k source memory slots corresponding to the target instances, the next most important step is to align those features in order to improve detection performance for unlabeled target images. To this end, a novel attention-weighted contrastive loss is utilized. Unlike traditional pairwise contrastive losses, which push features of the same class closer together across domains and push features belonging to different classes apart, the processor 701 uses attention scores to harmonize this alignment process. As a result, the domain adaptation system 100 can realize training of pre-trained models on unlabeled datasets for object detection tasks.

[0074] The present invention is not limited to the configurations described in the embodiments, and can be appropriately configured based on the content described in the claims. The above-described embodiments are examples for implementing the present invention. When implementing the present invention, only a part of the configurations of the embodiments may be implemented. [Explanation of symbols]

[0075] 100: Domain adaptation system, 104, 112, 118: Object detection model, 701: Processor, 710: Storage device (source instance storage area)

Claims

1. 1. A domain adaptation system for adapting an object detection model trained on a source domain image to a target domain image, comprising: The processor executes a program to extracting a plurality of source instances, which are image features of the source domain image, from the source domain image, and storing the plurality of source instances in a memory by category; extracting a target instance, which is an image feature of the target domain image, from the target domain image, and selecting at least one source instance that belongs to the same category as the target instance from among the multiple source instances stored in the memory based on the category of the target instance; aligning the selected source instances to the extracted target instances; training the object detection model based on the results of the alignment; Domain-adaptive systems.

2. The memory comprises a plurality of memory chunks, and each of the plurality of source instances is The plurality of memory chunks are stored by category, The domain adaptation system of claim 1 .

3. The processor: selecting, from the memory, a source instance that is most similar to the target instance from among the plurality of source instances that are in the same category as the target instance based on the category of the target instance; training the object detection model based on the target instances and the selected source instances; The domain adaptation system of claim 1 .

4. The processor: determining an attention-weighted contrastive loss calculated from the target instances and the selected source instances for training the object detection model; The domain adaptation system of claim 3 .

5. The processor: updating the weights of the object detection model based on the attention-weighted contrastive loss; The domain adaptation system of claim 4.

6. The processor: selecting a source instance that is most similar to the target instance based on an attention score between each of the plurality of source instances and the target instance; The domain adaptation system of claim 5.

7. 1. A domain adaptation method for adapting an object detection model trained on a source domain image to a target domain image, comprising: The computer extracting a plurality of source instances, which are image features of the source domain image, from the source domain image, and storing the plurality of source instances in a memory by category; extracting a target instance, which is an image feature of the target domain image, from the target domain image, and selecting at least one source instance that belongs to the same category as the target instance from among the multiple source instances stored in the memory based on the category of the target instance; aligning the selected source instances to the extracted target instances; training the object detection model based on the results of the alignment; Domain adaptation methods.

Images