Apparatus and Method for Domain Adaptive Semantic Segmentation Learning Using Bi-directional Contrastive Learning

Abstract

The disclosed embodiment provides a semantic segmentation learning apparatus and method capable of training to achieve semantic segmentation performance equivalent to that of real images even with a synthetic dataset by inputting a source image of a source domain and a target image of a target domain into a source neural network and a target neural network, respectively, which perform semantic segmentation, to obtain a source feature map and a target feature map; classifying the source pixel features and target pixel features of each of the source feature map and the target feature map according to class to obtain a source prototype and a target prototype representing representative features of each class; and backpropagating a bidirectional contrast loss obtained by calculating a forward contrast loss based on the similarity between the source prototype and the target pixel features and a backward contrast loss based on the similarity between the target prototype and the source pixel features.

Description

Technology Field

[0001] The disclosed embodiments relate to a semantic segmentation learning apparatus and method, and more specifically, to a domain-adaptive semantic segmentation learning apparatus and method using bidirectional contrast learning. Background Technology

[0002] Semantic segmentation (also known as semantic image segmentation) refers to the process of assigning an identifiable class to each of the multiple pixels in an input image to divide it into regions corresponding to each class, and it can be applied to various fields such as autonomous driving, medical imaging, and image editing. This semantic segmentation aims to classify and label each of the multiple pixels in the input image into a designated class, such as a person, car, or bicycle.

[0003] Currently, most semantic segmentation devices are implemented using semantic segmentation neural networks based on artificial neural networks. Semantic segmentation neural networks require prior training. While supervised learning-based models utilizing training images with pre-labeled pixel-by-pixel classes demonstrate good performance, obtaining the labels for training requires a significant amount of time. Consequently, methods utilizing synthetic datasets, which allow for the easy acquisition of images and their corresponding labels, are being employed as an alternative. For instance, programs that generate graphic images, such as simulations or game programs, can be used to acquire synthetic datasets. However, models trained on synthetic datasets have a limitation in that they struggle to properly segment real-world images due to the domain difference between graphic images and images acquired from reality. Prior art literature

[0004] Korean Registered Patent No. 10-2335702 (Registered Dec. 01, 2021) The problem to be solved

[0005] The disclosed embodiments aim to provide a domain-adaptive semantic segmentation learning device and method that enables a synthetic dataset to achieve semantic segmentation performance similar to that of real images using bidirectional contrast learning.

[0006] The disclosed embodiments aim to provide a semantic segmentation learning device and method capable of further improving learning performance by not only performing class-specific feature alignment through comparison between each pixel and a prototype, but also generating accurate pseudo-labels by progressively updating the prototype. means of solving the problem

[0007] The semantic segmentation learning device according to the embodiment inputs a source image of a source domain and a target image of a target domain into a source neural network and a target neural network, respectively, which perform semantic segmentation, to obtain a source feature map and a target feature map; separates the source pixel features and target pixel features of each of the source feature map and the target feature map according to class to obtain a source prototype and a target prototype representing representative features for each class; and backpropagates a bidirectional contrast loss obtained by calculating a forward contrast loss based on the similarity between the source prototype and the target pixel features and a backward contrast loss based on the similarity between the target prototype and the source pixel features.

[0008] The semantic segmentation learning device can obtain the source prototype for each class by average pooling the source pixel features extracted from the source feature map corresponding to each pixel of the source image and separated by class, and obtain the target prototype for each class by average pooling the target pixel features extracted from the target feature map corresponding to each pixel of the target image and separated by class.

[0009] The semantic segmentation learning device can distinguish the class of the source pixel feature based on the truth value labeled in the source image, and distinguish the class of the target pixel feature based on a pseudo-truth value that synthesizes the static truth value according to the semantic segmentation result of the target neural network for the target image and the dynamic truth value obtained according to the similarity between the target pixel feature and the correction prototype obtained by transferring the source prototype for each class to the target domain.

[0010] The semantic segmentation learning device can obtain a static truth value by setting the class having a matching probability exceeding the threshold value as the static pixel truth value of the corresponding pixel when the matching probability of a class identified pixel by pixel by the target neural network performs a neural network operation on the target image exceeds the threshold value.

[0011] The semantic segmentation learning device calculates a bias for each class based on the difference between the accumulated source prototype and the accumulated target prototype for the same class, which are accumulated by classifying the source prototypes repeatedly acquired during the learning process and classifying the target prototypes by class and classifying the target prototypes; adds the calculated bias to the source prototype of the corresponding class to obtain a corrected prototype that has been transferred to the target domain from the source prototype; and sets the class having the highest similarity among the similarities between the corrected prototype obtained for each class and the target pixel feature, and the similarity exceeding a threshold value, as the dynamic pixel truth value of the corresponding pixel to obtain the dynamic truth value.

[0012] The semantic segmentation learning device can obtain the pseudo-truth value by labeling the set dynamic pixel truth value as the pseudo-pixel truth value for each pixel of the target image that has a set dynamic pixel truth value constituting the dynamic truth value, and labeling the set static pixel truth value as the pseudo-pixel truth value for pixels that have not a set dynamic pixel truth value constituting the static truth value among pixels that have a set static pixel truth value constituting the static truth value.

[0013] The semantic segmentation learning device can periodically acquire and update the pseudo-truth value according to the number of iterations of learning, or when the source image or the target image changes.

[0014] The semantic segmentation learning device can calculate by setting the forward contrast loss such that if the class of the target pixel feature and the class of the source prototype are of the same class, it is accumulated as a positive value, and if they are of different classes, it is accumulated as a negative value, and can calculate by setting the backward contrast loss such that if the class of the source pixel feature and the class of the target prototype are of the same class, it is accumulated as a positive value, and if they are of different classes, it is accumulated as a negative value.

[0015] A semantic segmentation learning method according to an embodiment comprises: a step of acquiring a source feature map and a target feature map by inputting a source image of a source domain and a target image of a target domain into a source neural network and a target neural network, respectively, which perform semantic segmentation; a step of acquiring a source prototype and a target prototype representing representative features for each class by classifying the source pixel features and target pixel features of the source feature map and the target feature map, respectively, according to class; and a step of backpropagating a bidirectional contrast loss obtained by calculating a forward contrast loss based on the similarity between the source prototype and the target pixel features and a backward contrast loss based on the similarity between the target prototype and the source pixel features. Effects of the invention

[0016] Accordingly, the semantic segmentation learning device and method according to the embodiment utilize bidirectional contrast learning to enable a synthetic dataset to achieve semantic segmentation performance similar to that of a real image, and not only perform class-specific feature alignment through contrast between each pixel and a prototype representing a representative feature of each class, but also further improve learning performance by generating accurate pseudo-labels through progressively updating the prototype. Brief explanation of the drawing

[0017] FIG. 1 shows a schematic configuration of a semantic partitioning learning device according to one embodiment, divided according to the operation performed. Figure 2 is a diagram illustrating the operation of each component of the bidirectional contrast learning module of Figure 1. Figure 3 shows an example of the detailed configuration of the pseudo-label acquisition module of Figure 1. Figure 4 is a diagram illustrating the operation of the pseudo-label acquisition module of Figure 3. Figure 5 shows an example of dynamic pseudo-labels and hybrid pseudo-labels updated by the pseudo-label acquisition module of Figure 3. FIG. 6 illustrates a semantic partitioning learning method according to one embodiment. FIG. 7 is a diagram illustrating a computing environment including a computing device according to one embodiment. Specific details for implementing the invention

[0018] Hereinafter, a specific embodiment of one embodiment will be described with reference to the drawings. The following detailed description is provided to facilitate a comprehensive understanding of the methods, devices, and / or systems described herein. However, this is merely illustrative and the invention is not limited thereto.

[0019] In describing the embodiments, if it is determined that a detailed description of known technology related to the present invention might unnecessarily obscure the essence of the embodiment, such detailed description will be omitted. Furthermore, the terms described below are defined with consideration of their functions in the present invention, and these may vary depending on the intentions or practices of the user or operator. Therefore, such definitions should be based on the content throughout this specification. Terms used in the detailed description are intended merely to describe the embodiments and should not be limiting. Unless explicitly stated otherwise, expressions in the singular form include the meaning of the plural form. In this description, expressions such as "include" or "comprise" are intended to refer to certain characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof, and should not be interpreted to exclude the existence or possibility of one or more other characteristics, numbers, steps, actions, elements, parts thereof, or combinations thereof other than those described. Additionally, terms such as "...part," "...unit," "module," and "block" described in the specification refer to a unit that processes at least one function or operation, and this may be implemented in hardware, software, or a combination of hardware and software.

[0020] FIG. 1 shows a configuration schematically divided according to the operation performed by a semantic partitioning learning device according to one embodiment, and FIG. 2 is a diagram for explaining the operation of each configuration of the bidirectional contrast learning module of FIG. 1.

[0021] Referring to FIG. 1, a semantic segmentation learning device according to an embodiment may include a data acquisition module (10), a source neural network (20), a target neural network (30), and a learning module (40).

[0022] The data acquisition module (10) acquires an image to be subjected to semantic segmentation. As described above, semantic segmentation assigns an identifiable class to each of the multiple pixels in the input image to divide the area of ​​the input image by each class, and accordingly, the data acquisition module (10) acquires an image (x) to be divided into areas by identifiable classes. S , x T Acquires ).

[0023] In the embodiment, the data acquisition module (10) acquires an image as training data for training a target neural network (30), which is a semantic segmentation device. Here, the data acquisition module (10) can acquire an image of the actual reality. However, for the real-world image to be used as training data, the class to be identified for each pixel of the image must be labeled as a ground truth value, but it is very difficult to acquire an image with a ground truth value labeled. Accordingly, in the embodiment, the data acquisition module (10) can acquire both the actual image and a synthetic image with a ground truth value that is easily acquired and labeled as training data.

[0024] Accordingly, the training data acquisition module (10) may include a source data acquisition module (11) and a target data acquisition module (12) as shown in FIG. 1. The source data acquisition module (11) is a source image (x S ) obtain the source image (x S ) is not an image of the target domain that is subject to semantic segmentation during the actual operation of the learned semantic segmentation device, but rather an image of a source domain different from the target domain; it may be a synthetic image virtually generated so that pre-identified classes for each pixel can be easily labeled. Here, the synthetic image may be a virtual image generated by a computer simulation method, for example, an image generated by various games or simulation programs. Source image (x S When ) is acquired as a virtual synthetic image, the source image (x S) has a class assigned to each pixel at creation, and the truth value (y S ) can be obtained in a labeled state.

[0025] The target data acquisition module (12) is an image of the target domain, i.e., a target image (x) on which the trained target neural network (30) must perform semantic segmentation. T ) acquires. At this time, the target image (x T ) may be a general image where the class for each pixel is not labeled.

[0026] The source data acquisition module (11) and the target data acquisition module (12) each have a source image and a target image (x T It is implemented as a storage medium that stores ) or source image (x) from an external device S ) and target image(x T It may be implemented as a communication module that receives the data, but is not limited thereto. For example, the source data acquisition module (11) may be implemented as a synthetic image generation module, and the target data acquisition module (12) may be implemented as a camera module, etc.

[0027] The source neural network (20) and the target neural network (30) each have a source image (x S ) and target image(x T A semantic segmentation device that receives ) and performs semantic segmentation, is implemented as an artificial neural network, and can be implemented as a neural network having the same structure. The source neural network (20) receives a source image (x) from a source data acquisition module (11). S ) is received and semantic segmentation is performed by neural network operation, and the target neural network (30) receives the target image (x) from the target data acquisition module (12). S It receives ) and performs semantic partitioning using neural network operations.

[0028] The source neural network (20) and the target neural network (30) may each include a feature extractor (21, 31) and a classifier (22, 32). In the source neural network (20), the source feature extractor (21) is a source image (xS ) is input, neural network operations are performed, and the source feature map(f) S ) is extracted, and the source classifier (22) extracts the source feature map (f S ) receives ) and performs neural network operations to source image(x S By identifying the class (c ∈ C (where C is the set of classes to be identified)) for each pixel of ), and assigning the class (c) based on the identified result to the corresponding pixel, the source image (x S ) is divided by class. At this time, the source classifier (22) is the source image (x S Estimate the probability that each pixel (p) of ) is matched to each of the multiple classes (c), and assign the class (c) with the highest probability among the estimated probabilities to the corresponding pixel (p).

[0029] Since the target neural network (30) has the same structure as the source neural network (20), the target feature extractor (31) is similar to the source neural network (20) and the target image (x T ) is input, neural network operations are performed, and the target feature map (f) is input. T ) extracts, and the target classifier (32) extracts the target feature map (f T ) receives ) and performs neural network operations to target image(x T By identifying the class (c) for each pixel of ) and assigning the class (c) based on the identified result to the corresponding pixel, the target image (x T ) is divided by class. The target classifier (32) also divides the target image (x T Estimate the probability that each pixel (p) of ) is matched to each of the multiple classes (c), and assign the class (c) with the highest probability among the estimated probabilities to the corresponding pixel (p).

[0030] In the semantic segmentation learning device of FIG. 1, the source neural network (20) is a source image (x) of a source domain unrelated to actual application. S It is a device that performs semantic segmentation on ), and a target image (x) of a target domain in which only the target neural network (30) is actually utilized.T It is a semantic segmentation device that performs semantic segmentation by receiving ). However, the target neural network (30) is a target image (x T In order to perform semantic partitioning normally on ), training must be performed prior to it. In the semantic partitioning learning device of the example, the truth value (y S A source image (x) in a source domain that can be easily obtained S A target neural network (30) is trained based on ), and to this end, a source image (x S A source neural network (20) that performs semantic segmentation by receiving ) is additionally provided. That is, all components other than the target neural network (30) in FIG. 1 are components that are used only during training to train the target neural network (30).

[0031] Source image (x) obtained from the source data acquisition module (11) S ) is the truth value (y S Since ) can be pre-labeled, the source neural network (20) is the source image (x S The truth value (y) labeled for each pixel of ) S It can be pre-trained according to a supervised learning method using ). For example, the source neural network (20) can be trained based on classification loss and cross-entropy loss. Here, it is assumed that the source neural network (20) is already trained to perform semantic segmentation on images of the source domain, and the target neural network (30) is initialized by receiving the weights of the trained source neural network (20). That is, the target neural network (30) can be initially trained as the same neural network that shares weights with the source neural network (20).

[0032] The learning module (40) is configured to train the target neural network (30) together with the pre-trained source neural network (20). The initially trained target neural network (30) is, like the source neural network (20), a source image (x) of the source domain. S Semantic segmentation can be performed on ), but the target image (x) of the target domainT It is difficult to perform semantic segmentation with the required performance for ). Accordingly, the learning module (40) allows the target neural network (30) to perform target image (x T It is trained to accurately perform semantic segmentation on ), and in the embodiment, the training module (40) is a source image (x S ) and target image(x T By ensuring that features for each class (c) are aligned with each other at the pixel (p) level for each angle, the target neural network (30) can be trained based on a bidirectional contrast learning method so that class-specific features can be obtained regardless of the domain.

[0033] The learning module (40) may include a source label acquisition module (41), a pseudo-label acquisition module (42), a bidirectional comparison module (43), and a backpropagation module (44).

[0034] The source label acquisition module (41) is a source image (x S Source feature map (f) according to the class for each pixel of ) S By distinguishing ), the source prototype acquisition module (51) of the bidirectional comparison module (43) then obtains a source prototype (ρ) representing a representative characteristic of each class according to each class (c) in the source domain. S (c)) Source image (x) to obtain S Truth value (y) labeled in ) S ) obtain the source image (x S Truth value (y) for ) S As mentioned above, ) is the source image (x S Since it is pre-labeled in ), the source label acquisition module (41) is the source image (x S Truth value (y) labeled in ) S You can obtain ) as is.

[0035] The pseudo-label acquisition module (42) is configured to correspond to the source label acquisition module (41), and the target prototype acquisition module (52) of the bidirectional comparison module (43) represents a target prototype (ρ) that exhibits class-specific representative features according to each class (c) in the target domain. T (c)) to acquire the target image (x T The truth value for each pixel of ) must be obtained. However, the source image (x S Unlike ), the target image (x T The truth value for ) cannot be obtained in advance. Therefore, the pseudo-label acquisition module (42) obtains a pseudo-truth value (y), which is a virtual truth value. T By obtaining ), the target prototype acquisition module (52) obtains a class-specific target prototype (ρ) according to each class (c) in the target domain. T (c)) enable acquisition.

[0036] The pseudo-label acquisition module (42) is a source prototype (ρ S (c)) and target prototype (ρ T (c) Depending on the bias indicating the difference between the source prototypes (ρ S (c)) This target prototype (ρ T (c)) corrected to correspond to the corrected prototype (ρ S→T (c)) and pixel features (f) extracted from the target feature extractor (31) of the target neural network (30) T (p)) based on the class (c) determined by the similarity between the class (c) and the target neural network (30) being trained, a pseudo-truth value (y T ) is obtained. As the target neural network (30) is updated during the learning process, a corrected prototype (ρ) is obtained. S→T (c)) is updated together, and since the class (c) result determined by the target neural network (30) can also be updated, the pseudo-truth value (y T ) is the truth value of the source domain (y S Unlike ), it is a truth value that can be varied during learning.

[0037] At this time, the pseudo-truth value (y T Since ) is a virtual truth value and may be inaccurate, the pseudo-label acquisition module (42) determines the pseudo-truth value (y) only when the class for each pixel is determined to be accurate with a very high probability. T ) can be labeled. That is, pseudo-truth value (y T ) is the target image(x T A class may not be assigned to all pixels (p) of ).

[0038] The pseudo-label acquisition module (42) obtains a pseudo-truth value (y T A detailed description of the operation to obtain ) will be provided later.

[0039] As shown in FIG. 2, the bidirectional comparison module (43) is a source feature map (f S ) and target feature map(f T Authorized to ) source prototypes (ρ), which are representative class characteristics in the source domain and target domain, respectively S (c)) and target prototype (ρ T (c)) obtain the obtained source prototype (ρ S (c)) and target pixel features for each pixel (f T Forward Contrastive Loss (FCL) (L based on class-specific similarity between (p)) FC While acquiring ), the acquired target prototype (ρ T (c)) and target pixel features for each pixel (f T Backward Contrastive Loss (BCL) (L based on class-specific similarity between (p)) BC ) obtains. That is, through bidirectional contrast, forward contrast loss (L FC ) and backward contrast loss (L BC By calculating ), the learning module (40) is enabled to perform bidirectional contrast learning.

[0040] The bidirectional comparison module (43) may include a source prototype acquisition module (51), a target prototype acquisition module (52), a forward comparison module (53), and a backward comparison module (54).

[0041] The source prototype acquisition module (51) is a source feature map (f) acquired from the source feature extractor (21) of the source neural network (20). S ) and source image(x S Pixel truth values ​​(y) labeled with class (c) for each pixel of ) S (p,c)) is received, and the truth value (y S Using the pixel-specific class (c) determined from (p,c) to the source feature map (f S Source pixel features (f) for each pixel in ) S (p)) is classified by class to create source prototypes (ρ) that represent the source characteristics of each class. S (c)) obtain

[0042] The source prototype acquisition module (51) is a source feature map (f S For ), pixel truth value(y S By applying a mask to each class region separated by (p,c), the source pixel features (f) of the pixel (p) corresponding to each class (c) S (p)) is extracted, and the extracted class-specific source pixel features (f S By average pooling (p)), the source prototype (ρ), which is a representative source feature for each class, S (c)) can be obtained. In other words, the source feature map (f S ) is the source image(x S Separate and extract based on each pixel (p) of ), and the extracted pixel-level source pixel features (f S (p)) corresponds to the pixel truth value (y S Classify into classes based on (p,c), and for each classified class, source pixel features (f S By taking the average value for (p)), the source prototype (ρ S(c)) can be obtained.

[0043] Source prototype (ρ) for each class (c) S (c)) can be obtained according to mathematical formula 1.

[0044]

[0045] The target prototype acquisition module (52) obtains a target feature map (f) from the target feature extractor (31) of the target neural network (30). T ) and target image(x T Pixel pseudo-truth values ​​(y) for each pixel of ) labeled with class (c) T (p,c)) is received, and the pixel pseudo-truth value (y T Using the pixel-specific class (c) determined from (p,c) to the target feature map (f T Target pixel feature (f) for each pixel in ) T (p)) is classified by class to create a target prototype (ρ) representing the target characteristics for each class. T (c)) obtain

[0046] The target prototype acquisition module (52) also has a target feature map (f) similar to the source prototype acquisition module (51). T For ), pixel pseudo-truth value(y T By applying a mask to class regions separated by (p,c), the target pixel feature (f) of the pixel (p) corresponding to each class (c) T (p)) is extracted, and the extracted class-specific target pixel features (f T By pooling the mean of (p)), the target prototype (ρ), which is the representative target feature for each class, T (c)) can be obtained.

[0047]

[0048] The forward comparison module (53) is a source prototype (ρ) per class of the source domain. S (c)) and target pixel features of the target domain (f TForward contrast loss based on class-specific similarity between (p)) (L FC ) calculates. At this time, the forward contrast module (53) calculates the target pixel feature (f T (p))'s class (c) and class-specific source prototypes (ρ S Forward contrast loss (L) such that if the classes (c)) are of the same class, they are accumulated as positive values, whereas if they are of different classes, they are accumulated as negative values. FC ) can be calculated by setting it as in mathematical formula 3.

[0049]

[0050] (Here, τ is the temperature parameter, and s(·, ·) is the cosine similarity function.)

[0051] This is the source prototype (ρ S (c)) and target pixel features (f T (p)) is to train the target feature extractor (31) so that even if the features are obtained from different domains, features for the same class become similar to each other, while features for different classes become different from each other.

[0052] The backward comparison module (54) is a class-specific target prototype (ρ) of the source domain. T (c)) and source pixel features (f S Backward contrast loss based on similarity between (p)) (L BC ) calculates. Similar to the forward contrast module (53), the backward contrast module (54) calculates the source pixel feature (f S (p)) class(c) and class-specific target prototype(ρ) T Backward contrast loss (L) such that if the classes (c)) are of the same class, they are accumulated as positive values, whereas if they are of different classes, they are accumulated as negative values. BC By setting ) as in Equation 4 and calculating, the target prototypes (ρ) of different domains T(c)) and source pixel features (f S (p)) is to train the target feature extractor (31) so that if the features are of the same class, they become similar to each other, whereas if they are features of a different class, they become different from each other.

[0053]

[0054] As mentioned above, even though the bidirectional contrast module (43), the forward contrast module (53), and the backward contrast module (54) are in different domains, each pixel feature (f S (p), f T Among (p)) pixel features for the same class are prototypes (ρ S (c), ρ T (c)) By making them more similar to each other with centering on (c)), while pixel features for other classes become more different, the target feature extractor (31) initially trained based on the source feature extractor (21) trained in the source domain is the target image (x) of the target domain. T It enables accurate extraction of features for each class even in ).

[0055] Here, forward contrast loss (L FC ) and backward contrast loss (L BC ) is a loss based on similarity in different domain directions, and when combined, it can be called bidirectional comparison loss.

[0056] The backpropagation module (44) obtains the forward comparison loss (L) from the bidirectional comparison module (43). FC ) and backward contrast loss (L BC The target neural network is trained by receiving ) and backpropagating it to the target neural network (30). However, the forward contrast loss (L FC ) and backward contrast loss (L BC) is a loss to enable the target feature extractor (31) of the target neural network (30) to extract features more accurately, so the backpropagation module (44) is such that the target neural network (30), including the target feature extractor (31) and the target classifier (32), [examines] the target image (x) of the target domain. T In order to perform accurate semantic segmentation even for ), an additional loss can be calculated and backpropagated together. Here, the additional loss is a pseudo-truth value (y) obtained from the pseudo-label acquisition module (42), similar to the training of the source neural network (20). T It can be calculated as a loss according to a supervised learning method based on ). That is, the additional loss is the semantic segmentation result of the target neural network (30) and the pseudo-truth value (y T Classification loss based on the difference between (L seg ) and cross-entropy loss (L ent It can be calculated as ). Classification loss (L seg ) and cross-entropy loss (L ent The method for calculating ) is publicly known, so it will not be explained in detail here.

[0057] Below, pseudo-truth values ​​(y T Describes the detailed configuration and operation of the pseudo-label acquisition module that obtains ).

[0058] FIG. 3 shows an example of the detailed configuration of the pseudo-label acquisition module of FIG. 1, and FIG. 4 is a diagram for explaining the operation of the pseudo-label acquisition module of FIG. 3. FIG. 5 shows an example of dynamic pseudo-labels and hybrid pseudo-labels updated by the pseudo-label acquisition module of FIG. 3.

[0059] Referring to FIG. 3, the pseudo-label acquisition module (42) may include a static label module (61), a dynamic label module (62), and a pseudo-label module (68).

[0060] The static label module (61) is a target image (x) by the target classifier (32) of the target neural network (30). T For each pixel (p) of ), the matching probability of the assigned class (c) is a specified threshold value ( If it exceeds , for example 0.8), the class (c) assigned to the pixel is the static pixel truth value (y F It is set to (p,c). That is, the matching probability of class (c) is set to 1, corresponding to the truth value. On the other hand, the threshold value ( If it is less than ), the matching probability of the class (c) assigned to the pixel is adjusted to 0 to set it as not matched. Here, the static label module (61) sets the matching probability to a threshold value ( Only classes (c) exceeding ) static pixel truth values ​​(y F The reason for setting it to (p,c) is that the semantic segmentation accuracy is not high because the target neural network (30) is in the learning process. Even though the target neural network (30) is in an initial state of learning based on the weights of the source neural network (20), this means that learning has been performed on the source domain, and the performance in the target domain increases during the learning process. Therefore, the static label module (61) is such that the target classifier (32) has a threshold value ( Only when class (c) is identified with a high probability exceeding ), this is the static pixel truth value (y F Set to (p,c)), and target image (x T The determined static pixel truth value (y) for all pixels (p) of ) F If (p,c)) is set, this is the static truth value (y F Obtained as ).

[0061] Meanwhile, the dynamic label module (62) is a class-specific correction prototype (ρ) that corrects the class-specific features of each class in the source domain to the class-specific features of the target domain. S→T (c)) obtain the obtained class-specific correction prototype (ρ S→T (c) Each target pixel feature (f T(p)) Depending on the similarity between the target images (x T Determining the class for each pixel (p) of ), the dynamic pixel truth value (y) for each pixel (p). D It is set to (p,c). At this time, the dynamic label module (62) is the target pixel feature (f T The similarity with (p)) is the threshold value ( Dynamic pixel truth value (y) for the corresponding pixel only if ) D (p,c)) can be set.

[0062] The dynamic label module (62) may include a source update module (63), a target update module (64), a bias estimation module (65), a prototype correction module (66), and a dynamic label acquisition module (67).

[0063] The source update module (63) and the target update module (65) each have a source prototype (ρ S (c)) and target prototype (ρ T (c)) authorized cumulative source prototype (μ S (c)) and cumulative target prototype (μ T Update (c)). As mentioned above, the source prototype (ρ S (c)) and target prototype (ρ T (c)) is the source image (x) currently input to the semantic segmentation learning device. S ) and target image(x T Since these are representative features for each class obtained from ), the source image (x S ) and target image(x T Not only is it difficult to represent the features of class (c) not included in ), but also various different source images (x S ) and target image(x T It may not accurately represent the class-specific features obtained from ). Accordingly, the source update module (63) and the target update module (65) use multiple source images (x) during the learning process. S ) and target image(x TSource prototypes (ρ), which are representative class-specific features extracted from ) S (c)) and target prototype (ρ T (c)) accumulating the current input source image (x S ) and target image(x T Cumulative source prototype (μ) as a representative class-specific feature not limited to ) S (c)) and cumulative target prototype (μ T (c)) obtain

[0064] Cumulative source prototype (μ S (c)) and cumulative target prototype (μ T (c)) can be obtained according to mathematical formulas 5 and 6, respectively.

[0065]

[0066] (Here, λ represents the weighting ratio.)

[0067]

[0068] The bias estimation module (65) accumulates the bias (ξ) of class-specific feature values ​​according to the domain difference between the target domain and the source domain, i.e., the bias, for each class, and the target prototype (μ T (c)) and cumulative source prototype (μ S (c)) Calculate as the difference between them as in Equation 7.

[0069]

[0070] When the bias (ξ) is calculated according to mathematical formula 7, the source prototype (ρ) for each class of the source domain is obtained using the bias (ξ). S (c) Correct each as a prototype for the target domain to obtain a corrected prototype (ρ S→T (c)) is obtained. That is, the source prototype (ρ S (c)) target prototype (ρ T Corrected as in Equation 8 to correspond to (c)).

[0071]

[0072] The dynamic label acquisition module (67) is a calibration prototype (ρ S→T (c)) and target image (x T Target pixel features (f) extracted from each pixel of ) T Similarity between (p)) (s(f T (p), ρ S→T (c))) Calculate the dynamic pixel truth value (y D (p,c)) can be set according to mathematical formula 9.

[0073]

[0074] According to mathematical formula 9, the dynamic label acquisition module (67) calculates the similarity (s(f) T (p), ρ S→T (c))) is the threshold value( If it exceeds ), the threshold value( The dynamic pixel truth value (y) of the corresponding pixel (p) for class (c) exceeding ) D Set to (p,c). In this case, if the threshold value ( Similarity (s(f) exceeding ) T (p), ρ S→T If there are multiple classes with (c))), the class with the largest similarity (s(f T (p), ρ S→T (c))) class(c', where ) is the dynamic pixel truth value (y D It is set to (p,c)). However, the calculated similarity (s(f T (p), ρ S→T (c))) is the threshold value( If ) or less, the corresponding pixel (p) has a dynamic pixel truth value (y D (p,c)) is not set. The dynamic label acquisition module (67) is the target image (x T The determined dynamic pixel truth value (y) for all pixels (p) of ) D When (p,c)) is set, this is the dynamic truth value (y D Obtained as ).

[0075] At this time, the dynamic label module (62) provides a dynamic truth value (y) for each iteration of learning. D You can acquire it anew by updating ), but you can also update it periodically according to a pre-specified number of repetitions, or the source image (x S ) or target image(x T Even if ) changes, the dynamic truth value (y D It may also be set to update the ). As shown in FIG. 5, the source image (x S If ) is changed, the dynamic truth value (y D Dynamic pixel truth values ​​(y) included in ) D The number of (p,c)) can increase significantly. Although the source prototype (ρ) at every training iteration S (c)) and target prototype (ρ T (c)) is acquired again, but the source image (x S ) and target image(x T If ) does not change, source prototype(ρ S (c)) and target prototype (ρ T (c)) The change is not significant, so the correction prototype (ρ S→T (c)) change is also not significant. Accordingly, the dynamic label module (62) takes into account learning efficiency and dynamic truth values ​​(y D ) periodic or source image(x S ) or target image(x T Dynamic truth value (y) when ) changes D The amount of computation can be reduced by updating ).

[0076] Target image (x T Static truth value (y) for ) F ) and dynamic truth value(y D When ) is obtained, the pseudo-label module (68) obtains a static truth value (y) according to Equation 10. F ) and dynamic truth value(y D Synthesizing the target image (x T Pseudopixel truth value (y) for each pixel (p) of ) TBy setting (p,c)), the pseudo-truth value (y T Acquires ).

[0077]

[0078] According to mathematical formula 10, the pseudo-label module (68) is a dynamic pixel truth value (y D The pixel (p) with (p,c) set is the dynamic pixel truth value (y D (p,c)) pseudopixel truth value (y T Label the corresponding pixel (p) with (p,c) and dynamic pixel truth value (y D Among pixels where (p,c)) is not set, the static pixel truth value (y F The pixel (p) with (p,c) set has a static pixel truth value (y F (p,c)) is the pseudopixel truth value (y) of the corresponding pixel (p). T Labeled with (p,c)), dynamic pixel truth value (y D (p,c)) or static pixel truth value (y F Pixels where (p,c)) is not set have pseudopixel truth values ​​(y T By not labeling (p,c), the target image (x) T Pseudo-truth value (y) for ) T Acquires ).

[0079] That is, the dynamic label module (62) is, as shown in FIG. 4, the source truth value (y S ) correction prototype(ρ S→T By making the features of each class similar so that they can be transitioned to truth values ​​for the target image based on (c)), even though the target image (x T Even though a target truth value for ) is not provided, a pseudo-truth value (y) corresponding to the target truth value T Enables obtaining ).

[0080] Consequently, the semantic segmentation learning device according to the exemplary embodiment has a target image (x) of the target domain. T ) and a source image (x) in a different source domain S ) and source truth value(yS Learning is performed based on ), but the target neural network (30) is trained to extract similar class-specific features regardless of the domain based on bidirectional contrast learning, and also pseudo-truth values ​​(y) corresponding to the target truth values. T Since learning is performed by acquiring ), the performance of the semantic segmentation device can be significantly improved.

[0081] In the illustrated embodiments, each component may have different functions and capabilities other than those described below, and may include additional components other than those not described below. Additionally, in one embodiment, each component may be implemented using one or more physically separated devices, or by one or more processors or a combination of one or more processors and software, and may not be clearly distinguished in specific operation as in the illustrated examples.

[0082] And the semantic partitioning learning device illustrated in FIG. 1 may be implemented in a logic circuit by hardware, firmware, software, or a combination thereof, or may be implemented using a general-purpose or specific-purpose computer. The device may be implemented using a hardwired device, a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc. Additionally, the device may be implemented as a system-on-chip (SoC) including one or more processors and controllers.

[0083] Furthermore, the semantic partitioning learning device may be installed in the form of software, hardware, or a combination thereof on a computing device or server equipped with hardware elements. A computing device or server may refer to various devices that include, in whole or in part, communication devices such as communication modems for communicating with various devices or wired / wireless communication networks, memory for storing data for executing programs, and microprocessors for executing programs to perform calculations and commands.

[0084] FIG. 6 illustrates a semantic partitioning learning method according to one embodiment.

[0085] Referring to FIGS. 1 to 5, the semantic segmentation method of FIG. 6 is explained as follows: First, a source neural network (20) is trained (71). The source neural network (20) receives an input source image (x S Truth value (y) in ) S Since ) is already labeled, it can be easily trained according to the existing supervised learning method. Then, the target neural network (30) is initialized by applying the weights of the source neural network (20) trained to the target neural network (30) (72). That is, the target neural network (30) is made to have a state that is trained identically to the source neural network (20).

[0086] Afterwards, the source image (x) is applied to the source neural network (20). S Input ) and input the target image (x) to the target neural network (30). T By inputting ) and extracting features, each source feature map (f S ) and target feature map(f T ) obtain , and the obtained source feature map (f S ) and target feature map(f T Based on ) source image(x S ) and target image(x T Semantic segmentation is performed by identifying the class for each pixel of ) (73).

[0087] And the source image (x S Truth value (y) labeled in )S ) and target image(x T For ), the pseudo-truth value (y), which is a previously generated virtual truth value T Based on ), source feature map(f S ) and target feature map(f T Source pixel features (f) which are pixel-level features in ) S (p)) and target pixel features (f T (p)) is obtained by distinguishing according to each class (c), and the source pixel features (f) obtained by distinguishing according to each class (c) S (p)) and target pixel features (f T By pooling the mean of (p)), source prototypes (ρ) representing the source and target features for each class (c) are obtained. S (c)) and target prototype (ρ T (c)) obtain (74).

[0088] Here, the target image (x T Since the truth value for ) is unlabeled, the source prototype (ρ S Unlike (c)), the target prototype (ρ T (c)) is the pseudo-truth value (y T It is obtained using ). Pseudo-truth value (y T ) is initially a target image (x) identified by the target neural network (30). T A static truth value (y) based on the class matching probability for each pixel of ) F It can be obtained as ), and thereafter as source prototype(ρ S (c)) and target prototype (ρ T (c)) Dynamic truth value (y) obtained based on D ) and static truth value(y F It can be obtained by synthesizing ).

[0089] Source prototype (ρ S (c)) and target prototype (ρ T (c)) When the source prototype (ρ) for each class (c) is obtained S (c)) and target pixel features (fT Forward contrast loss (L based on similarity between (p)) FC Calculate ) and the target prototype (ρ) for each class (c). T (c)) and source pixel features (f S Backward contrast loss (L based on similarity between (p)) BC Calculate ). Here, forward contrast loss (L FC ) and backward contrast loss (L BC The bidirectional contrast loss is calculated as the sum of ) (75).

[0090] Then, the target neural network (30) is trained by backpropagating the calculated bidirectional contrast loss. At this time, the pseudo-truth value (y T Based on ) the target image (x) of the target neural network (30) T Classification loss (L) for ) seg ) and cross-entropy loss (L ent The target neural network (30) can be trained by calculating together and backpropagating with the bidirectional contrast loss and weighted sum.

[0091] And pseudo-truth value (y T Determines whether to update (77). Pseudo-truth value (y T ) may be updated during every training session, but to reduce computational load and increase training efficiency, it is performed periodically at specified intervals during iterative training or the source image(x S It can be performed when ) is changed, and in some cases, the target image (x T It can also be performed even if ) is changed.

[0092] If pseudo-truth value (y T If it is determined that ) will be updated, the pseudo-truth value (y T Correct by updating ) (78).

[0093] Pseudo-truth value (y T When updating ), first the target neural network (30) updates the target image (x TThe matching probability of the identified class for each pixel of ) is the threshold value ( Only if it exceeds ), the class of the corresponding pixel is the static pixel truth value (y F Set to (p,c)) and the static truth value (y F Acquires ).

[0094] And the source prototype (ρ), which is a representative characteristic of each class in the source domain S (c)) is transferred to the target domain to obtain the target image (x T The dynamic truth value (y) which is the truth value determined for ). D Obtains ). Dynamic truth value(y D To obtain ), first, the source prototype (ρ S (c)) and target prototype (ρ T (c)) Cumulative source prototype (μ) according to weighting ratio (λ) S (c)) and cumulative target prototype (μ T (c)) weighted sum to cumulative source prototype (μ S (c)) and cumulative target prototype (μ T Update (c)). Afterwards, the cumulative target prototype (μ T (c)) and cumulative source prototype (μ S (c)) Calculate the bias (ξ) representing the difference in class-specific features due to domain differences as the difference between them, and the bias (ξ) is used for the source prototype (ρ S (c) Add to source prototype (ρ S (c)) target prototype (ρ T Correction prototype (ρ) corrected to correspond to (c)) S→T (c)) obtain

[0095] Correction prototype (ρ S→T (c)) When this is obtained for each class (c), the correction prototype (ρ) for each class S→T (c)) and target pixel features (f T Similarity between (p)) (s(f T (p), ρ S→TCalculate (c))) to dynamic truth value (y D ) obtains. At this time, the dynamic truth value (y D Dynamic pixel truth value (y) D (p,c)) is the threshold value of the calculated similarity ( It can be set as the class that has the highest similarity compared to other classes while exceeding ) at the same time.

[0096] Static truth value (y F ) and dynamic truth value(y D If ) is obtained, the obtained static truth value (y F ) and dynamic truth value(y D By combining ) the pseudo-truth value(y T ) can be obtained. At this time, the dynamic truth value (y D By giving greater priority to ), the dynamic truth value (y D Dynamic pixel truth value (y) D For the pixel (p) where (p,c)) is set, the dynamic pixel truth value (y D Label (p,c)) and dynamic pixel truth value (y D For pixels (p) where (p,c)) are not set, a static truth value (y F The static pixel truth value (y) of ) F Labels (p,c). However, dynamic pixel truth values ​​(y D (p,c)) or static pixel truth value (y F Pixels (p) where (p,c)) are not both set are not labeled with a class, thereby preventing errors caused by incorrect labels.

[0097] Although FIG. 6 describes each process as being executed sequentially, this is merely an illustrative description, and a person skilled in the art can apply various modifications and variations by changing the order described in FIG. 6, executing one or more processes in parallel, or adding other processes, within the scope of not departing from the essential characteristics of the embodiment of the present invention.

[0098] FIG. 7 is a diagram illustrating a computing environment including a computing device according to one embodiment.

[0099] In the illustrated embodiments, each component may have different functions and capabilities in addition to those described below, and may include additional components in addition to those not described below. The illustrated computing environment (90) may include a computing device (91) to perform the semantic partitioning learning method illustrated in FIG. 6. In one embodiment, the computing device (91) may be one or more components included in the semantic partitioning learning device illustrated in FIG. 1.

[0100] A computing device (91) includes at least one processor (92), a computer-readable storage medium (93), and a communication bus (95). The processor (92) may enable the computing device (91) to operate according to the exemplary embodiment described above. For example, the processor (92) may execute one or more programs (94) stored in the computer-readable storage medium (93). The one or more programs (94) may include one or more computer-executable instructions, and the computer-executable instructions may be configured to enable the computing device (91) to perform operations according to the exemplary embodiment when executed by the processor (92).

[0101] The communication bus (95) interconnects various other components of the computing device (91), including the processor (92) and the computer-readable storage medium (93).

[0102] The computing device (91) may also include one or more input / output interfaces (96) and one or more communication interfaces (97) that provide an interface for one or more input / output devices (98). The input / output interfaces (96) and communication interfaces (97) are connected to a communication bus (95). The input / output devices (98) may be connected to other components of the computing device (91) through the input / output interfaces (96). An exemplary input / output device (98) may include an input device such as a pointing device (such as a mouse or trackpad), a keyboard, a touch input device (such as a touchpad or touchscreen), a voice or sound input device, various types of sensor devices and / or imaging devices, and / or an output device such as a display device, a printer, a speaker and / or a network card. An exemplary input / output device (98) may be included inside the computing device (91) as a component constituting the computing device (91), or it may be connected to the computing device (91) as a separate device distinct from the computing device (91).

[0103] Although the present invention has been described in detail above through representative embodiments, those skilled in the art will understand that various modifications and equivalent alternative embodiments are possible therefrom. Accordingly, the true technical scope of protection of the present invention should be determined by the technical spirit of the appended claims. Explanation of the symbols

[0104] 10: Data acquisition module 11: Source data acquisition module 12: Target data acquisition module 20: Source neural network 21: Source Feature Extractor 22: Source Classifier 30: Target Neural Network 31: Target Feature Extractor 32: Target Classifier 40: Training Module 41: Source Label Acquisition Module 42: Pseudo-label Acquisition Module 43: Bidirectional comparison module 44: Backpropagation module 51: Source Prototype Acquisition Module 52: Target Prototype Acquisition Module 53: Forward Comparison Module 54: Backward Comparison Module 61: Static Label Module 62: Dynamic Label Module 63: Source Update Module 64: Target Update Module 65: Bias Estimation Module 66: Prototype Correction Module 67: Dynamic Label Acquisition Module 68: Pseudo-label Module

Claims

Claim 1 One or more processors; A device comprising a memory for storing one or more programs executed by the above one or more processors, wherein the processor inputs a source image of a source domain and a target image of a target domain into a source neural network and a target neural network, respectively, which perform semantic segmentation, to obtain a source feature map and a target feature map; classifies the source pixel features and target pixel features of the source feature map and the target feature map, respectively, according to class to obtain a source prototype and a target prototype representing representative features for each class; estimates a bias between domains based on the vector difference between the cumulative average value of each class of the source prototype and the cumulative average value of each class of the target prototype; generates a correction prototype by adding the estimated bias to the source prototype; obtains a dynamic truth value based on the similarity between each pixel feature of the correction prototype and the target feature map and a static truth value based on the initial prediction value of the target neural network, respectively; labels pixels for which the dynamic truth value is set with the dynamic truth value, and labels pixels for which the static truth value is set among pixels for which the dynamic truth value is not set with the static truth value, thereby finally A semantic segmentation learning device that generates pseudo-truth values ​​and updates the target neural network by summing the classification loss based on the generated final pseudo-truth values ​​and the bidirectional comparison loss using the source prototype and the target prototype. Claim 2 A semantic segmentation learning device according to claim 1, wherein the processor extracts source pixel features from the source feature map corresponding to each pixel of the source image, obtains source prototypes for each class by pooling the source pixel features separated by class, extracts target pixel features from the target feature map corresponding to each pixel of the target image, and obtains target prototypes for each class by pooling the target pixel features separated by class. Claim 3 delete Claim 4 delete Claim 5 delete Claim 6 delete Claim 7 delete Claim 8 A semantic segmentation learning device according to claim 1, wherein the processor calculates the forward contrast loss of the bidirectional contrast loss by setting it such that if the class of the target pixel feature and the class of the source prototype are of the same class, it is accumulated as a positive value, and if they are of different classes, it is accumulated as a negative value, and calculates the backward contrast loss of the bidirectional contrast loss by setting it such that if the class of the source pixel feature and the class of the target prototype are of the same class, it is accumulated as a positive value, and if they are of different classes, it is accumulated as a negative value. Claim 9 In claim 1, the processor is a source prototype (ρ S (c)) mathematical expression (Here, p is pixel, c is class, f S (p) is a source pixel feature, y S (p,c) is obtained according to the pixel truth value for each pixel of the source image, and the target prototype (ρ) is obtained. T (c)) mathematical expression (here f T (p) is a target pixel feature, y T (p,c) is a semantic segmentation learning device acquired according to the final pseudo-truth value for each pixel of the target image. Claim 10 In paragraph 8, the processor is the forward contrast loss (L FC ) mathematical expression (Here, p is pixel, c is class, y T (p,c) is the pixel pseudo-truth value for each pixel of the target image, f T (p) is a target pixel feature, ρ S (c) is calculated using the source prototype, τ is the temperature parameter, and s(·, ·) is the cosine similarity function, and the backward contrast loss (L BC ) mathematical expression (y S (p,c) is the pixel truth value for each pixel of the source image, f S (p) is a source pixel feature, ρ T (c) is a semantic partitioning learning device that computes the target prototype. Claim 11 A method performed by a computing device having one or more processors and a memory for storing one or more programs executed by said one or more processors, comprising: a step of obtaining a source feature map and a target feature map by inputting a source image of a source domain and a target image of a target domain into a source neural network and a target neural network, respectively, which perform semantic partitioning; a step of obtaining a source prototype and a target prototype representing representative features for each class by classifying the source pixel features and target pixel features of the source feature map and the target feature map, respectively, according to class; a step of estimating a bias between domains based on the vector difference between the cumulative average value of each class of the source prototype and the cumulative average value of each class of the target prototype; a step of generating a correction prototype by adding the estimated bias to the source prototype; and a step of obtaining a dynamic truth value based on the similarity between each pixel feature of the correction prototype and the target feature map and a static truth value based on the initial prediction value of the target neural network, respectively. A semantic segmentation learning method comprising: a step of generating a final pseudo-truth value by labeling pixels with a dynamic truth value set with the dynamic truth value, and pixels with a static truth value set among pixels without a dynamic truth value set with the static truth value; and a step of updating the target neural network by summing the classification loss according to the generated final pseudo-truth value and the bidirectional comparison loss using the source prototype and the target prototype. Claim 12 In claim 11, the step of obtaining the prototype is a semantic segmentation learning method comprising: extracting source pixel features from a source feature map corresponding to each pixel of the source image; pooling the source pixel features separated by class to obtain the source prototype for each class; extracting target pixel features from a target feature map corresponding to each pixel of the target image; and pooling the target pixel features separated by class to obtain the target prototype for each class. Claim 13 delete Claim 14 delete Claim 15 delete Claim 16 delete Claim 17 delete Claim 18 In claim 11, the step of backpropagating the bidirectional contrast loss is calculated by setting the forward contrast loss of the bidirectional contrast loss such that if the class of the target pixel feature and the class of the source prototype are of the same class, it is accumulated as a positive value, and if they are of different classes, it is accumulated as a negative value, and the backward contrast loss of the bidirectional contrast loss is calculated such that if the class of the source pixel feature and the class of the target prototype are of the same class, it is accumulated as a positive value, and if they are of different classes, it is accumulated as a negative value. This is a semantic segmentation learning method. Claim 19 In paragraph 11, the step of obtaining the above prototype is a source prototype (ρ S (c)) mathematical expression (Here, p is pixel, c is class, f S (p) is a source pixel feature, y S (p,c) is obtained according to the pixel truth value for each pixel of the source image, and the target prototype (ρ) is obtained. T (c)) mathematical expression (here f T (p) is a target pixel feature, y T (p,c) is a semantic segmentation learning method obtained according to the final pseudo-truth value for each pixel of the target image. Claim 20 In paragraph 18, the step of backpropagating the bidirectional contrast loss is the forward contrast loss (L FC ) mathematical expression (Here, p is pixel, c is class, y T (p,c) is the pixel pseudo-truth value for each pixel of the target image, f T (p) is a target pixel feature, ρ S (c) is calculated using the source prototype, τ is the temperature parameter, and s(·, ·) is the cosine similarity function, and the backward contrast loss (L BC ) mathematical expression (y S (p,c) is the pixel truth value for each pixel of the source image, f S (p) is a source pixel feature, ρ T (c) is a semantic partitioning learning method that computes the target prototype.

Images