Method, apparatus and storage medium for domain adaptation applied to image segmentation

Abstract

To provide a method of domain adaptation applied to image segmentation, an apparatus, and a storage medium.SOLUTION: In a method of domain adaptation to be applied to image segmentation which executes image segmentation on a target domain image to be processed using a trained encoder 210 and a first decoder 220, the encoder extracts first and second source domain features from a source domain image Is and first and second target domain features from a target domain image It. The second decoder 230 sets a reconstruction loss function Lrec, reconstructs a target domain image based on the first and second target domain features, and generates a converted image based on the first and second target domain features. The first decoder sets first and second division loss functions Lseg and Lseg', performs image segmentation based on the first source domain feature, and executes image segmentation on the converted image based on the features extracted for the converted image.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates generally to unsupervised domain adaptation for image segmentation tasks, and more particularly to methods for performing image segmentation, neural networks, and methods for training neural networks in an unsupervised domain adaptation setting. [Background technology]

[0002] Image segmentation is one of the fundamental vision tasks in image analysis, which is used to divide an image at the pixel level into multiple distinct regions associated with multiple semantic classes (categories). In the predicted image obtained by performing image segmentation on an input image, pixels in a region associated with a specific semantic class are displayed in a color corresponding to the semantic class, allowing objects belonging to different semantic classes to be recognized from the predicted image based on their color.

[0003] On the other hand, unsupervised domain adaptation refers to transferring a model trained using labeled source domain data to a target domain of unlabeled data while maintaining the model's performance in the target domain as much as possible. Due to dataset deviation (domain deviation) between the source domain and the target domain and a lack of labeled data in the target domain, a model trained using labeled source domain data may perform poorly in the target domain. The training process of unsupervised domain adaptation utilizes both labeled data from the source domain and unlabeled data from the target domain, effectively mitigating the adverse effects of domain differences and improving the robustness of the model.

[0004] Currently, methods for learning domain-invariant features represented by adversarial training and multi-task learning have been proposed for unsupervised domain adaptation. In multi-task learning, domain-invariant features are learned using multiple tasks (e.g., classification of source domain data and reconstruction of target domain data). Here, the classification task and reconstruction task share a feature extractor, and training is alternated between classification of source domain data and reconstruction of target domain data so that the features extracted by the feature extractor simultaneously meet the performance requirements of the source domain classification task and the target domain reconstruction task. The shared features extracted in this way are called domain-invariant features.

[0005] Moreover, most of the current research on unsupervised domain adaptation is focused on image classification tasks, and there is little research on unsupervised domain adaptation for image segmentation tasks. Summary of the Invention [Problem to be solved by the invention]

[0006] In view of the above technical problems, the present invention provides a new unsupervised domain adaptation scheme based on feature disentanglement and knowledge distillation, which can be applied to image segmentation tasks. The scheme learns domain-invariant features by disentanglement of semantic and non-semantic features. The present invention also optimizes a loss function for the target domain through knowledge distillation. [Means for solving the problem]

[0007] In one aspect of the present invention, there is provided a computer-implemented method for performing domain adaptation applied to image segmentation using a neural network, the neural network including an encoder, a first decoder, and a second decoder, the method including the steps of the encoder extracting source domain features for a source domain image and extracting target domain features for a target domain image, and mapping the source domain features to first source domain features (v) related to content in the source domain image. s ) and a second source domain feature (d s ) and classifying the target domain features into first target domain features (v t ) and a second target domain feature (d t ), and the first decoder classifies the first source domain features (v s ) to perform image segmentation on the source domain image based on a first segmentation loss function (L seg ), and the second decoder sets the first target domain feature (v t ) and the second target domain feature (d t ) and reconstruct the target domain image based on the reconstruction loss function (L rec ), and the second decoder sets the first source domain feature (v s ) and the second target domain feature (d t ), the encoder extracts features from the transformed image, and the first decoder performs image segmentation on the transformed image based on content-related parts of the extracted features, and calculates a second segmentation loss function (L seg’), training the encoder, the first decoder, and the second decoder based on the first segmentation loss function, the second segmentation loss function, and the reconstruction loss function; and performing image segmentation on a target domain image to be processed using the trained encoder and first decoder.

[0008] In another aspect of the present invention, an apparatus for performing domain adaptation applied to image segmentation using a neural network, the neural network including an encoder, a first decoder, and a second decoder, the apparatus including a memory having a computer program stored therein, and one or more processors, the processors executing the computer program to cause the encoder to extract source domain features for a source domain image and to extract target domain features for a target domain image, and to convert the source domain features into first source domain features (v) related to content in the source domain image. s ) and a second source domain feature (d s ) and classifying the target domain features into first target domain features (v t ) and a second target domain feature (d t ), and the first decoder classifies the first source domain features (v s ) to perform image segmentation on the source domain image based on a first segmentation loss function (L seg ), and the second decoder sets the first target domain feature (v t ) and the second target domain feature (d t ) and reconstruct the target domain image based on the reconstruction loss function (L rec ), and the second decoder sets the first source domain feature (v s ) and the second target domain feature (dt ), the encoder extracts features from the transformed image, and the first decoder performs image segmentation on the transformed image based on content-related parts of the extracted features, and calculates a second segmentation loss function (L seg’ ), training the encoder, the first decoder, and the second decoder based on the first segmentation loss function, the second segmentation loss function, and the reconstruction loss function; and performing image segmentation on a target domain image to be processed using the trained encoder and first decoder.

[0009] In another aspect of the present invention, there is provided a storage medium having stored thereon a computer program which, when executed by a computer, causes the computer to perform the method for performing domain adaptation described above. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of a conventional multi-task learning model. [Figure 2] FIG. 1 is a schematic diagram illustrating a feature disentanglement-based domain adaptation model according to the present invention. [Figure 3] FIG. 10 is a schematic diagram illustrating the visualization effect of feature disentanglement. [Figure 4] FIG. 1 is a diagram illustrating a schematic of a U-net network structure. [Figure 5] FIG. 1 is a diagram illustrating a model of knowledge distillation according to the present invention. [Figure 6] 1 is a flowchart illustrating a method for performing image segmentation. [Figure 7] 1 is a flowchart showing how to transfer the knowledge of a trained D-net to a U-net. [Figure 8] FIG. 1 is a block diagram illustrating an exemplary configuration of computer hardware capable of implementing the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] FIG. 1 is a schematic diagram of a conventional multi-task learning model, i.e., a deep reconstruction-classification network (DRCN). As shown in FIG. 1, an image is input to a feature extractor 100, which performs the following operations: convolution (Conv), max pooling, flattening multidimensional data to one dimension, fully connected (Dense), and random deactivation (Dropout). As a result of the operations, the feature extractor 100 outputs features F. As shown in the upper branch of FIG. 1, the extracted features F are used for supervised classification training of source domain data. The extracted features F are also used for unsupervised reconstruction training of target domain data. As shown in the lower branch of FIG. 1, the reconstruction process is the inverse process of the process performed by the feature extractor 100 and includes the operations of unflattening one-dimensional data to multidimensional data, fully connected (Dense), upsampling, and convolution (Conv). The classification task in the source domain and the reconstruction task in the target domain share the feature extractor 100. By alternating between classification training and reconstruction training, feature extractor 100 can extract domain-invariant features (shared features) that meet the requirements of tasks in the two domains.

[0012] Based on the model shown in Figure 1, the present invention designs a new network structure for feature disentanglement. Generally, feature disentanglement means separating multiple types of features, for example, separating features that are useful for a specific task from features that are useless, so that the useful features can be used to perform the task.

[0013] 2 is a schematic diagram of a feature disentanglement-based domain adaptation model according to the present invention, which can be applied to image segmentation tasks. In particular, the following describes the present invention by taking the segmentation task of a retinal vascular image as an example, but the present invention is not limited thereto and may be applied to any image segmentation task.

[0014] As shown in Figure 2, the source domain image I s and target domain image I t is input to the encoder 210. Preferably, the source domain image I s and target domain image I t Preprocessing is performed on the preprocessed image I s-p and I t-p are input to the encoder 210. Preprocessing may include, for example, converting the image to a grayscale image, normalization, histogram equalization, and gamma correction. The preprocessing may also include data augmentation processes such as rotation, inversion, and color dithering to improve the generalization ability of the model. When performing a transfer task from a high-resolution dataset to a low-resolution dataset, the preprocessing may also include randomly adding Gaussian noise to the high-resolution image. This process can improve the effectiveness of domain adaptation. The preprocessing may also include resizing the image. For example, the image may be adjusted to a uniform size (e.g., 512 × 512) using bicubic interpolation.

[0015] The encoder 210 extracts features from the input image. FIG. 2 shows multiple (e.g., four) features output by multiple hidden layers of the encoder 210, each of which includes multiple (e.g., N) channels. In the present invention, the features are divided into two equal parts by channel, with the first part v including the first N / 2 channels and the second part d including the last N / 2 channels. Assume that the first part v represents semantic features related to content (referred to as "content features"), and the second part represents non-semantic features unrelated to content (referred to as "style features"). Taking a retinal vascular image as an example, the content features may correspond to features of the vascular structure, and the style features may include features such as the color, brightness, contrast, and background of the image.

[0016] Semantic features (content features) are domain-invariant features, while non-semantic features (style features) are features that cause domain deviation. Therefore, the purpose of feature disentanglement in this invention is to separate semantic features from non-semantic features, so that semantic features can be used to perform image segmentation tasks and achieve good performance in the target domain.

[0017] In FIG. 2, an encoder 210 receives a source domain image I s Regarding semantic features v s and non-semantic features d s Generate the target domain image I t Regarding semantic features v t and non-semantic features d t may be generated.

[0018] The segmentation decoder 220 extracts the semantic features v of the source domain image from the encoder 210. s receives the semantic features v s The split probability map p is generated based on the supervised split training, which is performed on a split network consisting of an encoder 210 and a split decoder 220.

[0019] The reconstruction decoder 230 extracts the semantic features v of the target domain image from the encoder 210. t and non-semantic features d t and receiving a reconstructed image I of the target domain image based on the received image. rec Generate the reconstructed image I rec includes semantic features (e.g., vascular structure) and non-semantic features (e.g., style factors) of the target domain image. Unsupervised reconstruction training is performed on a reconstruction network consisting of an encoder 210 and a reconstruction decoder 230.

[0020] The reconstruction decoder 230 also receives the semantic features v of the source domain image from the encoder 210. s and non-semantic features of the target domain image d t and further receives the semantic features v of the source domain image. s and non-semantic features of the target domain image d t New image based on I new Create a new image I new Inew includes semantic features of the source domain image (e.g., vascular structure) and non-semantic features of the target domain image (e.g., style factors). In the following, the new image Inew is also referred to as the translated image.

[0021] Next, the transformed image I new is input to the encoder 210. As a result, the encoder 210 and the segmentation decoder 220 perform supervised segmentation training. Specifically, the features extracted by the encoder 210 for the transformed image are classified into two parts v and d, and the first part v ("content feature") is input to the segmentation decoder 220 to generate a segmentation prediction result. In particular, for the transformed image I new Since contains the semantic features of the source domain image, in supervised segmentation training, the true label of the source domain image is assigned to the transformed image I new Use it as the true label of

[0022] The following describes in detail the model shown in FIG.

[0023] The segmentation network consisting of the encoder 210 and the segmentation decoder 220 is s : defined as x → y, where f s represents a function that maps the input image to a segmentation probability map (p). In the example of a retinal vascular image, the segmentation probability map represents the probability that each pixel in the image belongs to a blood vessel. More specifically, the encoder 210 in the segmentation network is defined as Enc:x→F, which represents the mapping of the input image to the feature space. Also, the segmentation decoder 220 in the segmentation network is defined as S dec :F v →y, where F v represents the semantic features of each hidden layer and feature layer of the encoder 210.

[0024] On the other hand, the reconstruction network consisting of the encoder 210 and the reconstruction decoder 230 is r :x→x, where f r represents a function that maps an input image to an input image. More specifically, the reconstruction decoder 230 uses R dec :{F v ,F d}→x, where F v represents the semantic features of each hidden layer and feature layer of the encoder 210, and F d represents the non-semantic features of each hidden and feature layer of the encoder 210. By changing the input of the reconstruction decoder 230, we can change the content or style in the reconstructed image. In particular, the transformed image I new If you generate (outside 1) It may be defined as TIFF0007823480000001.tif19170.

[0025] The input source domain image is x s and the input target domain image is x t If the function f sand f r may be expressed as the following equations (1) and (2), respectively.

number

number

[0026] Here, the symbol "◯" represents the connection between the decoder and the encoder.

[0027] The objective function used in training is the segmentation loss function L on the source domain images. seg , the reconstruction loss function L for the target domain image rec , and the segmentation loss function L for the transformed image seg’ As an example, the split loss function L seg and L seg’ may be a cross-entropy loss function, and the reconstruction loss function L rec The following equations (3), (4), and (5) are the loss functions L seg , L seg’ and L rec This shows:

number

number

number

[0028] where n s represents the number of source domain images, and n t represents the number of target domain images, and n s’ represents the number of transformed images, and n s ×n t K represents the number of classes (e.g., in segmenting a vascular image, there are two classes: "vascular" and "non-vascular"). H and W represent the width and height of the image, respectively. xs i represents the source domain image sample, and x s’ i represents the transformed image sample, and x t j represents the target domain image sample. i k,j is the source domain image sample x s i represents the binary label of the kth class of pixel j in p s (x s i ) k,j is the source domain image sample x s i represents the predicted probability that pixel j in belongs to the kth class. s (x s’ i ) k,j is the transformed image sample x s’ i represents the predicted probability that pixel j in belongs to the kth class.

[0029] In the above loss function, the segmentation loss function L for the transformed image is seg’ is effective in disentangling semantic features (e.g., vascular structure) and non-semantic features (e.g., image style). Figure 3 shows a schematic diagram of the visualization effect of feature disentanglement. In Figure 3, v s represents the vascular structure features of the source domain image, and v t represents the vascular structure features of the target domain image, and d s represents the style features of the source domain image, and d t represents the style features of the target domain image. Different combinations of these features can generate different images. s ,d t ) and (v t ,d s ) and the loss function L seg +L recIf only the loss function L is applied, the vascular structures in the source domain image and the target domain image are mixed together and cannot be distinguished. seg’ If you are also applying s ,d t ) contains only the vascular structure of the source domain image, and the image (v t ,d s ) contains only the vascular structure of the target domain image. As mentioned above, in the present invention, the feature v represents the content feature (vascular structure feature), and the feature d represents the style feature (features such as image color and brightness). Therefore, the ideal image (v) that can reflect the effect of feature disentanglement is s ,d t ) is an image that contains the vascular structure of the source domain image and the style of the target domain image. Similarly, the ideal image (v t ,d s ) is an image that contains the vascular structure of the target domain image and the style of the source domain image. However, since this image is a grayscale image, it is difficult to express style features such as color. Therefore, the image (v s ,d t ) and (v t ,d s ) the disentanglement of style features is not obvious. Nevertheless, the effect of our feature disentanglement is evident when the image (v s ,d t ) and (v t ,d s ) each contain only vascular features of one domain. For example, it can be considered that at least the vascular features are well disentangled.

[0030] The network model shown in Figure 2 is trained using the above loss function. After training is complete, the actual segmentation task may be performed using the trained encoder 210 and the segmentation decoder 220. Specifically, a target domain image to be processed (unlabeled) is input to the encoder 210, and a segmented predicted image is generated by the segmentation decoder 220.

[0031] This invention further proposes transferring the trained network described above to other networks through knowledge distillation. Specifically, it proposes transferring the trained network (referred to as a D-net) to the basic U-net network structure. This has the following advantages: (1) By utilizing the reliable knowledge provided by the D-net, the risk loss function for the target domain can be optimized, thereby further improving performance. Specifically, the target domain risk loss function includes a supervised segmentation loss function, but the target domain data is unlabeled, so the target domain risk loss function cannot be directly optimized. In the process of transferring the D-net knowledge to the U-net, the more reliable segmentation prediction results generated by the trained D-net for the target domain data can be used as more reliable labels for the target domain data, allowing the target domain risk loss function to be directly optimized. (2) Compared to a D-net network that includes a branching part for disentanglement and reconstruction, the structure of the U-net network is simpler and easier to deploy.

[0032] The U-net network is described in detail in the paper "U-net: Convolutional networks for biomedical image segmentation" by Olaf Ronneberger et al., presented at the 2015 International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), the contents of which are incorporated herein by reference.

[0033] Figure 4 is a schematic diagram of the U-net network structure. The network structure includes an encoder (left half) and a decoder (right half) with a basically symmetrical structure. The encoder includes four convolutional modules connected in series, each consisting of two 3x3 convolutional layers, with adjacent convolutional modules connected by a 2x2 max-pooling layer. The decoder includes four convolutional modules with the same structure as the encoder, but adjacent convolutional modules are connected by a 2x2 upsampling layer. The decoder has a symmetrical structure to the encoder. Furthermore, before performing convolution, each convolutional module in the decoder concatenates hidden features extracted by the encoder's hidden layer with those extracted by the corresponding hidden layer in the decoder, i.e., performs a skip connection as shown in Figure 4. The output of the decoder's last convolutional module is input to a 1x1 convolutional layer to reduce the feature dimensionality to an appropriate dimension, typically the number of classes. In the retinal blood vessel image segmentation example of the present invention, since it concerns two types of classification, the output dimension of the above 1x1 convolutional layer is 1, and a sigmoid activation layer may be connected after the 1x1 convolutional layer, which outputs a blood vessel segmentation probability map.

[0034] In particular, the encoder 210, the partition decoder 220, and the reconstruction decoder 230 described with reference to FIG. 2 above may employ the U-net structure shown in FIG. 4, but with appropriate modifications. Specifically, as described above, the features output by the encoder 210 are divided into two equal parts, v and d. Therefore, when the encoder 210 is implemented using the U-net structure shown in FIG. 4, the number of feature channels output by each hidden layer must be doubled. Furthermore, when the reconstruction decoder 230 is implemented using the U-net structure shown in FIG. 4, the number of feature channels output by the encoder 210 is doubled. Therefore, the reconstruction decoder 230, which receives both the semantic feature v and the non-semantic feature d from the encoder 210, must also increase the number of feature channels received via skip connections accordingly. However, when the partition decoder 220 is implemented using the U-net structure shown in FIG. 4, the partition decoder 220 receives only the semantic feature v from the encoder 210, so the number of feature channels received via skip connections does not need to be changed.

[0035] Figure 5 shows a schematic diagram of transferring the knowledge of a trained D-net network to a U-net network through knowledge distillation. As shown in Figure 5, a target domain image is input to a trained D-net, which generates a segmentation probability map p t ζ as the soft label of the target domain image. Also, transformations such as horizontal and vertical flips are performed on the target domain image, and the transformed image is also input to the D-net, which then generates a segmentation probability map p t_hflip ζ and p t_vflip ζ Next, the generated split probability map p t ζ , p t_hflip ζ and p t_vflip ζAs an example of the integration, the average value of the individual segmentation probability maps may be calculated to obtain an average probability map. Next, a binary segmentation process is performed on the integrated probability map based on a predetermined threshold. As an example, the threshold may be set to 0.5, and pixels in the integrated probability map whose pixel values ​​are greater than the threshold may be labeled as 1, and pixels whose pixel values ​​are less than the threshold may be labeled as 0. The obtained binary segmentation result may be used as a pseudo-label (hard label) for the input target domain image.

[0036] On the other hand, the target domain image is input to U-net, which generates a predicted probability map p t Also, source domain images with known labels are input to the U-net, which generates a predicted probability map p s Preferably, the target domain image and the source domain image are input to a D-net or U-net after preprocessing (as described above), as shown in FIG.

[0037] Next, based on the pseudo-labels, the features of each pixel in the source and target domain images are aligned at the class level. Specifically, for a specific class, the pixel segmentation features extracted for the source and target domain images by the U-net network are aligned to each other. The pixel segmentation features may refer to the features input to the 1x1 convolutional layer, i.e., the features output by the last convolutional module of the decoder shown in FIG. 4. For example, each pixel in an image may have a 64-dimensional pixel segmentation feature.

[0038] In this way, when training the U-net, supervised segmentation training is performed using labeled source domain images and pseudo-labeled target domain images (generated based on the segmentation prediction results of the D-net), and the segmentation probability map p predicted by the U-net for the target domain image is t is the segmentation probability map p tζ (Soft Label) as close as possible.

[0039] Specifically, the loss function expressed by the following formula (6) is used in training.

number

[0040] where L ce s (p s ,y s ) represents the segmentation loss function for U-net to perform segmentation prediction on the source domain image, which can be a standard cross-entropy loss function. s represents the true label of the source domain image, and p s represents the segmentation prediction probability of the source domain image.

[0041] L ce t (p t ,ξ(p t’ ζ )) represents the segmentation loss function for U-net to perform segmentation prediction on the target domain image, which can be a standard cross-entropy loss function. t’ ζ ) represents the pseudo-label generated based on the prediction results of D-net, where p t’ ζ is the probability map of the integration, e.g. (outside 2) TIFF0007823480000008.tif21170. Also, ξ may be expressed by the following mathematical formula (7), where σ is a threshold value, which may be set to, for example, 0.5.

number

[0042] KL t (p t ζ ||p t) is the KL divergence, and the segmentation probability map p predicted by the U-net for the target domain image t The segmentation probability map p predicted by the D-net for the target domain image is t ζ is used to get as close as possible to (Outside 3) TIFF0007823480000010.tif23170 represents a conditional feature alignment loss function, which is used to align, for a given class, the source and target domain features corresponding to that class, where k represents the number of semantic classes, and F s i represents the average feature value of all pixels belonging to the i-th class in the source domain image, and F t i represents the average value of the features of all pixels belonging to the i-th class in the target domain image. s i and F t i The initial value of may be set to 0 and updated as training progresses based on the following equation (8):

number

[0043] where F ’i represents the mean value of the feature of pixels of the i-th class in the source or target domain image calculated for the current mini-batch, and λ is a parameter that controls the update rate, which may be set to 0.7, for example.

[0044] FIG. 6 is a flow chart illustrating a method for performing image segmentation according to the present invention.

[0045] As shown in FIG. 6, in step S610, the encoder 210 extracts source domain features for the source domain image and extracts target domain features for the target domain image.

[0046] In step S620, the source domain features are converted into first source domain features (semantic features) v related to the content in the source domain image. s and a second source domain feature (non-semantic feature) d related to the style of the source domain image. s Similarly, in step S630, the target domain features are classified into first target domain features (semantic features) v t and a second target domain feature (non-semantic feature) d related to the style of the target domain image. t As mentioned above, classification may be performed based on the number of channels contained in the features.

[0047] In step S640, the partition decoder 220 extracts the first source domain feature v s and performing image segmentation on the source domain image based on a first segmentation loss function L seg Set.

[0048] In step S650, the reconstruction decoder 230 calculates the first target domain feature v t and a second target domain feature d t and reconstruct the target domain image based on the reconstruction loss function L rec Set.

[0049] In step S660, the reconstruction decoder 230 calculates the first source domain feature v s and a second target domain feature d t Generate a transformed image based on

[0050] The transformed image is input to the encoder 210 for feature extraction. Similarly, the extracted features are split into two parts with equal numbers of channels: semantic features and non-semantic features. Next, as shown in step S670, the segmentation decoder 220 performs image segmentation on the transformed image based on the semantic features of the transformed image, and calculates a second segmentation loss function L for the operation. seg’ Set.

[0051] In step S680, the first split loss function L seg , the second split loss function L seg’ and the reconstruction loss function L rec Based on this, we train the encoder 210, the partition decoder 220 and the reconstruction decoder 230.

[0052] Once training is complete, the trained encoder 210 and segmentation decoder 220 (segmentation network) are used to perform image segmentation on the target domain image to be processed, as shown in step S690.

[0053] Figure 7 is a flowchart showing how to transfer the knowledge of a trained D-net to a U-net.

[0054] As shown in FIG. 7, in step S710, the trained D-net performs image segmentation on the target domain image to generate a segmentation probability map p t ζ The D-net performs image segmentation on one or more transformed images (e.g., horizontally and vertically flipped images) of the target domain image to generate multiple segmentation probability maps, e.g., p t_hflip ζ and p t_vflip ζ Generate.

[0055] In step S720, a pseudo-label of the target domain image is generated based on the generated plurality of segmentation probability maps. Specifically, the segmentation probability map p t ζ , p t_hflipζ and p t_vflip ζ The mean value of may be calculated and a binarization process may be performed on the mean probability map based on a threshold (e.g., 0.5) to obtain pseudo labels.

[0056] In step S730, the U-net network performs image segmentation on the target domain image and the source domain image, respectively.

[0057] In step S740, the pseudo-labels of the target domain images and the known labels of the source domain images are used to perform supervised training on the U-net network. Specifically, during training, a segmentation loss function associated with the U-net network performing image segmentation on the target domain images, a segmentation loss function associated with the U-net network performing image segmentation on the source domain images, and a segmentation probability map p predicted by the U-net network for the target domain images are calculated. t The segmentation probability map p generated by the D-net network for the target domain image is t ζ KL divergence to approximate , and a conditional feature alignment loss function may be used.

[0058] Once training is complete, the trained U-net network may be used in place of the D-net to perform the actual image segmentation task, as shown in step S750.

[0059] The above describes in detail our unsupervised domain adaptation scheme for image segmentation tasks. We use multi-task training to learn a feature space shared by the source and target domains, feature disentanglement to learn better domain-invariant features, and knowledge distillation to further optimize the loss function for the target domain and simplify the network structure.

[0060] The inventors of the present invention conducted experiments based on retinal vascular images to evaluate the performance of the present invention. The experiments were performed on public datasets, including DRIVE (abbreviated as "D" in Table 1), CHASE_DB1 (abbreviated as "C"), STARE (abbreviated as "S"), HRF (abbreviated as "H"), and IOSTAR (abbreviated as "I"). In the experiments, one of the above datasets is used as labeled source domain data, and one of the other datasets is used as unlabeled target domain data to perform a domain transfer task.

[0061] Table 1 below shows the performance of the feature disentanglement network (D-net) of the present invention, the knowledge distillation scheme (KD) based on the D-net, and several existing schemes, using the F1 score (DICE) as the evaluation metric. The existing schemes include source-only (a scheme that uses only the source domain dataset, not the target domain dataset, for training), domain adversarial neural networks (DANN), and the deep reconstruction classification network (DRCN) shown in Figure 1. The target-only in Table 1 represents the score obtained when training using a labeled target domain dataset and can be used as a reference because it represents the highest score obtainable for each transfer task. As can be seen from Table 1, the performance of the D-net of the present invention is superior to that of other existing schemes, and the knowledge distillation scheme (KD) based on the D-net even outperforms the D-net. [Table 1]

[0062] The above describes embodiments of the present invention with reference to specific examples. The methods according to the above examples may be realized by software, hardware, or a combination of software and hardware. The programs included in the software may be pre-stored in a storage medium installed inside or outside the device. As an example, during execution, these programs are written into a random access memory (RAM) and executed by a processor (e.g., a CPU) to realize each process described herein.

[0063] 8 is a block diagram showing an exemplary configuration of computer hardware capable of implementing the present invention, on which the image segmentation execution device and neural network according to the present invention may be implemented.

[0064] As shown in FIG. 8, in a computer 800, a central processing unit (CPU) 801, a read-only memory (ROM) 802, and a random access memory (RAM) 803 are interconnected by a bus 804.

[0065] The input / output interface 805 is further connected to the bus 804. Connected to the input / output interface 805 are an input unit 806 including a keyboard, mouse, microphone, etc., an output unit 807 including a display, speakers, etc., a storage unit 808 including a hard disk, nonvolatile memory, etc., a communication unit 809 including a network interface card (such as a local area network (LAN) card or modem), and a driver 810 that drives a removable medium 811. The removable medium 811 is, for example, a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0066] In a computer having the above configuration, the CPU 801 loads a program stored in the memory unit 808 into the RAM 803 via the input / output interface 805 and the bus 804, and executes the program to perform the above method.

[0067] The program executed by the computer (CPU 801) may be recorded on a removable medium 811, which is a package medium. The package medium may be formed of, for example, a magnetic disk (including a floppy disk), an optical disk (including a compact disk read-only memory (CD-ROM) and a digital versatile disk (DVD)), a magneto-optical disk, or a semiconductor memory. The program executed by the computer (CPU 801) may also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0068] When the removable medium 811 is installed in the driver 810, the program can be installed in the storage unit 808 via the input / output interface 805. The program can also be received by the communication unit 809 via a wired or wireless transmission medium and installed in the storage unit 808. Alternatively, the program may be pre-installed in the ROM 802 or the storage unit 808.

[0069] The program executed by the computer may be a program that performs operations in the order described in this specification, or may be a program that performs operations in parallel, or may perform operations as needed (e.g., when called).

[0070] The devices or units described in this specification are logical and are not limited to physical devices or entities. For example, the functions of each unit described in this specification may be realized by multiple physical entities, or the functions of multiple units described in this specification may be realized by a single physical entity. Furthermore, features, components, elements, steps, etc. described in one embodiment are not limited to that embodiment and may be applied to other embodiments, for example, and may be used in place of or combined with specific features, components, elements, steps, etc. of other embodiments.

[0071] The scope of the present invention is not limited to the specific examples described herein. As will be understood by those skilled in the art, various modifications or variations may be made to the examples herein depending on design requirements and other factors without departing from the principles and spirit of the present invention. The scope of the present invention is limited by the appended claims and their equivalents.

[0072] Furthermore, the following supplementary notes are also disclosed regarding the embodiments including the above-described examples, but the present invention is not limited to these supplementary notes. (Appendix 1) 1. A computer-implemented method for performing domain adaptation applied to image segmentation using a neural network, the neural network including an encoder, a first decoder, and a second decoder, the method comprising: the encoder extracting source domain features for a source domain image and target domain features for a target domain image; The source domain features are then converted into first source domain features (v s ) and a second source domain feature (d s ) and classifying the target domain features into first target domain features (v t ) and a second target domain feature (d t ) and The first decoder generates the first source domain feature (v s ) to perform image segmentation on the source domain image based on a first segmentation loss function (L seg ), The second decoder generates the first target domain feature (v t ) and the second target domain feature (d t ) and reconstruct the target domain image based on the reconstruction loss function (L rec ), The second decoder generates the first source domain feature (v s ) and the second target domain feature (d t ) generating a transformed image based on the the encoder extracting features from the transformed image; The first decoder performs image segmentation on the transformed image based on the content-related portion of the extracted features, and calculates a second segmentation loss function (L seg’ ), training the encoder, the first decoder, and the second decoder based on the first partition loss function, the second partition loss function, and the reconstruction loss function; performing image segmentation on a target domain image to be processed using the trained encoder and the first decoder. (Appendix 2) The first source domain feature (v s ) and the second source domain feature (d s classifying the source domain features based on channels such that each of the source domain features contains the same number of channels; The first target domain feature (v t ) and the second target domain feature (d t 2. The method of claim 1, wherein the target domain features are classified based on channels such that both the first and second vectors contain the same number of channels. (Appendix 3) The first split loss function (L seg ) and the second split loss function (L seg’ ) is the cross-entropy loss function, The reconstruction loss function (L rec ) is the absolute value loss function. (Appendix 4) 2. The method of claim 1, further comprising transferring knowledge of the trained neural network to a U-net network by knowledge distillation, so as to utilize a U-net network to perform image segmentation on the target domain image to be processed. (Appendix 5) performing image segmentation using the trained neural network on a target domain image and one or more transformed images of the target domain image to generate a plurality of segmentation probability maps; generating pseudo-labels for the target domain image based on the plurality of segmentation probability maps; the U-net network performing image segmentation on the target domain image and the source domain image, respectively; training the U-net network using pseudo-labels of the target domain images and known labels of the source domain images; 5. The method of claim 4, further comprising: performing image segmentation on the target domain image to be processed using a trained U-net network. (Appendix 6) The step of generating pseudo labels includes: calculating an average probability map based on the plurality of split probability maps; and performing a binary splitting process on the average probability map based on a threshold to obtain the pseudo label. (Appendix 7) When training the U-net network, A segmentation loss function for the U-net network performing image segmentation on the target domain image. A segmentation loss function for the U-net network performing image segmentation on the source domain image. The segmentation probability map (p t ) is the segmentation probability map (p t ζ ) and KL divergence to get closer to 6. The method of claim 5, using a conditional feature alignment loss function to bring pixel segmentation features extracted by the U-net network for the target domain image and pixel segmentation features extracted for the source domain image closer to each other for a given class. (Appendix 8) 1. An apparatus for performing domain adaptation applied to image segmentation utilizing a neural network, comprising: the neural network includes an encoder, a first decoder, and a second decoder; The device comprises: a memory in which a computer program is stored; one or more processors; The processor executes the computer program, the encoder extracting source domain features for a source domain image and target domain features for a target domain image; The source domain features are then converted into first source domain features (v s ) and a second source domain feature (d s ) and classifying the target domain features into first target domain features (v t ) and a second target domain feature (d t ) and The first decoder generates the first source domain feature (v s ) to perform image segmentation on the source domain image based on a first segmentation loss function (L seg ), The second decoder generates the first target domain feature (v t ) and the second target domain feature (d t ) and reconstruct the target domain image based on the reconstruction loss function (L rec ), The second decoder generates the first source domain feature (v s ) and the second target domain feature (d t ) generating a transformed image based on the the encoder extracting features from the transformed image; The first decoder performs image segmentation on the transformed image based on the content-related portion of the extracted features, and calculates a second segmentation loss function (L seg’ ), training the encoder, the first decoder, and the second decoder based on the first partition loss function, the second partition loss function, and the reconstruction loss function; performing image segmentation on a target domain image to be processed using the trained encoder and the first decoder. (Appendix 9) The processor: performing image segmentation using the trained neural network on a target domain image and one or more transformed images of the target domain image to generate a plurality of segmentation probability maps; generating pseudo-labels for the target domain image based on the plurality of segmentation probability maps; a U-net network performing image segmentation on the target domain image and the source domain image, respectively; training the U-net network using pseudo-labels of the target domain images and known labels of the source domain images; 9. The apparatus of claim 8, further comprising: performing image segmentation on the target domain image to be processed using a trained U-net network. (Appendix 10) 10. A storage medium having stored thereon a computer program, the computer program, when executed by a computer, causing the computer to perform a method for performing domain adaptation according to any one of Supplementary Notes 1 to 7. (Appendix 11) performing pre-processing on the source domain image and the target domain image; and inputting the pre-processed source domain image and the target domain image to the encoder; 2. The method of claim 1, wherein the preprocessing includes at least one of converting to a grayscale image, normalizing, histogram equalizing, gamma correction, rotating, flipping, color dithering, resizing, and randomly adding Gaussian noise to the source domain image if the source domain image is a high-resolution image and the target domain image is a low-resolution image.

Claims

1. 1. A computer-implemented method for performing domain adaptation applied to image segmentation using a neural network, the neural network including an encoder, a first decoder, and a second decoder, the method comprising: the encoder extracting source domain features for a source domain image and target domain features for a target domain image; The source domain features are then converted into first source domain features (v) related to content in the source domain image. ｓ ) and a second source domain feature (d ｓ ) and classifying the target domain features into first target domain features (v ｔ ) and a second target domain feature (d ｔ ) and The first decoder detects the first source domain features (v ｓ ), and performing image segmentation on the source domain image based on a first segmentation loss function (L ｓｅｇ ) and The second decoder detects the first target domain feature (v ｔ ) and the second target domain feature (d ｔ ) and reconstruct the target domain image based on the reconstruction loss function (L ｒｅｃ ) and The second decoder detects the first source domain features (v ｓ ) and the second target domain feature (d ｔ ) generating a transformed image based on the the encoder extracting features from the transformed image; The first decoder performs image segmentation on the transformed image based on the content-related portion of the extracted features, and calculates a second segmentation loss function (L ｓｅｇ’ ) and training the encoder, the first decoder, and the second decoder based on the first partition loss function, the second partition loss function, and the reconstruction loss function; performing image segmentation on the target domain image to be processed using the trained encoder and the first decoder; performing image segmentation using the trained neural network on a target domain image and one or more transformed images of the target domain image to generate a plurality of segmentation probability maps; generating pseudo-labels for the target domain image based on the plurality of segmentation probability maps; a U-net network performing image segmentation on the target domain image and the source domain image, respectively; training the U-net network using pseudo-labels of the target domain images and known labels of the source domain images; The method further includes performing image segmentation on the target domain image to be processed using the trained U-net network.

2. The first source domain feature (v ｓ ) and the second source domain feature (d ｓ classifying the source domain features based on channels such that each of the source domain features contains the same number of channels; The first target domain feature (v ｔ ) and the second target domain feature (d ｔ 2. The method of claim 1, wherein the target domain features are classified based on channels such that both the first and second vectors contain the same number of channels.

3. The first split loss function (L ｓｅｇ ) and the second split loss function (L ｓｅｇ’ ) is the cross-entropy loss function, The reconstruction loss function (L ｒｅｃ 2. The method of claim 1 , wherein σ is an absolute value loss function.

4. The method of claim 1, further comprising a step of transferring knowledge of the trained neural network to the U-net network by knowledge distillation so as to use the U-net network to perform image segmentation on the target domain image to be processed.

5. The step of generating pseudo labels includes: calculating an average probability map based on the plurality of split probability maps; and performing a binary split operation on the average probability map based on a threshold to obtain the pseudo label.

6. When training the U-net network, a segmentation loss function for the U-net network performing image segmentation on the target domain image; a segmentation loss function for the U-net network performing image segmentation on the source domain image; The segmentation probability map (p ｔ ) is the segmentation probability map (p ｔ ζ ) and the KL divergence to get closer to The method of claim 1, further comprising: using a conditional feature alignment loss function to bring pixel segmentation features extracted by the U-net network for the target domain image and pixel segmentation features extracted for the source domain image closer to each other for a given class.

7. 1. An apparatus for performing domain adaptation applied to image segmentation utilizing a neural network, comprising: the neural network includes an encoder, a first decoder, and a second decoder; The device comprises: a memory in which a computer program is stored; one or more processors; The processor executes the computer program, the encoder extracting source domain features for a source domain image and target domain features for a target domain image; The source domain features are then converted into first source domain features (v) related to content in the source domain image. ｓ ) and a second source domain feature (d ｓ ) and classifying the target domain features into first target domain features (v ｔ ) and a second target domain feature (d ｔ ) and The first decoder detects the first source domain features (v ｓ ), and performing image segmentation on the source domain image based on a first segmentation loss function (L ｓｅｇ ) and The second decoder detects the first target domain feature (v ｔ ) and the second target domain feature (d ｔ ) and reconstruct the target domain image based on the reconstruction loss function (L ｒｅｃ ) and The second decoder detects the first source domain features (v ｓ ) and the second target domain feature (d ｔ ) generating a transformed image based on the the encoder extracting features from the transformed image; The first decoder performs image segmentation on the transformed image based on the content-related portion of the extracted features, and calculates a second segmentation loss function (L ｓｅｇ’ ) and training the encoder, the first decoder, and the second decoder based on the first partition loss function, the second partition loss function, and the reconstruction loss function; performing image segmentation on the target domain image to be processed using the trained encoder and the first decoder; The processor: performing image segmentation using the trained neural network on a target domain image and one or more transformed images of the target domain image to generate a plurality of segmentation probability maps; generating pseudo-labels for the target domain image based on the plurality of segmentation probability maps; a U-net network performing image segmentation on the target domain image and the source domain image, respectively; training the U-net network using pseudo-labels of the target domain images and known labels of the source domain images; and performing image segmentation on the target domain image to be processed using the trained U-net network.

8. 10. A storage medium having a computer program stored thereon, the computer program causing the computer to execute the method for performing domain adaptation according to any one of claims 1 to 6 when the computer program is executed by the computer.

Images