Neural Network for Domain Adaptation and Its Training Method

Abstract

To provide a neural network for domain adaptation and a training method thereof.SOLUTION: A neural network includes a division unit and a reconfiguration unit which are connected in series. The division unit includes a first encoder and a first decoder. The reconfiguration unit includes a second encoder and a second decoder. A training method includes: extracting, by the first encoder, first features on the basis of a source domain image or a target domain image; generating, by the first decoder, a source domain division prediction image or a target domain division prediction image on the basis of the first features; extracting, by the second encoder, second features on the basis of the target domain image and the target domain division prediction image; generating, by the second decoder, a target domain reconfiguration image on the basis of the first features and the second features; and training the division unit and the reconfiguration unit on the basis of a division loss function and a reconfiguration loss function, respectively.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to domain adaptation, and in particular, to unsupervised domain adaptation for image segmentation, a corresponding neural network, and a training method for the neural network.

Background Art

[0002] Image segmentation is one of the basic visual tasks for analyzing images, and it is used to divide an image into a plurality of different regions associated with a plurality of semantic categories at the pixel level. In the predicted image obtained by performing image segmentation on the input image, pixels in a region associated with a certain semantic category are represented by a color corresponding to the semantic category, so that each object belonging to a different semantic category in the predicted image can be recognized based on the color.

[0003] Unsupervised domain adaptation refers to transferring a model trained using labeled source domain data to a target domain of unlabeled data, and at the same time, maintaining the performance of the model on the target domain as much as possible. Since there is a deviation (regional deviation) in the data set between the source domain and the target domain, and there is a lack of labeled data in the target domain, in many cases, the performance of a model trained using only labeled source domain data on the target domain is low. The training process of unsupervised domain adaptation can effectively reduce the regional deviation and improve the robustness of the model by simultaneously using the labeled data of the source domain and the unlabeled data of the target domain.

[0004] Currently, for unsupervised domain adaptation, methods such as learning domain invariant features represented by adversarial training and multi-task learning methods have been proposed. In the multi-task learning method, multi-tasks (for example, source domain data classification task and target domain data reconstruction task) are used to learn domain invariant features. Among them, the classification task and the reconstruction task share a feature extractor, and by alternately training for the classification of source domain data and the reconstruction of target domain data, the features extracted by the feature extractor can satisfy the performance requirements of both the classification task of the source domain and the reconstruction task of the target domain simultaneously. The shared features extracted in such a manner are domain invariant features.

[0005] However, there is a problem in the above multi-task learning method that it is difficult to extract optimal domain invariant features. This is because excellent reconstruction of target domain data requires not only domain invariant features but also target domain specific features that can reflect the characteristics of the target domain data. In the existing methods, the reconstruction of target domain data is performed only based on shared features, so excellent reconstruction cannot be achieved, and it is also difficult for the shared features to converge to optimal domain invariant features through training.

[0006] Furthermore, so far, research on unsupervised domain adaptation has mainly focused on the image classification task, and there has been little research on unsupervised domain adaptation for the image segmentation task. Summary of the Invention Problems to be Solved by the Invention

[0007] In view of the above technical problems, an object of the present invention is to provide a new multi-task learning-based teacherless domain adaptation scheme that can extract optimal domain-invariant features by separating (decoupling) domain-invariant features and domain-specific features. Further, the domain adaptation scheme according to the present invention can also be applied to an image segmentation task. **Means for Solving the Problems**

[0008] According to one aspect of the present invention, there is provided a method for training a neural network for domain adaptation, wherein the neural network includes a segmentation unit and a reconstruction unit connected in series, the segmentation unit is configured to perform image segmentation on a source domain image and a target domain image, and includes a first encoder and a first decoder, the reconstruction unit is configured to perform image reconstruction on the target domain image, and includes a second encoder and a second decoder. The method includes extracting first features based on the source domain image or the target domain image by the first encoder; generating a source domain segmentation prediction image or a target domain segmentation prediction image based on the first features by the first decoder; extracting second features based on the target domain image and the target domain segmentation prediction image by the second encoder; generating a target domain reconstruction image based on the first features and the second features by the second decoder; and training the segmentation unit and the reconstruction unit based on a segmentation loss function and a reconstruction loss function, respectively.

[0009] According to another aspect of the present invention, there is provided an apparatus for training a neural network for domain adaptation, wherein the neural network includes a segmentation unit and a reconstruction unit connected in series, the segmentation unit configured to perform image segmentation on a source domain image and a target domain image and including a first encoder and a first decoder, and the reconstruction unit configured to perform image reconstruction on the target domain image and including a second encoder and a second decoder. The apparatus includes: a storage device storing a computer program; and a processor configured to execute the computer program to cause the first encoder to extract first features based on the source domain image or the target domain image; cause the first decoder to generate a source domain segmented predicted image or a target domain segmented predicted image based on the first features; cause the second encoder to extract second features based on the target domain image and the target domain segmented predicted image; and cause the second decoder to generate a target domain reconstructed image based on the first features and the second features; and to train the segmentation unit and the reconstruction unit based on a segmentation loss function and a reconstruction loss function, respectively.

[0010] According to another aspect of the present invention, there is provided a neural network for performing domain adaptation, implemented by a computer, which includes a segmentation unit configured to perform image segmentation on source domain images and target domain images; and a reconstruction unit connected in series to the segmentation unit and configured to perform image reconstruction on the target domain images. The segmentation unit includes a first encoder configured to extract first features based on the source domain image or the target domain image; and a first decoder configured to generate a source domain segmentation prediction image or a target domain segmentation prediction image based on the first features. The reconstruction unit includes a second encoder configured to extract second features based on the target domain image and the target domain segmentation prediction image; and a second decoder configured to generate a target domain reconstruction image based on the first features and the second features. Among them, the segmentation unit and the reconstruction unit are trained respectively based on a segmentation loss function and a reconstruction loss function, and image segmentation is performed on a target domain image waiting to be processed using the trained segmentation unit.

[0011] According to another aspect of the present invention, there is provided a storage medium storing a computer program, which, when executed by a computer, causes the computer to execute the method for training a neural network for domain adaptation as described above.

Brief Description of the Drawings

[0012] [Figure 1] It is a diagram showing an example of a multi-task learning model of the prior art. [Figure 2] It is a diagram showing a domain adaptation model based on multi-task learning of the present invention. [Figure 3] It is a diagram showing an exemplary architecture of a segmentation network and a reconstruction network of the present invention. [Figure 4] It is a module configuration block diagram of the domain adaptation model of the present invention. [Figure 5] This is a flowchart of a method for training a domain adaptation model of the present invention. [Figure 6] This is a diagram showing the visualization effect of retinal vessel image segmentation as an experimental result. [Figure 7] This is an exemplary configuration block diagram of computer hardware capable of implementing the present invention.

Embodiments for Carrying Out the Invention

[0013] Hereinafter, with reference to the accompanying drawings, preferred embodiments for carrying out the present invention will be described in detail. It should be noted that such embodiments are merely illustrative and do not limit the present invention.

[0014] FIG. 1 shows an example of a conventional multi-task learning model, namely, a deep reconstruction-classification network (DRCN). As shown in FIG. 1, an image is input to a feature extractor 100 and undergoes the following operations: convolution (Conv), max pooling, conversion of multi-dimensional data to one dimension (Flatten), fully connected (Dense), and dropout (Dropout). As shown in the upper branch in FIG. 1, as a result of the operations, the feature extractor 100 outputs a feature F. The extracted feature F is used for supervised classification training on source domain data. Further, the extracted feature F is also used for unsupervised reconstruction training on target domain data. As shown in the lower branch in FIG. 1, the reconstruction process is the reverse process of the process executed by the feature extractor 100 and includes the following operations: conversion of one-dimensional data to multi-dimensions (Unflatten), fully connected (Dense), upsampling, and convolution (Conv). The classification task in the source domain and the reconstruction task in the target domain share the feature extractor 100. By alternately executing the classification training and the reconstruction training, the feature extractor 100 can extract domain-invariant features (shared features) that satisfy all requirements for the tasks on the two domains.

[0015] However, the classification task on the source domain is generally only related to domain-invariant features. Moreover, the reconstruction on the target domain requires not only domain-invariant features but also target-domain-specific features to obtain excellent reconstruction effects. For the DRCN network, since there are no target-domain-specific features that can reflect the characteristics of the target-domain data, it is difficult to achieve excellent reconstruction. Correspondingly, it is also difficult for the shared feature F extracted by the feature extractor 100 to converge to optimal domain-invariant features through training.

[0016] FIG. 2 is a diagram showing a multi-task learning-based domain adaptation model of the present invention. Different from the model that shares the features extracted by the same feature extractor for the segmentation and reconstruction shown in FIG. 1, the model of the present invention includes two encoder-decoder networks connected in series to decouple domain-invariant features and domain-specific features.

[0017] As shown in FIG. 2, the encoder-decoder network 210 is used for an image segmentation task (hereinafter, also referred to as a "segmentation network"), and the encoder-decoder network 220 is used for an image reconstruction task (hereinafter, also referred to as a "reconstruction network"). As shown in the upper part of FIG. 2, in the training stage, source-domain data is input into the segmentation network 210 and undergoes supervised segmentation training in the segmentation network 210. Also, as shown in the lower part of FIG. 2, target-domain data is sequentially input into the segmentation network 210 and the reconstruction network 220. In particular, in the reconstruction network 220, reconstruction training is performed based on the segmentation prediction result output by the segmentation network 210 and the features extracted by the segmentation network 210.

[0018] The reconstructed network 220 uses the prediction results and extracted features of the segmentation network 210 as inputs. Therefore, through training, on the one hand, the segmentation network 210 can extract domain-invariant features that can simultaneously satisfy the requirements of both the segmentation task in the source domain and the reconstruction task in the target domain. On the other hand, the reconstructed network 220 can extract target-domain-specific features, thereby realizing the decoupling of domain-invariant features and domain-specific features.

[0019] Specifically, referring to the upper part of FIG. 2, for the source-domain image, define f s : X → Y, and regard it as a function that maps the input image X (for example, the image Img_ps in FIG. 2) to the segmentation result Y (for example, the image Img_ss). Denote the encoder of the segmentation network 210 as S enc : X → F s and define that it receives the input image X (for example, the image Img_ps) and outputs the feature F s . Also, denote the decoder of the segmentation network 210 as S dec : {F s , h s [[ID=...]]}→Y and define that it receives the feature F s output by the encoder and the feature h s of the hidden layer of the encoder, and outputs the segmentation prediction result Y (for example, the image Img_ss).

[0020] Also, referring to the lower part of FIG. 2, for the target-domain image, define f r : X → X, and regard it as a function that maps the input image X (for example, the image Img_pt) to the input image to be reconstructed (for example, the image Img_rt). The image data of the target domain sequentially passes through both the segmentation network 210 and the reconstructed network 220. Since the segmentation network 210 has already been described, its detailed description is omitted here. For the reconstructed network 220, denote its encoder as R enc : {X, f s (X)} → F r, which receives an input image (e.g., image Img_pt) and the segmentation prediction result (e.g., image Img_st) output by the segmentation network 210, and defines a feature F r Also, the decoder of the reconstruction network 220 is output as R dec :{F r ,F s ,h s}→X, which has feature F r , F s and h s (where, feature F s and h s is from the splitting network 210) and outputs a reconstructed image (eg, image Img_rt).

[0021] As shown in Figure 2, the source domain image processing and the target domain image processing jointly use a partitioning network 210. enc The features Fs generated by the encoder R are shared features, which correspond to the common characteristics of the data in the two domains. On the other hand, the reconstruction network 220 is only used to process the target domain image. Therefore, the encoder R enc The features Fr generated by

[0049] are target domain-specific features because they are specific to the characteristics of the target domain data. By using the target domain-specific features Fr, the reconstruction network 220 of the present invention can achieve reconstruction with better results than conventional models. Based on this, by training the model of the present invention, the shared features Fs can converge to optimal domain-invariant features. The model of the present invention can obtain optimal domain-invariant features. Therefore, when applied to the target domain dataset, it can still maintain good performance, thereby improving the effectiveness of domain adaptation.

[0022] The input source domain image is x s and the input target domain image is denoted as x t Then, the function f is expressed as sand f r can each represent

Number

[0023] Here, the symbol ○ represents the connection between the decoder and the encoder, and the code (Outer 1) TIFF0007715033000002.tif10127 represents the connection between the features of the encoder of the segmentation network and the features of the encoder of the reconstruction network.

[0024] In training, the segmentation loss function L seg and the reconstruction loss function L rec can be used to train the segmentation network 210 and the reconstruction network 220 respectively. As an example, the segmentation loss function L seg can adopt the cross-entropy loss function, and the reconstruction loss function L rec can adopt the absolute value loss function. In particular, in the case of binary segmentation of images (for example, segmentation of retinal blood vessel images described later), the segmentation loss function L seg can adopt the binary cross-entropy loss function.

[0025] The segmentation loss function L seg and the reconstruction loss function L rec can be represented by the following formulas (3) and (4).

Number

[0026] Here, y i k,j represents the k-th type of binary label of the pixel j in the source domain image sample x i s , n s indicates the number of source domain images in each batch, and n trepresents the number of target domain images in each batch, K represents the number of categories, and H and W represent the length and width of the image, respectively.

[0027] Preferably, preprocessing is performed on the original image before inputting it into the segmentation network. The preprocessing may include, for example, conversion of the image to a grayscale image, normalization, histogram equalization, and gamma correction. Further, the preprocessing may further include data enhancement processing such as rotation, folding, and color dithering in order to improve the generalization ability of the model. Also, when performing a transfer task from a high-resolution dataset to a low-resolution dataset, the preprocessing may further include randomly adding Gaussian noise to the high-resolution image, and this processing can improve the effect of domain adaptation. Further, the preprocessing may further include changing the size of the image. For example, the image may be adjusted to a unified size (e.g., 512×512) by bicubic interpolation.

[0028] As shown in FIG. 2, the source domain image Img_ps input to the segmentation network 210 is an image obtained after performing preprocessing on the original source domain image Img_s, and the target domain image Img_pt input to the segmentation network 210 is an image obtained after performing preprocessing on the original target domain image Img_t.

[0029] Also, as one exemplary application, image segmentation in the present invention can be a segmentation task for retinal blood vessel images. Each image shown in FIG. 2, for example, Img_s, Img_pt, Img_ss, Img_rt, etc., are respectively the original retinal blood vessel image and its corresponding preprocessed image, segmentation prediction image, reconstructed image, etc. In this example, the domain-invariant feature Fs extracted by the segmentation network 210 can represent the common characteristics (e.g., characteristics related to the blood vessel structure) of the blood vessel images in the source domain and the target domain, and the domain-specific feature Fr extracted by the reconstruction network 220 can represent the characteristics specific to the blood vessel images in the target domain (e.g., characteristics such as the background and style of the image). However, as is clear, the model shown in FIG. 2 can be applied to the image segmentation task performed on any type of image.

[0030] Hereinafter, an exemplary network architecture of the segmentation network and the reconstruction network of the present invention will be described with reference to FIG. 3. As an example, both the segmentation network 210 and the reconstruction network 220 in the present invention can adopt a U-Net structure. The U-Net structure is described in detail in the paper "U-net: Convolutional networks for biomedical image segmentation" published by Olaf Ronneberger et al. at the "International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI)" held in 2015, and the content of the paper is incorporated herein by reference.

[0031] Specifically, each of the decomposition network 210 and the reconstruction network 220 includes an encoder and a decoder. Referring to the left half of Figure 3, the encoder includes five serially connected convolutional modules, each consisting of two 3x3 convolutional layers, with adjacent convolutional modules connected by a 2x2 max-pooling layer. Referring to the right half of Figure 3, the decoder includes five convolutional modules with the same structure as the encoder, but with adjacent convolutional modules connected by a 2x2 upsampling layer. Furthermore, the last four convolutional modules of the decoder connect hidden features extracted by the encoder's hidden layer with those extracted by the corresponding hidden layer in the decoder before performing convolution, i.e., skip connections as shown in Figure 3.

[0032] It should be noted that the division network and reconstruction network of the present invention are not limited to the U-Net network structure, and those skilled in the art can realize them using any suitable known network.

[0033] FIG. 4 is a block diagram showing the module configuration of the domain adaptation model in the present invention, and FIG. 5 is a flowchart showing the method for training the domain adaptation model in the present invention.

[0034] 4, the domain adaptation model of the present invention includes a division unit 410 and a reconstruction unit 420 connected in series. The division unit 410 is used to perform image division on a source domain image and a target domain image, the reconstruction unit 420 is used to perform image reconstruction on the target domain image, and the reconstruction unit 420 receives the division prediction result output by the division unit 410. In addition, the division unit 410 includes a first encoder 411 and a first decoder 412, and the reconstruction unit 420 includes a second encoder 421 and a second decoder 422.

[0035] As shown in FIG. 5, in step S510, the first encoder 411 of the segmentation unit 410 extracts a first feature (e.g., Fs) based on the input source domain image (e.g., Img_ps in FIG. 2) or target domain image (e.g., Img_pt).

[0036] In step S520, the first decoder 412 of the segmentation unit 410 generates a source domain segmentation prediction image (e.g., Img_ss) or target domain segmentation prediction image (e.g., Img_st) based on the first feature.

[0037] In step S530, the second encoder 421 of the reconstruction unit 420 extracts a second feature (e.g., Fr) based on the input target domain image and the target domain segmentation prediction image generated by the segmentation unit 410.

[0038] In step S540, the second decoder 422 of the reconstruction unit 420 generates a target domain reconstruction image (e.g., Img_rt) based on the first feature and the second feature.

[0039] In step S550, the segmentation unit 410 and the reconstruction unit 420 are trained based on a segmentation loss function and a reconstruction loss function, respectively. As described above, for example, a cross-entropy loss function and an absolute value loss function can be adopted.

[0040] After the training is completed, image segmentation can be performed on the target domain image waiting for processing using the trained segmentation unit 410. Specifically, the first encoder 411 of the trained segmentation unit 410 extracts domain-invariant features from the target domain image waiting for processing, and then, the first decoder 412 of the trained segmentation unit 410 can generate a segmentation prediction image based on the domain-invariant features. As described above, since the trained segmentation unit 410 can extract optimal domain-invariant features, it can still maintain good performance when applied to the target domain dataset.

[0041] As described above, the multi-task learning-based domain adaptation model of the present invention has been described, and decoupling of domain-invariant features and domain-specific features can be achieved using this model. In the reconstruction task, since features specific to the target domain are used, better reconstruction can be realized. Therefore, through multi-task training, the shared features can converge to optimal domain-invariant features, thereby improving the effect of domain adaptation.

[0042] The inventor of the present invention is evaluating the performance of the present invention by conducting experiments based on retinal vascular images. The experiments are conducted based on open datasets, and the open datasets include DRIVE (abbreviated as "D" in Table 1), CHASE_DB1 (abbreviated as "C"), STARE (abbreviated as "S"), HRF (abbreviated as "H"), and IOSTAR (abbreviated as "I"). In the experiment, among the above datasets, any one dataset is used as the labeled source domain data, and one of the other datasets is used as the unlabeled target domain data to perform the domain transfer task. The evaluation metrics adopted in the experiment include the widely used F1 score (DICE), Matthews correlation coefficient (Matthews Correlation Coeffcient, MCC), and area under the ROC curve (Area Under Curve, AUC).

[0043] We are testing the performance of the present invention and several conventional schemes in experiments. The conventional schemes include source-only (a scheme that uses only the source domain dataset during training and does not use the target domain dataset), domain adversarial neural network (DANN), deep reconstruction-classification network (DRCN) shown in FIG. 1, supervised training based on synthetic images, and a method of fine-tuning using pseudo levels (Pseudo-l). The following Table 1 shows the experimental results. Among them, the evaluation indicators "F1 score / MCC / AUC" are represented as percentages, and the upper bound represents the score obtained when training is performed using the labeled target domain dataset, which can be used as a reference to represent the optimal score that can be obtained for each transfer task. Therefore, as can be seen, the performance of the scheme of the present invention is superior to other conventional schemes.

Table 1

[0044] Note that "DRNN (ours)" in Table 1 means the present invention.

[0045] Also, FIG. 6 is a diagram showing the visualization effect of retinal vessel image segmentation as an experimental result. In this experiment, a transfer task from the HRF dataset to the CHASE_DB1 dataset is executed. The HRF dataset contains 30 high-quality images with a size of 3504×2336, and the CHASE_DB1 dataset contains 20 images with a size of 999×960. In the training stage, 15 images in the HRF dataset are used as labeled source domain images, and 8 images in the CHASE_DB1 dataset are used as unlabeled target domain images. In the verification stage, the remaining images in the target domain CHASE_DB1 dataset are used as test images for model performance verification.

[0046] As can be seen from FIG. 6, when performing image segmentation on a retinal blood vessel image, the segmentation effect of the scheme of the present invention is superior to other conventional schemes.

[0047] The method described in the above embodiments can be implemented by software, hardware, or a combination of software and hardware. The programs included in the software may be stored in a storage medium installed inside or outside the device in advance. As an example, during the execution period, these programs are written into the RAM and executed by a processor (e.g., a CPU), thereby realizing the various methods and processes described above.

[0048] FIG. 7 is a configuration diagram of a hardware configuration (general-purpose machine) 700 capable of realizing an embodiment of the present invention.

[0049] The general-purpose machine 700 may be, for example, a computer system. Note that the general-purpose machine 700 is merely an example and does not limit the application scope or functions of the method and apparatus according to the present invention. Also, the general-purpose machine 700 does not depend on any module, assembly, etc. in the above-described method and apparatus or a combination thereof.

[0050] In FIG. 7, the central processing unit (CPU) 701 performs various processes based on a program stored in the ROM 702 or a program loaded from the storage unit 708 into the RAM 703. In the RAM 703, data and the like necessary when the CPU 701 performs various processes can also be stored according to needs. The CPU 701, ROM 702, and RAM 703 are connected to each other via a bus 704. The input / output interface 705 is also connected to the bus 704.

[0051] In addition, the following components are further connected to the input / output interface 705, that is, an input unit 706 including a keyboard and the like, a display unit such as a liquid crystal display (LCD), an output unit 707 including a speaker and the like, a storage unit 708 including a hard disk and the like, and a communication unit 709 including a network interface card, for example, a LAN card, a modem, and the like. The communication unit 709 performs communication processing via a network such as the Internet or a LAN. The drive 710 may be connected to the input / output interface 705 according to needs. A removable medium 1311, for example, a semiconductor memory or the like, can be set in the drive 710 as needed, and a computer program read from it can be installed in the storage unit 708.

[0052] Furthermore, the present invention further provides a program product including machine-readable instruction codes. When such instruction codes are read and executed by a machine (computer), the method in the above-described embodiment of the present invention can be executed. Correspondingly, various storage media that carry such a program product, for example, magnetic disks (including floppy disks (registered trademarks)), optical disks (including CD-ROMs and DVDs), magneto-optical disks (including MDs (registered trademarks)), and semiconductor memories, are also included in the present invention.

[0053] The above-described storage media may include, for example, magnetic disks, optical disks, magneto-optical disks, semiconductor memories, etc., but are not limited thereto.

[0054] In addition, each operation (processing) in the above-described method can also be realized in the form of a computer-executable program stored in various machine-readable storage media.

[0055] In addition, regarding the above embodiments and the like, the following is further disclosed as an appendix.

[0056] (Appendix 1) A method for training a neural network for domain adaptation, comprising: The neural network includes a splitting unit and a reconstruction unit connected in series. The splitting unit is configured to perform image splitting on a source domain image and a target domain image, and includes a first encoder and a first decoder. The reconstruction unit is configured to perform image reconstruction on the target domain image and includes a second encoder and a second decoder. The method includes: Extracting first features based on the source domain image or the target domain image by the first encoder; Generating a source domain segmentation prediction image or a target domain segmentation prediction image based on the first features by the first decoder; Extracting second features based on the target domain image and the target domain segmentation prediction image by the second encoder; Generating a target domain reconstruction image based on the first features and the second features by the second decoder; and Training the splitting unit and the reconstruction unit based on a segmentation loss function and a reconstruction loss function, respectively.

[0057] (Appendix 2) The method according to Appendix 1, wherein The first features are domain-invariant features corresponding to common characteristics of the source domain image and the target domain image, and the second features are domain-specific features corresponding to unique characteristics of the target domain image.

[0058] (Appendix 3) The method according to Appendix 2, further comprising: Extracting the domain-invariant features for a target domain image to be processed using the first encoder of the trained splitting unit; and Performing image splitting on the target domain image to be processed based on the domain-invariant features using the first decoder of the trained splitting unit.

[0059] (Appendix 4) 2. The method of claim 1, comprising: the first encoder has a plurality of hidden layers, and extracts hidden features at at least some of the hidden layers; Wherein, the first decoder generates the source domain divided predicted image or the target domain divided predicted image according to the first feature and the hidden feature; and wherein the second decoder generates the target domain reconstructed image based on the first feature and the hidden feature from the first encoder and the second feature from the second encoder.

[0060] (Appendix 5) 2. The method of claim 1, comprising: The method, wherein the splitting loss function is a cross-entropy loss function and the reconstruction loss function is an absolute value loss function.

[0061] (Appendix 6) The method of claim 1, further comprising: performing preprocessing on the source domain image, and inputting the preprocessed source domain image to the first encoder; and The method includes performing preprocessing on the target domain image and inputting the preprocessed target domain image to the first encoder and the second encoder.

[0062] (Appendix 7) 7. The method of claim 6, The pre-processing includes at least one of the following operations: conversion to grayscale image, normalization, histogram equalization, gamma correction, rotation, folding transformation, color dithering, and resizing; Wherein, if the source domain image is a high-resolution image and the target domain image is a low-resolution image, the preprocessing further includes randomly adding Gaussian noise.

[0063] (Appendix 8) The method according to Appendix 1, wherein both the splitting unit and the reconstruction unit have a U-Net network structure.

[0064] (Appendix 9) A training device for a neural network for domain adaptation, comprising a splitting unit and a reconstruction unit connected in series, the splitting unit being configured to perform image segmentation on a source domain image and a target domain image, and including a first encoder and a first decoder, the reconstruction unit being configured to perform image reconstruction on the target domain image and including a second encoder and a second decoder, the device comprising a memory storing a computer program; and a processor, which, by executing the computer program, causes the first encoder to extract first features based on the source domain image or the target domain image; causes the first decoder to generate a source domain segmentation prediction image or a target domain segmentation prediction image based on the first features; causes the second encoder to extract second features based on the target domain image and the target domain segmentation prediction image; causes the second decoder to generate a target domain reconstruction image based on the first features and the second features; and is configured to train the splitting unit and the reconstruction unit respectively based on a segmentation loss function and a reconstruction loss function.

[0065] (Appendix 10) A neural network for performing domain adaptation implemented by a computer, comprising a splitting unit configured to perform image segmentation on a source domain image and a target domain image, wherein the splitting unit A first encoder for extracting a first feature based on the source domain image or the target domain image; and including a first decoder for generating a source domain segmentation prediction image or a target domain segmentation prediction image based on the first feature, further including a reconstruction unit connected in series with the segmentation unit, the reconstruction unit being for performing image reconstruction on the target domain image, The reconstruction unit, a second encoder for extracting a second feature based on the target domain image and the target domain segmentation prediction image; and including a second decoder for generating a target domain reconstruction image based on the first feature and the second feature, wherein, respectively, based on a segmentation loss function and a reconstruction loss function, the segmentation unit and the reconstruction unit are trained, and image segmentation is performed on a target domain image waiting to be processed using the trained segmentation unit. An apparatus.

[0066] (Appendix 11) A storage medium storing a computer program, when the computer program is executed by a computer, causing the computer to execute a method for training a neural network for domain adaptation according to any one of Appendices 1-8. A storage medium.

[0067] As described above, the preferred embodiments of the present invention have been described. However, the present invention is not limited to this embodiment. Without departing from the spirit of the present invention, any changes to the present invention belong to the technical scope of the present invention.

Claims

1. A method for training a neural network for domain adaptation, comprising: the neural network includes a segmentation unit and a reconstruction unit connected in series, the segmentation unit is configured to perform image segmentation on a source domain image and a target domain image, and includes a first encoder and a first decoder, the reconstruction unit is configured to perform image reconstruction on the target domain image, and includes a second encoder and a second decoder; the method includes: the first encoder extracts first features based on the source domain image or the target domain image; the first decoder generates a source domain segmentation prediction image or a target domain segmentation prediction image based on the first features; the second encoder extracts second features based on the target domain image and the target domain segmentation prediction image; the second decoder generates a target domain reconstruction image based on the first features and the second features; and training the segmentation unit and the reconstruction unit based on a segmentation loss function and a reconstruction loss function respectively.

2. The method according to claim 1, wherein: the first features are domain-invariant features corresponding to common characteristics of the source domain image and the target domain image, and the second features are domain-specific features corresponding to unique characteristics of the target domain image.

3. The method according to claim 2, further comprising: extracting the domain-invariant features for a target domain image to be processed using the first encoder of the trained segmentation unit; and performing image segmentation on the target domain image to be processed based on the domain-invariant features using the first decoder of the trained segmentation unit.

4. The method according to claim 1, wherein: the first encoder has a plurality of hidden layers, and hidden features are extracted at at least some of the plurality of hidden layers; the first decoder generates the source domain segmentation prediction image or the target domain segmentation prediction image based on the first features and the hidden features; and the second decoder generates the target domain reconstruction image based on the first features from the first encoder, the hidden features, and the second features from the second encoder.

5. The method according to claim 1, wherein the segmentation loss function is a cross-entropy loss function, and the reconstruction loss function is an absolute value loss function. **Claim 6** The method according to claim 1, further comprising performing preprocessing on the source domain image and inputting the preprocessed source domain image into the first encoder; and performing preprocessing on the target domain image and inputting the preprocessed target domain image into the first encoder and the second encoder. **Claim 7** The method according to claim 6, wherein the preprocessing includes at least one of the following processes, namely, conversion to a grayscale image, normalization, histogram equalization, gamma correction, rotation, folding, color dithering, and resizing, when the source domain image is a high-resolution image and the target domain image is a low-resolution image, the preprocessing further includes randomly adding Gaussian noise. **Claim 8** The method according to claim 1, wherein both the segmentation unit and the reconstruction unit have a U-Net network structure. **Claim 9** An apparatus for training a neural network for domain adaptation, comprising the neural network includes a segmentation unit and a reconstruction unit connected in series, the segmentation unit is configured to perform image segmentation on a source domain image and a target domain image, and includes a first encoder and a first decoder, the reconstruction unit is configured to perform image reconstruction on the target domain image, and includes a second encoder and a second decoder, the apparatus further comprises a memory storing a computer program; and a processor connected to the memory, wherein the processor, by executing the computer program, causes the first encoder to extract first features based on the source domain image or the target domain image; causes the first decoder to generate a source domain segmentation prediction image or a target domain segmentation prediction image based on the first features; causes the second encoder to extract second features based on the target domain image and the target domain segmentation prediction image; causes the second decoder to generate a target domain reconstruction image based on the first features and the second features; and An apparatus configured to train the segmentation unit and the reconstruction unit respectively based on a segmentation loss function and a reconstruction loss function. **Claim 10** A program for causing a computer to execute a method for training a neural network for domain adaptation according to any one of claims 1 to 8.

Images