Neural Networks for Domain Adaptation and Their Training Methods

Abstract

A neural network for domain adaptation and its training method are disclosed. The neural network includes a segmentation unit and a reconstruction unit connected in series. The segmentation unit performs image segmentation on a source domain image and a target domain image and includes a first encoder and a first decoder. The reconstruction unit performs image reconstruction on the target domain image and includes a second encoder and a second decoder. The training method includes: extracting a first feature based on the source domain image or the target domain image by the first encoder; generating a source domain segmentation prediction image or a target domain segmentation prediction image by the first decoder based on the first feature; extracting a second feature based on the target domain image and the target domain segmentation prediction image by the second encoder; generating a target domain reconstructed image by the second decoder based on the first and second features; and training the segmentation unit and the reconstruction unit based on a segmentation loss function and a reconstruction loss function, respectively.

Description

Technical Field

[0001] This invention generally relates to domain adaptation, and more specifically, to unsupervised domain adaptation for image segmentation and the corresponding neural network, as well as a method for training the neural network. Background Technology

[0002] Image segmentation is one of the fundamental visual tasks in image analysis, used to divide an image into multiple distinct regions associated with various semantic categories at the pixel level. In the predicted image obtained by performing image segmentation on the input image, pixels in regions associated with a certain semantic category are displayed with the color corresponding to that semantic category, thus enabling the identification of objects belonging to different semantic categories in the predicted image based on color.

[0003] On the other hand, unsupervised domain adaptation refers to transferring a model trained using labeled source domain data to a target domain with unlabeled data, while simultaneously maintaining the model's performance in the target domain as much as possible. Due to dataset bias (domain bias) between the source and target domains, and the lack of labeled data in the target domain, models trained using labeled source domain data often perform poorly in the target domain. Unsupervised domain adaptation utilizes both labeled data from the source domain and unlabeled data from the target domain, effectively mitigating domain bias and improving model robustness.

[0004] Currently, for unsupervised domain adaptation, methods such as adversarial training for learning domain-invariant features and multi-task learning methods have been proposed. In multi-task learning methods, multiple tasks (e.g., source domain data classification and target domain data reconstruction) are used to learn domain-invariant features. The classification and reconstruction tasks share a feature extractor, which is trained alternately for source domain data classification and target domain data reconstruction. This ensures that the features extracted by the feature extractor simultaneously meet the performance requirements of both the source domain classification task and the target domain reconstruction task. The shared features extracted in this way are called domain-invariant features.

[0005] However, the aforementioned multi-task learning methods suffer from the difficulty of extracting optimal domain-invariant features. This is because high-performance target domain data reconstruction requires both domain-invariant features and target domain-specific features that reflect the characteristics of the target domain data. In existing methods, target domain data reconstruction is based solely on shared features, thus failing to achieve excellent reconstruction. Consequently, these shared features are also difficult to converge to the optimal domain-invariant features through training.

[0006] Furthermore, most current research on unsupervised domain adaptation focuses on image classification tasks, while research on unsupervised domain adaptation for image segmentation tasks is still relatively limited. Summary of the Invention

[0007] To address the aforementioned technical problems, this invention proposes a novel unsupervised domain adaptation scheme based on multi-task learning. This scheme decouples domain-invariant features and domain-specific features, thereby extracting the optimal domain-invariant features. Furthermore, the domain adaptation scheme according to this invention can be applied to image segmentation tasks.

[0008] According to one aspect of the present invention, a method for training a domain-adaptive neural network is provided, wherein the neural network includes a segmentation unit and a reconstruction unit connected in series, the segmentation unit being configured to perform image segmentation on a source domain image and a target domain image and including a first encoder and a first decoder, and the reconstruction unit being configured to perform image reconstruction on the target domain image and including a second encoder and a second decoder. The method includes: extracting a first feature by the first encoder based on the source domain image or the target domain image; generating a source domain segmentation prediction image or a target domain segmentation prediction image by the first decoder based on the first feature; extracting a second feature by the second encoder based on the target domain image and the target domain segmentation prediction image; generating a target domain reconstructed image by the second decoder based on the first feature and the second feature; and training the segmentation unit and the reconstruction unit based on a segmentation loss function and a reconstruction loss function, respectively.

[0009] According to another aspect of the present invention, an apparatus is provided for training a neural network for domain adaptation, wherein the neural network includes a segmentation unit and a reconstruction unit connected in series, the segmentation unit being configured to perform image segmentation for a source domain image and a target domain image and including a first encoder and a first decoder, and the reconstruction unit being configured to perform image reconstruction for the target domain image and including a second encoder and a second decoder. The apparatus includes: a memory storing a computer program; and a processor configured to, by executing the computer program, cause the first encoder to extract a first feature based on the source domain image or the target domain image; cause the first decoder to generate a source domain segmentation prediction image or a target domain segmentation prediction image based on the first feature; cause the second encoder to extract a second feature based on the target domain image and the target domain segmentation prediction image; cause the second decoder to generate a target domain reconstructed image based on the first feature and the second feature; and train the segmentation unit and the reconstruction unit based on a segmentation loss function and a reconstruction loss function, respectively.

[0010] According to another aspect of the present invention, a computer-implemented domain-adaptive neural network is provided, comprising: a segmentation unit configured to perform image segmentation on a source domain image and a target domain image; and a reconstruction unit connected in series with the segmentation unit, configured to perform image reconstruction on the target domain image. The segmentation unit includes: a first encoder configured to extract a first feature based on the source domain image or the target domain image; and a first decoder configured to generate a source domain segmentation prediction image or a target domain segmentation prediction image based on the first feature. The reconstruction unit includes: a second encoder configured to extract a second feature based on the target domain image and the target domain segmentation prediction image; and a second decoder configured to generate a target domain reconstructed image based on the first feature and the second feature. The segmentation unit and the reconstruction unit are trained based on a segmentation loss function and a reconstruction loss function, respectively, and the trained segmentation unit is used to perform image segmentation on the target domain image to be processed.

[0011] According to another aspect of the invention, a storage medium storing a computer program is provided, which, when executed by a computer, causes the computer to perform the training method for a domain-adaptive neural network as described above. Attached Figure Description

[0012] Figure 1 An example of an existing multi-task learning model is illustrated schematically.

[0013] Figure 2 A domain adaptation model based on multi-task learning according to the present invention is illustrated schematically.

[0014] Figure 3 An exemplary architecture of the segmented network and the reconstructed network according to the present invention is shown.

[0015] Figure 4 A block diagram of the domain adaptive model according to the present invention is shown.

[0016] Figure 5 A flowchart of a training method for a domain-adaptive model according to the present invention is shown.

[0017] Figure 6 The visualization of retinal vessel image segmentation as an experimental result is shown.

[0018] Figure 7 An exemplary configuration block diagram of computer hardware implementing the present invention is shown. Detailed Implementation

[0019] Figure 1An example of an existing multi-task learning model, namely the Deep Reconstruction-Classification Network (DRCN), is illustrated schematically. Figure 1 As shown, the image is input to feature extractor 100 and undergoes the following operations: convolution (Conv), max pooling, flattening (converting multidimensional data to one dimension), fully connected (Dense), and dropout. As a result of these operations, feature extractor 100 outputs features F. The extracted features F are used for supervised classification training on source domain data, such as... Figure 1 The upper branch is shown in the diagram. Furthermore, the extracted features F are used for unsupervised reconstruction training on the target domain data. The reconstruction process is the inverse of the process performed by the feature extractor 100, including the following operations: transforming one-dimensional data into multi-dimensional data (unflattening), fully connected (dense), upsampling, and convolution (Conv), as shown in the diagram. Figure 1 The lower branch is shown in the diagram. The classification task in the source domain and the reconstruction task in the target domain share the feature extractor 100. By alternately performing classification training and reconstruction training, the feature extractor 100 is able to extract domain-invariant features (shared features) that meet the requirements for both tasks in the two domains.

[0020] However, classification tasks in the source domain are generally only related to domain-invariant features, while reconstruction in the target domain requires not only domain-invariant features but also target-domain-specific features to achieve excellent reconstruction results. For DRCN networks, since there are no target-domain-specific features that reflect the characteristics of the target domain data, it is difficult to achieve excellent reconstruction. Consequently, the shared features F extracted by the feature extractor 100 are also difficult to converge to the optimal domain-invariant features through training.

[0021] Figure 2 A domain adaptation model based on multi-task learning according to the present invention is illustrated schematically. Unlike... Figure 1 The model shown, which segments and reconstructs features extracted by the same feature extractor, includes two cascaded encoder-decoder networks to decouple domain-invariant features and domain-specific features.

[0022] like Figure 2 As shown, encoder-decoder network 210 is used for image segmentation tasks (hereinafter referred to as the "segmentation network"), and encoder-decoder network 220 is used for image reconstruction tasks (hereinafter referred to as the "reconstruction network"). During the training phase, source domain data is fed into segmentation network 210, and supervised segmentation training is performed in segmentation network 210, such as... Figure 2 As shown in the upper part. Furthermore, as... Figure 2As shown in the lower part, the target domain data is sequentially fed into the segmentation network 210 and the reconstruction network 220. In particular, reconstruction training is performed in the reconstruction network 220 based on the segmentation prediction results output by the segmentation network 210 and the features extracted by the segmentation network 210.

[0023] Since the reconstruction network 220 uses the prediction results and extracted features of the segmentation network 210 as input, through training, on the one hand, the segmentation network 210 can extract domain-invariant features that simultaneously meet the requirements of the source domain segmentation task and the target domain reconstruction task, and on the other hand, the reconstruction network 220 can extract target domain-specific features, thereby achieving decoupling of domain-invariant features and domain-specific features.

[0024] Specifically, see Figure 2 The upper part defines the image for the source domain. As input image (For example Figure 2 The image (Img_ps) is mapped to the segmentation result. (e.g., a function for the image Img_ss). The encoder of the segmentation network 210 is defined as... It receives input images (e.g., image Img_ps) and output features Fs. Furthermore, the decoder of segmentation network 210 is defined as... It receives the characteristics Fs of the encoder output and the characteristics h of the encoder's hidden layer. s And output segmentation prediction results (e.g., image Img_ss).

[0025] In addition, see Figure 2 The lower part defines the image for the target domain. As input image A function that maps an image (e.g., Img_pt) to a reconstructed input image (e.g., Img_rt). The image data in the target domain passes sequentially through both the segmentation network 210 and the reconstruction network 220. Since the segmentation network 210 has already been described above, its description is omitted here. Regarding the reconstruction network 220, its encoder is defined as... It receives an input image (e.g., image Img_pt) and a segmentation prediction result (e.g., image Img_st) output by segmentation network 210, and outputs features Fr. Furthermore, the decoder of reconstruction network 220 is defined as... It receives features Fr, Fs and hs (where features Fs and hs come from segmentation network 210) and outputs a reconstructed image (e.g. image Img_rt).

[0026] like Figure 2As shown, the processing of the source domain image and the processing of the target domain image both use the segmentation network 210, therefore the encoder S enc The generated features Fs are shared features, corresponding to characteristics common to the data from both domains. On the other hand, the reconstruction network 220 is only used to process the target domain image, therefore the encoder R... enc The generated feature Fr corresponds to characteristics unique to the target domain data, and is therefore a target domain-specific feature. Due to the use of the target domain-specific feature Fr, the reconstruction network 220 according to the present invention can achieve better reconstruction results compared to existing models. Furthermore, through training the model according to the present invention, the shared feature Fs can converge to the optimal domain-invariant feature. Since the model according to the present invention can obtain the optimal domain-invariant feature, it can still maintain good performance when applied to the target domain dataset, thereby improving the domain adaptation effect.

[0027] Suppose the input source domain image is represented as x s And the input target domain image is represented as x t Then the function f can be represented by the following mathematical expressions (1) and (2) respectively. s and f r :

[0028] f s (x s )=(S dec ○S enc (x) s ), -(1)

[0029]

[0030] The symbol “○” represents the connection between the decoder and the encoder. This represents the connection between the features of the encoder in the segmentation network and the features of the encoder in the reconstruction network.

[0031] During training, the segmentation loss function L can be used separately. seg and reconstruction loss function L rec To train segmentation network 210 and reconstruction network 220. As an example, the segmentation loss function L... seg The cross-entropy loss function can be used to reconstruct the loss function L. rec An absolute value loss function can be used. Specifically, in the case of binary image segmentation (such as the segmentation of retinal vessel images described later), the segmentation loss function L... seg A binary cross-entropy loss function can be used.

[0032] Segmentation loss function L seg and reconstruction loss function L recIt can be represented by the following mathematical expressions (3) and (4):

[0033]

[0034]

[0035] in, Represents source domain image samples The binary label of the k-th class of pixel j in n s n represents the number of source domain images in each batch. t This indicates the number of target domain images in each batch, where K represents the number of classes, and H and W represent the length and width of the image, respectively.

[0036] Preferably, the original image is preprocessed before being input into the segmentation network. Preprocessing may include, for example, converting the image to grayscale, normalization, histogram equalization, and gamma correction. Furthermore, preprocessing may include data augmentation techniques such as rotation, flipping, and color dithering to improve the model's generalization ability. Additionally, when performing transfer tasks from high-resolution datasets to low-resolution datasets, preprocessing may include randomly adding Gaussian noise to the high-resolution image, which improves the effectiveness of domain adaptation. Furthermore, preprocessing may include resizing the image, for example, by adjusting the image to a uniform size (e.g., 512×512) using bicubic interpolation.

[0037] like Figure 2 As shown, the source domain image Img_ps input to the segmentation network 210 is the image obtained after preprocessing the original source domain image Img_s, and the target domain image Img_pt input to the segmentation network 210 is the image obtained after preprocessing the original target domain image Img_t.

[0038] Furthermore, as an example application, the image segmentation in this invention can be a segmentation task for retinal blood vessel images. Figure 2 The images shown, such as Img_s, Img_pt, Img_ss, Img_rt, etc., are the original retinal vessel images and their corresponding preprocessed images, segmentation prediction images, reconstructed images, etc., respectively. In this example, the domain-invariant features Fs extracted in the segmentation network 210 can represent the characteristics common to the vessel images in the source and target domains (e.g., characteristics related to vessel structure), while the domain-specific features Fr extracted in the reconstruction network 220 can represent the characteristics specific to the vessel images in the target domain (e.g., characteristics related to the image background, style, etc.). However, it is obvious that... Figure 2 The model shown will be applicable to image segmentation tasks performed on any type of image.

[0039] The following combination Figure 3 This paper describes the illustrative network architecture of the segmentation network and reconstruction network according to the present invention. As an example, both the segmentation network 210 and the reconstruction network 220 in this paper can adopt a U-Net structure. The U-Net structure is described in detail in the paper "U-net: Convolutional networks for biomedical image segmentation" presented by Olaf Ronneberger et al. at the 2015 International Conference on Computational Medical Images and Computer-Aided Intervention (MICCAI), the contents of which are incorporated herein by reference.

[0040] Specifically, each of the segmentation network 210 and the reconstruction network 220 includes an encoder and a decoder. See also Figure 3 The left half of the encoder consists of five convolutional modules connected in series. Each convolutional module comprises two 3×3 convolutional layers, and adjacent convolutional modules are connected by 2×2 max-pooling layers. See also Figure 3 The right half of the decoder includes five convolutional modules with the same structure as those in the encoder, but adjacent convolutional modules are connected by 2×2 upsampling layers. Furthermore, in the last four convolutional modules of the decoder, before performing convolution, the hidden features extracted by the encoder's hidden layers are concatenated with the hidden features extracted by the corresponding hidden layers in the decoder; that is, as shown... Figure 3 The skip connection shown.

[0041] It should be noted that the segmentation network and reconstruction network according to the present invention are not limited to the U-Net network structure, and those skilled in the art can easily implement them using any suitable known network.

[0042] Figure 4 A block diagram of the domain adaptive model according to the present invention is shown. Figure 5 A flowchart of a training method for a domain-adaptive model according to the present invention is shown.

[0043] like Figure 4 As shown, the domain adaptive model according to the present invention includes a segmentation unit 410 and a reconstruction unit 420 connected in series. The segmentation unit 410 is used to perform image segmentation on a source domain image and a target domain image, the reconstruction unit 420 is used to perform image reconstruction on the target domain image, and the reconstruction unit 420 receives the segmentation prediction result output by the segmentation unit 410. Furthermore, the segmentation unit 410 includes a first encoder 411 and a first decoder 412, and the reconstruction unit 420 includes a second encoder 421 and a second decoder 422.

[0044] like Figure 5 As shown, in step S510, the first encoder 411 of the segmentation unit 410, based on the input source domain image (such as... Figure 2 The first feature (e.g., Fs) is extracted from the Img_ps image or the target domain image (e.g., Img_pt).

[0045] In step S520, the first decoder 412 of the segmentation unit 410 generates a source domain segmentation prediction image (e.g., Img_ss) or a target domain segmentation prediction image (e.g., Img_st) based on the first feature.

[0046] In step S530, the second encoder 421 of the reconstruction unit 420 extracts a second feature (such as Fr) based on the input target domain image and the target domain segmentation prediction image generated by the segmentation unit 410.

[0047] In step S540, the second decoder 422 of the reconstruction unit 420 generates a target domain reconstructed image (e.g., Img_rt) based on the first feature and the second feature.

[0048] In step S550, segmentation unit 410 and reconstruction unit 420 are trained based on segmentation loss function and reconstruction loss function, respectively. As described above, cross-entropy loss function and absolute value loss function can be used, for example.

[0049] After training, the trained segmentation unit 410 can be used to perform image segmentation on the target domain image to be processed. Specifically, the first encoder 411 of the trained segmentation unit 410 extracts domain-invariant features from the target domain image to be processed, and then the first decoder 412 of the trained segmentation unit 410 generates a segmentation prediction image based on these domain-invariant features. As mentioned above, since the trained segmentation unit 410 can extract the best domain-invariant features, it can still maintain good performance when applied to the target domain dataset.

[0050] The above describes a domain adaptation model based on multi-task learning according to the present invention, which can decouple domain-invariant features and domain-specific features. In the reconstruction task, the use of target domain-specific features enables better reconstruction results. Therefore, through multi-task training, shared features can converge to the optimal domain-invariant features, thereby improving the effectiveness of domain adaptation.

[0051] The inventors of this invention have conducted experiments based on retinal vascular images to evaluate the performance of this invention. The experiments were conducted on publicly available datasets including DRIVE (referred to as "D" in Table 1), CHASE_DB1 (referred to as "C"), STARE (referred to as "S"), HRF (referred to as "H"), and IOSTAR (referred to as "I"). In the experiments, a domain transfer task was performed using any one of the above datasets as labeled source domain data and one of the other datasets as unlabeled target domain data. The evaluation metrics used in the experiments included the widely used F1 score (DICE), Matthews Correlation Coefficient (MCC), and Area Under the Curve (AUC).

[0052] The performance of this invention and some existing solutions was tested in experiments. Existing solutions include: source-only (solutions that utilize only the source domain dataset during training and not the target domain dataset), Domain Adversarial Neural Networks (DANNs). Figure 1 The Deep Reconstruction-Classification Network (DRCN), supervised training based on synthetic images, and a method for fine-tuning using pseudo-labels (Pseudo-1) are illustrated below. Table 1 below presents the experimental results, where the evaluation metrics "F1 score / MCC / AUC" are expressed as percentages, and the upper bound represents the score obtained when training with the labeled target domain dataset, which represents the best score obtainable for each transfer task and can therefore be used as a reference. It can be seen that the performance of the proposed solution outperforms other existing solutions.

[0053]

[0054] also, Figure 6 A visualization of the retinal vessel image segmentation results is shown. In this experiment, a transfer task was performed from the HRF dataset to the CHASE_DB1 dataset. The HRF dataset contains 30 high-quality images of size 3504×2336, and the CHASE_DB1 dataset contains 20 images of size 999×960. During the training phase, 15 images from the HRF dataset were used as labeled source domain images, and 8 images from the CHASE_DB1 dataset were used as unlabeled target domain images. During the validation phase, the remaining images from the target domain CHASE_DB1 dataset were used as test images to validate the model's performance.

[0055] from Figure 6As can be seen, in image segmentation of retinal blood vessel images, the segmentation effect of the present invention is better than other existing solutions.

[0056] The methods described in the above embodiments can be implemented by software, hardware, or a combination of software and hardware. Programs included in the software can be pre-stored in a storage medium located internally or externally to the device. As an example, during execution, these programs are written to random access memory (RAM) and executed by a processor (e.g., a CPU) to implement the various methods and processes described herein.

[0057] Figure 7 An example configuration block diagram of computer hardware for executing the method of the present invention according to a program is shown, and the neural network according to the present invention and the means for training the neural network can be implemented based on the computer hardware.

[0058] like Figure 7 As shown, in computer 700, central processing unit (CPU) 701, read-only memory (ROM) 702 and random access memory (RAM) 703 are connected to each other via bus 704.

[0059] The input / output interface 705 is further connected to the bus 704. The input / output interface 705 is connected to the following components: an input unit 706 formed by a keyboard, mouse, microphone, etc.; an output unit 707 formed by a display, speaker, etc.; a storage unit 708 formed by a hard disk, non-volatile memory, etc.; a communication unit 709 formed by a network interface card (such as a local area network (LAN) card, modem, etc.); and a driver 710 for driving a removable medium 711, such as a disk, optical disk, magneto-optical disk, or semiconductor memory.

[0060] In a computer with the above structure, the CPU 701 loads the program stored in the storage unit 708 into the RAM 703 via the input / output interface 705 and the bus 704, and executes the program to perform the method described above.

[0061] The program to be executed by the computer (CPU 701) can be recorded on a portable medium 711, which is formed as a packaging medium, such as a magnetic disk (including a floppy disk), an optical disk (including a compact optical disk-read-only memory (CD-ROM)), a digital multifunction optical disk (DVD), etc.), a magneto-optical disk, or a semiconductor memory. Furthermore, the program to be executed by the computer (CPU 701) can also be provided via wired or wireless transmission media such as a local area network, the Internet, or digital satellite broadcasting.

[0062] When the removable medium 711 is installed in the driver 710, the program can be installed in the storage unit 708 via the input / output interface 705. Alternatively, the program can be received by the communication unit 709 via a wired or wireless transmission medium and installed in the storage unit 708. Alternatively, the program can be pre-installed in the ROM 702 or the storage unit 708.

[0063] A program executed by a computer may be a program that performs processing in the order described in this specification, or it may be a program that performs processing in parallel or when needed (such as when invoked).

[0064] The units or devices described herein are for logical purposes only and do not strictly correspond to physical devices or entities. For example, the function of each unit described herein may be implemented by multiple physical entities, or the function of multiple units described herein may be implemented by a single physical entity. Furthermore, the features, components, elements, steps, etc., described in one embodiment are not limited to that embodiment, but can also be applied to other embodiments, such as replacing specific features, components, elements, steps, etc., in other embodiments, or in combination with them.

[0065] The scope of this invention is not limited to the specific embodiments described herein. Those skilled in the art will understand that various modifications or variations can be made to the embodiments described herein, depending on design requirements and other factors, without departing from the principles and spirit of the invention. The scope of this invention is defined by the appended claims and their equivalents.

[0066] Postscript:

[0067] (1). A training method for a domain-adaptive neural network, wherein the neural network includes a segmentation unit and a reconstruction unit connected in series, the segmentation unit being configured to perform image segmentation for a source domain image and a target domain image and including a first encoder and a first decoder, the reconstruction unit being configured to perform image reconstruction for the target domain image and including a second encoder and a second decoder, the method comprising:

[0068] The first encoder extracts a first feature based on the source domain image or the target domain image;

[0069] The first decoder generates a source domain segmentation prediction image or a target domain segmentation prediction image based on the first feature;

[0070] The second encoder extracts the second feature based on the target domain image and the target domain segmentation prediction image;

[0071] The second decoder generates a target domain reconstructed image based on the first feature and the second feature;

[0072] The segmentation unit and the reconstruction unit are trained based on the segmentation loss function and the reconstruction loss function, respectively.

[0073] (2). According to the method in (1), the first feature is a domain-invariant feature corresponding to the common characteristics of the source domain image and the target domain image, and the second feature is a domain-specific feature corresponding to the unique characteristics of the target domain image.

[0074] (3). The method described in (2) further includes:

[0075] The domain-invariant features are extracted from the target domain image to be processed using the first encoder of the trained segmentation unit;

[0076] Image segmentation is performed on the target domain image to be processed using the first decoder of the trained segmentation unit based on the domain-invariant features.

[0077] (4). According to the method of (1), wherein the first encoder has a plurality of hidden layers, and hidden features are extracted at at least some of the plurality of hidden layers.

[0078] Wherein, the first decoder generates the source domain segmentation prediction image or the target domain segmentation prediction image based on the first feature and the hidden feature; and

[0079] The second decoder generates the target domain reconstructed image based on the first feature and the hidden feature from the first encoder and the second feature from the second encoder.

[0080] (5). According to the method described in (1), wherein the segmentation loss function is the cross-entropy loss function and the reconstruction loss function is the absolute value loss function.

[0081] (6). The method according to (1) further includes:

[0082] The source domain image is preprocessed, and the preprocessed source domain image is input to the first encoder; and

[0083] The target domain image is preprocessed, and the preprocessed target domain image is input to the first encoder and the second encoder.

[0084] (7). The method according to (6), wherein the preprocessing includes at least one of the following processes: converting to grayscale image, normalization, histogram equalization, gamma correction, rotation, flipping, color dithering, and resizing.

[0085] In the case where the source domain image is a high-resolution image and the target domain image is a low-resolution image, the preprocessing further includes randomly adding Gaussian noise.

[0086] (8). According to the method described in (1), wherein: both the segmentation unit and the reconstruction unit have a U-Net network structure.

[0087] (9) An apparatus for training a neural network for domain adaptation, wherein the neural network includes a segmentation unit and a reconstruction unit connected in series, the segmentation unit being configured to perform image segmentation for a source domain image and a target domain image and including a first encoder and a first decoder, and the reconstruction unit being configured to perform image reconstruction for the target domain image and including a second encoder and a second decoder.

[0088] The device includes:

[0089] A memory that stores computer programs; and

[0090] A processor, the processor being configured to execute the computer program

[0091] The first encoder extracts a first feature based on the source domain image or the target domain image;

[0092] The first decoder generates a source domain segmentation prediction image or a target domain segmentation prediction image based on the first feature;

[0093] The second encoder extracts a second feature based on the target domain image and the target domain segmentation prediction image;

[0094] The second decoder generates a target domain reconstructed image based on the first feature and the second feature;

[0095] The segmentation unit and the reconstruction unit are trained based on the segmentation loss function and the reconstruction loss function, respectively.

[0096] (10). A computer-implemented execution domain adaptive neural network, comprising:

[0097] A segmentation unit, configured to perform image segmentation on a source domain image and a target domain image, includes:

[0098] A first encoder is configured to extract a first feature based on the source domain image or the target domain image;

[0099] A first decoder is configured to generate a source domain segmentation prediction based on the first feature.

[0100] Image or target domain segmentation prediction image;

[0101] A reconstruction unit connected in series with the segmentation unit is configured to perform image reconstruction for the target domain image and includes:

[0102] A second encoder is configured to extract a second feature based on the target domain image and the target domain segmentation prediction image;

[0103] The second decoder is configured to generate a target domain reconstructed image based on the first feature and the second feature.

[0104] Specifically, the segmentation unit and the reconstruction unit are trained based on the segmentation loss function and the reconstruction loss function, respectively, and the trained segmentation unit is used to perform image segmentation on the target domain image to be processed.

[0105] (11). A storage medium storing a computer program, which, when executed by a computer, causes the computer to perform a training method for a domain-adaptive neural network according to any one of (1)-(8).

Claims

1. A training method for a domain-adaptive neural network, wherein, The neural network includes a segmentation unit and a reconstruction unit connected in series. The segmentation unit is configured to perform image segmentation on a source domain image and a target domain image and includes a first encoder and a first decoder. The reconstruction unit is configured to perform image reconstruction on the target domain image and includes a second encoder and a second decoder. The method includes: The first encoder extracts a first feature based on the source domain image or the target domain image, and extracts hidden features at at least some of the hidden layers included in the first encoder. The first decoder generates a source domain segmentation prediction image or a target domain segmentation prediction image based on the first feature and the hidden feature; The second encoder extracts the second feature based on the target domain image and the target domain segmentation prediction image; The second decoder generates a target domain reconstructed image based on the first feature and the hidden feature from the first encoder and the second feature from the second encoder. The segmentation unit and the reconstruction unit are trained based on the segmentation loss function and the reconstruction loss function, respectively.

2. The method according to claim 1, wherein, The first feature is a domain-invariant feature corresponding to the common characteristics of the source domain image and the target domain image, and the second feature is a domain-specific feature corresponding to the unique characteristics of the target domain image.

3. The method according to claim 2, further comprising: The domain-invariant features are extracted from the target domain image to be processed using the first encoder of the trained segmentation unit; Image segmentation is performed on the target domain image to be processed using the first decoder of the trained segmentation unit based on the domain-invariant features.

4. The method according to claim 1, wherein, The segmentation loss function is the cross-entropy loss function, and the reconstruction loss function is the absolute value loss function.

5. The method according to claim 1, further comprising: The source domain image is preprocessed, and the preprocessed source domain image is input to the first encoder; as well as The target domain image is preprocessed, and the preprocessed target domain image is input to the first encoder and the second encoder.

6. The method according to claim 5, wherein, The preprocessing includes at least one of the following: conversion to grayscale image, normalization, histogram equalization, gamma correction, rotation, flipping, color dithering, and resizing. In the case where the source domain image is a high-resolution image and the target domain image is a low-resolution image, the preprocessing further includes randomly adding Gaussian noise.

7. The method according to claim 1, wherein: Both the segmentation unit and the reconstruction unit have a U-Net network structure.

8. An apparatus for training a neural network for domain adaptation, wherein, The neural network includes a segmentation unit and a reconstruction unit connected in series. The segmentation unit is configured to perform image segmentation on a source domain image and a target domain image and includes a first encoder and a first decoder. The reconstruction unit is configured to perform image reconstruction on the target domain image and includes a second encoder and a second decoder. The device includes: A memory that stores computer programs; and A processor, the processor being configured to execute the computer program The first encoder is made to extract a first feature based on the source domain image or the target domain image, and hidden features are extracted at at least some of the hidden layers included in the first encoder. The first decoder generates a source domain segmentation prediction image or a target domain segmentation prediction image based on the first feature and the hidden feature. The second encoder extracts a second feature based on the target domain image and the target domain segmentation prediction image; The second decoder generates a target domain reconstructed image based on the first feature and the hidden feature from the first encoder and the second feature from the second encoder. The segmentation unit and the reconstruction unit are trained based on the segmentation loss function and the reconstruction loss function, respectively.

9. A storage medium storing a computer program, which, when executed by a computer, causes the computer to perform a training method for a domain-adaptive neural network according to any one of claims 1-7.

Images