A multi-domain medical image segmentation method based on domain adaptation

Abstract

The present invention discloses a multi-domain medical image segmentation method based on domain adaptation. The method comprises: training a variational autoencoder with a set loss function as the target to extract latent space codes of different domains and style vectors of the corresponding domains, wherein the variational autoencoder comprises an encoder and a decoder; for the image to be processed, using the variational autoencoder to infer domain information and extract domain style vectors, subtracting the style vector of the corresponding domain from the latent space code of the image to obtain a de-stylized latent space code; inputting the de-stylized latent space code into the decoder to reconstruct an image with a unified style; and inputting the image with a unified style into a trained segmentation network to obtain a segmentation result. The present invention eliminates the domain gap between multi-domain data, increases the amount of data for training the segmentation network, and improves image segmentation accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of medical image processing, and more particularly to a multi-domain medical image segmentation method based on domain adaptation. Background Art

[0002] Accurate segmentation of medical images is of great clinical significance. Volume calculation and shape assessment of segmented structures can assist clinicians in diagnosing the health of the affected area or the type of disease. Analyzing dose distribution within the target region is essential for radiotherapy planning. Furthermore, calculating features such as grayscale distribution within the target region can also aid in therapeutic efficacy analysis and prognosis. Currently, image segmentation in clinical practice is typically performed manually, a time-consuming and labor-intensive process.

[0003] In recent years, the development of artificial intelligence technology has brought new opportunities for medical image segmentation, requiring the collection of large amounts of medical image data to train machine learning models. However, due to privacy concerns, medical image data is difficult to collect. For a specific area, it is relatively easy to collect small amounts of data from multiple hospitals. However, due to the varying instrumentation and image acquisition standards of each hospital, even data from the same modality can exhibit significant discrepancies between these multiple sources. Furthermore, because doctors are accustomed to simultaneously observing and comparing images from multiple modalities and tend to outline target areas in the modality with the clearest view, even data collected within the same hospital can exhibit significant differences in labeled data. These factors contribute to the multi-domain nature of medical images, with only a small amount of data available within each domain. This data distribution poses significant challenges to the training of a single image segmentation network.

[0004] In the existing technology, methods for segmenting target regions using multi-domain medical images are mainly divided into three categories: transfer learning, style transfer, and mapping features to a common space. Transfer learning involves training a pre-trained segmentation model on a domain with a large amount of data. This model is then fine-tuned using a small amount of labeled data from other domains to obtain segmentation models suitable for data from each domain. The disadvantage of this approach is that it requires finding a domain with a large amount of data to pre-train the model. However, the sample size of medical images collected from each domain is generally small, resulting in poor performance of the pre-trained model and affecting the performance of the transferred model.

[0005] Style transfer is achieved by using Generative Adversarial Networks (GANs) to transfer the style of medical images. Specifically, a domain with a large amount of data needs to be used as the source domain, and a segmentation network is trained on the source domain. Then, the style of images in other domains is converted to the style of the source domain through GAN, and finally the segmentation network of the source domain is used for segmentation. This method also requires a large number of medical image samples. This method first selects a source domain, and all other domains are used as target domains. In order to convert the style of the target domain image to the source domain style, a GAN needs to be trained between each target domain and the source domain. The number of parameters increases linearly with the number of domains. Especially when the number of domains is large, more resources are required.

[0006] In addition, mapping features to a common space is achieved by adding adversarial losses or distribution constraints to features extracted from different domains to narrow the distance between feature spaces, thereby obtaining a common feature space, and then uniformly decoding the features in this space to obtain the segmentation results.

[0007] The current solution for mapping features to a common space usually uses VAE (Variational Autoencoder) to extract features. In order to map images in each domain to the same feature space, KL divergence is often used to constrain the distribution of the VAE's encoding space. For example, the distribution of the encoding space of all domains is restricted to a standard Gaussian distribution. When all domains share a VAE, the mean absolute error loss in the VAE enables the features in the common space to be decoded and return images with domain style, which indicates that the feature distribution of each domain deviates from the standard Gaussian distribution, and the deviation directions of each domain differ greatly, so there is still a domain gap when performing image segmentation. And when a VAE is trained for each domain separately, it will cause a huge resource occupation.

[0008] In summary, accurate segmentation of medical images can help doctors more effectively diagnose and treat related diseases. However, current medical images often exist in a dispersed, multi-source or multi-modal manner, with only a small amount of data available in each domain. Furthermore, simply mixing images from multiple domains together to train image segmentation models, while aiming to train a model applicable to all domains, complicates the model's function mappings. Consequently, underfitting or overfitting to specific domains is highly likely to occur during training. Summary of the Invention

[0009] The purpose of the present invention is to overcome the above-mentioned defects of the prior art and provide a multi-domain medical image segmentation method based on domain adaptation, which comprises the following steps:

[0010] Step S1: training a variational autoencoder with a set loss function as a target to extract latent space codes of different domains and style vectors of corresponding domains, wherein the variational autoencoder includes an encoder and a decoder;

[0011] Step S2: For the image to be processed, use the variational autoencoder to infer domain information and extract the domain style vector, subtract the style vector of the corresponding domain from the latent space code of the image to obtain a de-stylized latent space code;

[0012] Step S3: inputting the de-stylized latent space code into the decoder to reconstruct an image with a unified style;

[0013] Step S4: inputting the image of unified style into the trained segmentation network to obtain a segmentation result.

[0014] Compared with existing techniques, the present invention has the advantage of eliminating domain gaps between multi-domain data by de-stylizing the latent space encoding, resulting in images with a unified style for training the segmentation network. Compared with direct training in a single domain, this approach increases the amount of data required to train the segmentation network, thereby improving segmentation accuracy. Furthermore, the present invention theoretically has no limit on the number of domains, and increasing the number of domains does not significantly increase the number of network parameters.

[0015] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0017] Figure 1 is a schematic diagram of a multi-domain medical image segmentation process based on domain adaptation according to an embodiment of the present invention;

[0018] Figure 2 4 is a flowchart of a multi-domain medical image segmentation method based on domain adaptation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that unless otherwise specifically stated, the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention.

[0020] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0021] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0022] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0023] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0024] The present invention aims to extract latent space coding features of different domains by training a shared VAE to reduce resource usage. Then, by de-stylizing the latent space coding, the domain gap between multi-domain data is eliminated. And when training the VAE, the latent space coding features of images in different domains are guided to non-overlapping distributions, thereby avoiding the directional bias problem that occurs when all domains are guided to the standard Gaussian distribution. Furthermore, after the latent space coding is de-stylized, all domains will be mapped to a common feature space, and then images of a unified style will be decoded to train subsequent segmentation networks. In this way, using images from all domains to train the segmentation network is equivalent to enriching the amount of training data.

[0025] See also Figure 1 As shown in the figure, for the sake of clarity, the function modules are used to illustrate the whole process, which includes VAE module, de-stylization module and segmentation module. Assume that there are N domains, where M i represents the image of the i-th domain, It represents the latent space features extracted from the i-th domain image and the reconstructed image of the j-th domain obtained after transform decoding. represents the image with unified style decoded after decentralization of the feature vector of the i-th domain, and Y i Represent the segmentation results and gold standards of the i-th domain image respectively.

[0026] Specifically, combined Figure 1 and Figure 2 As shown, the provided multi-domain medical image segmentation method based on domain adaptation includes the following steps.

[0027] Step S210: training a variational autoencoder to extract latent space coding features for multi-domain images.

[0028] A variational autoencoder consists of an encoder and a decoder. The encoder takes a multi-domain image as input and extracts latent space encoding features. The decoder reconstructs the image based on these latent space encoding features. The variational autoencoder encodes the image and produces a feature representation vector that contains information about the original image (i.e., it can be decoded to obtain the original image).

[0029] In one embodiment, the loss function for training the variational autoencoder is expressed as:

[0030]

[0031] Among them D KL represents the loss of KL divergence, which makes the two distributions as close as possible. For example, the calculation of this term is expressed as:

[0032]

[0033] in, The style vector of the domain is a vector of length N, where the value of the i-th component is 5 and the others are 0. Represents an N×N unit matrix, which means that we want the latent space encoding of samples in different domains to be close to different Gaussian distributions. i represents the encoding of the covariance matrix of the image latent space of the i-th domain, Consistent with the covariance matrix of the standard Gaussian distribution, x represents the sample obtained by sampling the Gaussian distribution N(μ,Σ), μ i Represents the encoding of the mean of the image latent space of the i-th domain. Indicates the absolute error between the reconstructed image and the original image, calculated pixel by pixel. The mean can be used. This loss ensures that the latent space code can be decoded back to the original image, that is, to ensure that the extracted latent space code retains the structure and other information of the original image. In addition to decoding back to the same domain image, the latent space code can also be transformed. After decoding, it returns the image of the jth domain, so that images of N-1 other domains can be obtained, and then the images of the corresponding domains can be extracted from the real data to calculate the adversarial loss between the two Make the reconstructed image more realistic.

[0034] In one embodiment, and Set to:

[0035]

[0036]

[0037] Where W represents the width of the image, H represents the height of the image, Represents image M i The value at position (x,y), Representing an image The value at position (x, y), D represents the discriminator, represents the data distribution of the i-th image domain, represents the data distribution of images in the jth domain generated from images in the i-th domain.

[0038] In this step, the essence of the VAE is to use the encoder to learn the image style (latent space encoding) while preserving the structural information. Then, the structure and selected style are decoded to reconstruct an image with consistent structure but different styles. During training, the style encoding vector is preferably aligned with the set one-hot encoding, making it easier to control and eliminate the image style during later segmentation.

[0039] Step S220 , extracting the image style of each domain, subtracting the style vector of the domain from the latent space code of the image, obtaining a de-stylized latent space code, and reconstructing an image with a unified style.

[0040] After training the VAE module, the style of the image can be extracted and controlled. In order to unify the image styles of different domains, we first need to clarify the domain of the image, and then encode the latent space of the image μ i Subtract the style vector of the domain The latent space coding of images in all domains is made close to (0,0,…,0), which is the de-stylized latent space coding. The de-stylized latent space coding vector is input into the decoder to reconstruct an image with a unified style. For subsequent unified segmentation network training.

[0041] It should be noted that the domain information of each image is known only during the training phase, but is unknown during the testing phase. During testing (or when actually segmenting the image to be processed), the domain-unknown image is obtained. At this time, it is necessary to input it into the VAE encoder, infer the domain information based on the resulting latent space encoding (as shown in Equation 5), and finally perform decentralization based on the determined domain style vector.

[0042] domain * =argmax(softmax(μ i )) (5)

[0043] Step S230: Using images of uniform style to train a segmentation network.

[0044] In one embodiment, the image segmentation network uses the U-Net framework, and supervises the network parameter update through the weighted loss of Dice and cross entropy. The total segmentation loss L SegExpressed as:

[0045] L Seg =L Dice +λ3L CE (6)

[0046]

[0047]

[0048] Among them, Y represents the gold standard of the segmentation task, Represents the prediction result of the segmentation network, N c Indicates the number of categories to be segmented, x and y represent spatial coordinates, It represents the probability that the (x, y) position predicted by the segmentation network belongs to the cth class, and λ3 is the set weight parameter.

[0049] In this step, images from all domains can be used to train the segmentation network, which makes full use of the data and increases the amount of training samples for the segmentation network.

[0050] It should be noted that those skilled in the art may make appropriate changes or modifications to the above embodiments without departing from the spirit and scope of the present invention. For example, other loss functions may be used to train the variational autoencoder or image segmentation network, such as likelihood loss, exponential loss, etc.

[0051] In summary, the present invention designs a segmentation method for multi-domain medical images based on domain adaptation, maps samples from all domains to a common feature space, improves the utilization rate of labeled multi-domain medical images, and makes the trained unified segmentation network more robust. And a method based on VAE encoding domain style is proposed, which brings the style encoding vector closer to the set encoding value, making the image style in the later segmentation easier to control, which is beneficial to the later style elimination. In addition, a method for style removal is proposed, which subtracts the latent space vector obtained by VAE encoding from the defined domain style vector to obtain the style-removed latent space vector. Compared with directly approaching the distribution of the latent space feature vector and the standard Gaussian distribution, the domain deviation obtained by this method of the present invention will not show strong directional differences, thereby providing a more effective solution for mapping images from different domains to a common feature space.

[0052] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0053] Computer-readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device.Computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device or any suitable combination thereof.More specific examples (non-exhaustive list) of computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove having instructions stored thereon, and any suitable combination thereof.Computer-readable storage medium used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0054] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in the computer-readable storage medium in each computing / processing device.

[0055] The computer program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, Python, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The computer readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), is personalized by utilizing the state information of the computer readable program instructions, and the electronic circuit can execute the computer readable program instructions, thereby realizing various aspects of the present invention.

[0056] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0057] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device is generated that implements the functions / actions specified in one or more blocks in the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, where these instructions cause the computer, programmable data processing device, and / or other device to operate in a specific manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing various aspects of the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0058] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more blocks in the flowchart and / or block diagram.

[0059] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0060] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A multi-domain medical image segmentation method based on domain adaptation, comprising the following steps: Step S1: training a variational autoencoder with a set loss function as a target to extract latent space codes of different domains and style vectors of corresponding domains, wherein the variational autoencoder includes an encoder and a decoder; Step S2: For the image to be processed, use the variational autoencoder to infer domain information and extract the domain style vector, subtract the style vector of the corresponding domain from the latent space code of the image to obtain a de-stylized latent space code; Step S3: inputting the de-stylized latent space code into the decoder to reconstruct an image with a unified style; Step S4: inputting the image of unified style into the trained segmentation network to obtain a segmentation result.

2. The method according to claim 1, wherein The loss function for training the variational autoencoder is set to: Among them D KL The loss is represented by KL divergence, which is expressed as: in represents a vector of length N, represents the identity matrix of dimension N×N, ∑ i Represents the encoding of the covariance matrix of the image latent space of the i-th domain, μ i represents the encoding of the mean of the image latent space of the i-th domain, Indicates that the absolute error between the reconstructed image and the original image is calculated pixel by pixel, and the latent space encoding is transformed After decoding, it returns the image of the jth domain. represents the adversarial loss between the image extracted from the corresponding domain in the real data and the reconstructed image, λ1 and λ2 represent the weights of the corresponding terms, and x represents the sample obtained by sampling the Gaussian distribution N(μ, ∑).

3. The method according to claim 2, characterized in that and Set to: Where W represents the width of the image, H represents the height of the image, Represents image M i The value at position (x, y), Representing an image The value at position (x, y), D represents the discriminator, represents the data distribution of the i-th image domain, represents the data distribution of images in the jth domain generated from images in the i-th domain.

4. The method according to claim 1, wherein In step S2, domain information is inferred according to the following formula: domain*=argmax(softmax(μ i )) Among them, μ i Represents the encoding of the mean of the image latent space of the i-th domain.

5. The method according to claim 1, characterized in that The loss function for training the segmentation network is set to: L Seg =L Dice +λ3L CE Among them, L Seg represents the total segmentation loss, L Dice represents the Dice loss, L CE represents the cross entropy loss, and λ3 represents the weight coefficient.

6. The method according to claim 5, characterized in that The Dice loss and cross entropy loss are set as: Among them, Y represents the gold standard of the segmentation task, Represents the prediction result of the segmentation network, N c Indicates the number of segmented categories, x and y represent the horizontal and vertical coordinates of the space respectively, Represents the probability that the (x, y) position predicted by the segmentation network belongs to the cth class.

7. The method according to claim 1, characterized in that During the training of the variational autoencoder, the style encoding vector is brought closer to the set one-hot encoding.

8. The method according to claim 1, characterized in that The segmentation network adopts the U-Net framework and is trained using images from multiple domains.

9. A computer-readable storage medium having a computer program stored thereon, wherein: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer device comprising a memory and a processor, wherein a computer program capable of being run on the processor is stored in the memory, characterized in that: When the processor executes the program, the steps of the method according to any one of claims 1 to 8 are implemented.

Images