Active learning cross-domain cardiac image segmentation method based on few pixel labeling

Abstract

The present application relates to a kind of active learning cross-domain cardiac image segmentation method based on few pixel labeling, belong to image processing technical field.The method includes the following steps: obtaining the nuclear magnetic resonance sequence of heart or obtaining the three-dimensional image of heart's electronic computer tomography, after the slice processing of the three-dimensional image obtained, carry out rotation alignment, image preprocessing operation of cutting and scaling;The parameter training of designed dual-domain active learning segmentation network model is carried out, and optimization is carried out using stochastic gradient descent method, and the trained segmentation model is obtained.The method of the present application is superior to the forefront of unsupervised domain adaptive algorithm.When only in target domain image annotation total pixel 0.1%, the method exceeds the performance of fully supervised model.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology and relates to an active learning cross-domain cardiac image segmentation method based on minimal pixel annotation. Background Technology

[0002] Currently widely used medical imaging technologies, such as magnetic resonance imaging (MRI), computed tomography (CT), and ultrasound (US), can provide qualitative and quantitative non-invasive assessments of cardiac anatomy and function. Segmentation is often the first step in cardiac imaging analysis and plays a crucial role in the diagnosis and prognosis of cardiovascular diseases. However, in the field of medical imaging, manual annotation of multimodal images is extremely laborious and challenging. Building a segmentation model from the source domain and transferring it to the target domain may be a good practical solution. However, variations in the distribution between different image modalities can lead to significant performance degradation. Especially in the field of cross-modal cardiac segmentation, state-of-the-art unsupervised domain adaptation (UDA) methods have not yet met the requirements of clinical practice. One key challenge is accurately segmenting cardiac structures in regional gaps caused by variations in image quality, resolution, and acquisition protocols. To meet clinical segmentation needs, obtaining annotations from the target domain is worth considering if performance can be significantly improved by providing the necessary transferable knowledge. In the field of autonomous driving, researchers have developed several active domain adaptation methods in recent years for image detection, classification, and image segmentation. These methods have greatly reduced the amount of annotation work and achieved good practical results, providing inspiration for work in the field of medical imaging. Summary of the Invention

[0003] In view of this, the purpose of this invention is to provide an active learning method for cross-domain cardiac image segmentation based on minimal pixel annotation.

[0004] To achieve the above objectives, the present invention provides the following technical solution:

[0005] An active learning-based cross-domain cardiac image segmentation method with minimal pixel annotations includes the following steps:

[0006] Obtain the MRI sequence of the heart or the 3D computed tomography image of the heart. After slicing the obtained 3D image, perform image preprocessing operations such as rotation alignment, cropping and scaling.

[0007] The parameters of the designed dual-domain active learning segmentation network model are trained and optimized using stochastic gradient descent to obtain the trained segmentation model.

[0008] The dual-domain active learning segmentation network model consists of three parts: an encoder, a decoder, and a segmenter. It is trained using data from the source and target domains, respectively. The specific training steps are as follows:

[0009] Model training for source domain data: The source domain image is passed through a convolutional neural network layer of the encoder; the latent features Z extracted by the encoder are... S The input is fed into the deconvolutional neural network layer of the decoder; simultaneously, the feature Z is... S Input is sent to the segmenter to complete the target segmentation;

[0010] Model training for target domain data: The target domain image is passed through a convolutional neural network layer of an encoder; the latent features Z extracted by the encoder are... T The input is fed into the deconvolutional neural network layer of the decoder; simultaneously, the feature Z is... T Input is sent to the segmenter to complete the target segmentation;

[0011] The preprocessed image is input into the trained encoder for feature extraction, and then the extracted features are input into the trained segmenter to complete the target segmentation of the left ventricular cavity, left ventricular myocardium and right ventricular cavity in the image.

[0012] Furthermore, in the acquisition of cardiac MRI sequence images, in the short-axis MRI cine sequence, the operator will locate the intersection of the ventricle and atrium;

[0013] Based on the location of the ventricular-atrial junction, the image is rotated and aligned so that the line connecting the heart junctions is perpendicular to the horizontal axis relative to the image coordinate system.

[0014] The rotated and aligned heart area is cropped and scaled to a 192*192 image size.

[0015] During offline training, the cropped image sequence is input into the segmentation network to be trained in five sequences at a time. First, for the source domain data, the source domain image is passed through the convolutional neural network layer of the encoder.

[0016] The latent features Z extracted by the encoder S The input is fed into the deconvolutional neural network layer of the decoder; simultaneously, the feature Z is... S Input is sent to the segmenter to complete the target segmentation;

[0017] Image reconstructed by the decoder With the input image x S To compare them, the Euclidean distance between them is calculated, and the training loss function is designed as follows:

[0018]

[0019] Where ||·||2 is the Euclidean distance calculation formula;

[0020] Image P after segmentation by the segmenter S With real labels For comparison, its training loss function is designed as follows:

[0021]

[0022]

[0023] in, For cross-entropy loss, Let KL divergence loss function be used.

[0024] Select the minimum number of pixels to label as active tags. Applying a risk region selection strategy, the training loss function is designed as follows:

[0025]

[0026] The inherent shape of the heart structure is incorporated by ensuring the predictive consistency between pixels in the source image and their neighboring pixels. The consistency regularization term is expressed as follows:

[0027]

[0028] Where |·| represents the number of pixels in the set, It is the average prediction of all pixels in the neighborhood;

[0029] The objective function for global model optimization in the source domain is written as:

[0030]

[0031] in, and They are used for balancing respectively loss, loss and The harmonic parameters of the loss;

[0032] For target domain data, if there are no real labels to constrain the reconstruction process, then pseudo-labels are used. Replace and utilize active labels. To supervise the segmentation of the target domain, the objective function for overall model optimization in the target domain is written as:

[0033]

[0034]

[0035]

[0036]

[0037] Among them, P T For segmentation prediction of target input data, and It is a harmonic parameter used to balance losses;

[0038] Extracted source domain features θ S and target domain features θ T The constraints are of the same distribution, and the distance calculation between them is designed as follows:

[0039]

[0040] Where β is a constant coefficient, This represents a distance function used to measure the parameter θ. S and θ T The difference between probability distributions, For latent variable z i Probability estimates obtained from the source domain encoder For latent variable z i Probability estimates obtained from the target domain encoder;

[0041] For network parameter training, during the model training process, active labels are obtained by using the category diversity maximization method and the two-domain uncertainty graph selection method, and regions with the risk of missegmentation in the active learning environment are identified.

[0042] Methods for maximizing category diversity include: given an image I, the predicted image after segmentation is denoted as P, and the pseudo-labels are... We obtain: Divide a region R(i,j) of image I into c subsets:

[0043]

[0044] Where (u,v) represents a pixel in R(i,j);

[0045] To statistically analyze the category information within a region, reflecting the diversity of categories; given R(i,j), its category diversity. The calculation is as follows:

[0046]

[0047] Where |·| represents the number of pixels in the set, and C is the number of predicted categories;

[0048] The methods for selecting a two-domain uncertainty graph include: given an input image X, inputting it into an encoder to obtain the posterior probability; inputting a random sample z into a segmenter to obtain a predicted value P0; repeating the sampling process M times to generate a series of predictions {P1, P2, ..., P}. M Finally, the uncertainty graph U is estimated using the following formula:

[0049]

[0050] in, P represents the prediction entropy. k (i,j,c) This represents the predicted value obtained by the segmenter for the (i, j)th pixel of the c-th category after repeating k times; P0 (i,j,c) This indicates that the predicted value of the (i, j)th pixel of the cth category is obtained by the segmenter after being repeated 0 times.

[0051] Adjusting the model's focus, the final acquisition function is:

[0052]

[0053] Where ⊙ represents element-matrix multiplication. U represents category diversity information, and U represents uncertainty graph information.

[0054] The beneficial effects of this invention are as follows: it develops a segmentation network that processes cross-modal cardiac image segmentation by annotating the minimum pixel in the target domain, thereby reducing the annotation workload of clinicians and improving the accuracy of segmentation.

[0055] The method of this invention outperforms state-of-the-art unsupervised domain adaptation algorithms. When only 0.1% of the total pixels in the target domain image are annotated, this method surpasses the performance of a fully supervised model.

[0056] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0057] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0058] Figure 1 This is a flowchart of the heart segmentation system.

[0059] Figure 2 It is a preprocessing workflow for multiple modal images of the heart.

[0060] Figure 3 It is a training framework for dual-domain active learning segmentation network models.

[0061] Figure 4 and Figure 5 This is a demonstration of the results of the embodiments of the present invention and the results of comparative experiments. Detailed Implementation

[0062] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0063] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures, and should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0064] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0065] To address the existing problems in cardiac image segmentation techniques, this paper proposes a segmentation network that handles cross-modal cardiac image segmentation by annotating the minimum pixel in the target domain. The invention first decomposes the cardiac segmentation task and designs a dual-domain active learning strategy framework, proposing the segmented network CUADA. This algorithm employs a tilted variational autoencoder (tVAE) and a distance-based regularization term to extract and align domain-invariant features of the image, reducing the domain gap. First, a tilted variational autoencoder (tVAE) is used in both the source and target domains to extract domain-invariant features of the image. Then, the proposed class diversity maximization method and uncertainty map are used to identify regions at risk of missegmentation for further annotation. To achieve the above-mentioned objectives, this invention provides the following technical solutions:

[0066] The algorithm first designs a dual-domain active learning strategy framework based on the cross-modal cardiac image segmentation task. Specifically, it utilizes a tilted variational autoencoder (tVAE) to extract domain-invariant features from the source and target domain images.

[0067] This invention constructs a distance-based regularized objective loss function and uses stochastic gradient descent to optimize model parameters, ensuring that the extracted features of the source and target domains belong to the same distribution.

[0068] This invention designs an innovative method for maximizing category diversity and a dual-domain uncertainty graph, which selects the minimum number of pixels as active labels to identify regions at risk of missegmentation in an active learning environment.

[0069] After the network model has been trained, in actual application, the system will rotate and align the acquired images and automatically crop the images to a size of 192*192.

[0070] After normalization, multiple sequence images of the heart are input into a trained neural network to automatically segment heart-related targets.

[0071] In one specific embodiment of the present invention, the above technical solution includes the following steps, such as... Figure 3 As shown.

[0072] Step 1: Obtain MRI sequences or 3D computed tomography images of the heart, and perform image preprocessing such as slicing, rotation alignment, cropping and scaling.

[0073] The second step is to train the parameters of the designed network model and optimize it using stochastic gradient descent.

[0074] Step 3: Input the preprocessed image into the trained dual-domain active learning segmentation network and complete the target segmentation of the left ventricular cavity (LV), left ventricular myocardium (MYO), and right ventricular cavity (RV) in the image.

[0075] The specific implementation process is as follows: First, the system will obtain the heart's cinematic sequence from an MRI or X-ray computed tomography scan;

[0076] Furthermore, in the short-axis MRI cine sequence, the operator will locate the intersection of the ventricle and atrium;

[0077] Furthermore, based on the position of the ventricular-atrial junction, the image is rotated and aligned so that the line connecting the cardiac junctions is perpendicular to the horizontal axis relative to the image coordinate system.

[0078] Furthermore, the system crops the rotated and aligned heart region and scales it to an image size of 192*192.

[0079] Furthermore, during offline training, the cropped image sequences (5 sequences at a time) are input into the segmentation network to be trained. First, for the source domain data, the images of the source domain are passed through the convolutional neural network layer of the encoder.

[0080] Furthermore, the latent features Z extracted by the encoder S The input is fed into the deconvolutional neural network layer of the decoder; simultaneously, the feature Z is... S Input is sent to the segmenter to complete the target segmentation;

[0081] Furthermore, the image reconstructed by the decoder With the input image x S For comparison, its training loss function is designed as follows:

[0082]

[0083] Furthermore, the image P segmented by the segmenter S With real labels For comparison, its training loss function is designed as follows:

[0084]

[0085]

[0086] in, For cross-entropy loss, Let KL divergence loss function be used.

[0087] Furthermore, to achieve a good training level, a risk region selection strategy is applied, selecting the minimum number of pixels to be labeled as active labels. The training loss function is designed as follows:

[0088]

[0089] Furthermore, the inherent shape of the heart structure is incorporated by ensuring predictive consistency between pixels in the source image and their neighboring pixels. This consistency regularization term is expressed as:

[0090]

[0091] Where |·| represents the number of pixels in the set, It is the average prediction of all pixels in the neighborhood.

[0092] Furthermore, the objective function for global model optimization in the source domain can be written as:

[0093]

[0094] in, and It is a harmonic parameter used to balance losses.

[0095] Furthermore, for target domain data, since there are no real labels to constrain the reconstruction process, pseudo-labels are used. Replace and utilize active labels. To supervise the segmentation of the target domain, the objective function for overall model optimization in the target domain can be written as:

[0096]

[0097]

[0098]

[0099]

[0100] Among them, P T For segmentation prediction of target input data, and It is a harmonic parameter used to balance losses.

[0101] Furthermore, the extracted source domain features and target domain features are constrained to the same distribution, and the distance calculation between them is designed as follows:

[0102]

[0103] Where β is a constant coefficient.

[0104] Furthermore, all calculated function losses will be backpropagated using stochastic descent to optimize the model parameters;

[0105] Furthermore, in practical applications, the system performs image preprocessing (slicing, rotating, shearing, scaling) on ​​the cardiac MRI cine sequence to be analyzed using the previous method.

[0106] Furthermore, each preprocessed cardiac film sequence will be processed by a trained dual-domain active learning segmentation network;

[0107] Furthermore, the network will output an image of the segmented target for each frame, which includes the target segments of the left ventricular cavity (LV), left ventricular myocardium (MYO), and right ventricular cavity (RV).

[0108] Figure 4 and Figure 5 This is a demonstration of the results of the embodiments of the present invention and the results of comparative experiments. Figure 4 Visual comparison of the segmentation effects of different segmentation methods on MM-WHS (CT→MR). LV, RV, and Myo are marked in red, green, and blue, respectively. Each row corresponds to one patient. The left side shows the image and ground truth label, the middle shows the UDA method (distance-based construction method: VarDA, GAN-based construction method: SIFA), and the right side shows the ADA method. This paper proposes the CUADA method. Figure 5 Visual comparison of the segmentation effects of different segmentation methods on MM-WHS (MRI→CT). LV, RV, and Myo are marked in red, green, and blue, respectively. Each row corresponds to one patient. The left side shows the image and ground truth label, the middle shows the UDA method (distance-based construction method: VarDA, GAN-based construction method: SIFA), and the right side shows the ADA method. This paper proposes the CUADA method.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. An active learning-based cross-domain cardiac image segmentation method based on minimal pixel annotation, characterized by: The method includes the following steps: Obtain an MRI sequence of the heart or a 3D computed tomography (CT) image of the heart. After slicing the obtained 3D image, perform image preprocessing operations such as rotation alignment, cropping and scaling on the 3D image. The parameters of the designed dual-domain active learning segmentation network model are trained and optimized using stochastic gradient descent to obtain the trained segmentation model. The dual-domain active learning segmentation network model comprises three parts: an encoder, a decoder, and a segmenter, which are trained using source domain data and target domain data, respectively. It employs a tilted variational autoencoder (tVAE) and a distance-based regularization term to extract and align domain-invariant features of the image, thereby reducing the domain gap. When training a model on source domain data, the source domain image is input into the convolutional neural network layer of the encoder, the latent features extracted by the encoder are input into the deconvolutional neural network layer of the decoder, and the latent features are input into the segmenter to complete the target segmentation. The image reconstructed by the decoder is compared with the input image, and the Euclidean distance between them is calculated. The training loss function is designed as follows: in, This represents the source domain image reconstruction loss. Represents the source domain input image, Represents the source domain reconstructed image , The Euclidean distance calculation formula; The segmented images are compared with the ground truth labels, and the training loss function is designed as follows: in, Represents the source domain segmentation loss. Represents the cross-entropy loss function. Indicates the true label of the source domain. This indicates the source domain prediction result; The KL divergence loss function is expressed as: in, Represents the source domain KL Divergence loss, This represents the mean of the distribution of latent variables in the source domain. This represents the reference mean of the source domain; The minimum number of pixels is selected as the active label, and a risk region selection strategy is applied. The training loss function is designed as follows: in, This represents the source domain active label loss function. Represents the cross-entropy loss function. Indicates the source domain active label; The inherent shape of the heart structure is incorporated by ensuring predictive consistency between pixels in the source image and their neighboring pixels. This consistency regularization term is expressed as follows: in, This represents the source domain consistency regularization loss. Indicates the first k A set of pixels within a local region. This represents the number of pixels in the set. i , j () represents the pixel coordinates in the image. It is the average prediction of all pixels in the neighborhood. This represents the average of the prediction results for all pixels in the neighborhood. The objective function for overall model optimization in the source domain is: in, This represents the overall training loss of the source domain model; , , and These are harmonic parameters used to balance the corresponding losses; When training a model for target domain data, the target domain image is input into the convolutional neural network layer of the encoder, the latent features extracted by the encoder are input into the deconvolutional neural network layer of the decoder, and the latent features are input into the segmenter to complete target segmentation. For the target domain data, since there are no true labels to constrain the reconstruction process, pseudo-labels are used instead, and active labels are used to supervise the segmentation of the target domain. The objective function for the overall model optimization of the target domain is: in: in, This represents the overall training loss of the target domain model. Indicates the target domain input image, Represents the image reconstructed from the target domain. Indicates the target domain active label, This represents the segmentation prediction of the target input data. and The harmonic parameters used to balance the loss, Indicates the target domain reconstruction loss; The extracted source domain features and target domain features are constrained to the same distribution, and the distance between them is calculated as follows: in, This represents the loss due to the difference in feature distribution between the source and target domains. It is a constant coefficient. This represents a distance function used to measure parameters. and The difference between probability distributions, This represents the source domain encoder parameters. Indicates the target domain encoder parameters. n Indicates the number of latent variables sampled. Indicates the first i’ One latent variable, For latent variables Probability estimates obtained from the source domain encoder For latent variables Probability estimates obtained from the target domain encoder; For the network parameter training process, active labels are obtained by using the category diversity maximization method and the two-domain uncertainty graph selection method to identify regions with the risk of missegmentation; The preprocessed image is input into the trained encoder for feature extraction, and then the extracted features are input into the trained segmenter to complete the target segmentation of the left ventricular cavity, left ventricular myocardium and right ventricular cavity in the image.

2. The active learning cross-domain cardiac image segmentation method based on minimal pixel annotation as described in claim 1, characterized in that: When acquiring the cardiac magnetic resonance imaging sequence, the intersection of the ventricle and atrium is located in the short-axis cine sequence of the magnetic resonance imaging. Based on the intersection point of the ventricle and atrium, the acquired image is rotated and aligned so that the line connecting the intersection points of the ventricle and atrium is perpendicular to the horizontal axis relative to the image coordinate system. The rotated and aligned heart region is cropped, and the cropped image is scaled to a size of 192×192. During offline training, the cropped image sequences are input into the dual-domain active learning segmentation network model to be trained in five sequences at a time. The method for maximizing category diversity includes: obtaining a predicted image of the input image after passing through a segmentation model, and generating pseudo-labels based on the predicted image; dividing the predicted image into multiple sub-regions, and statistically analyzing the category information within each sub-region to reflect the category diversity within the corresponding region; and obtaining category diversity information according to the category diversity calculation formula. The formula for calculating the category diversity is: Given image The predicted image after the segmentation model is denoted as Pseudo-tags are Get; get the image an area Divided into Subset: in, express One pixel in; To statistically analyze category information within a region, reflecting category diversity; given Its category diversity The calculation is as follows: in, This indicates the number of pixels in the set. To predict the number of categories; The dual-domain uncertainty graph selection method includes: inputting the input image into the encoder to obtain the posterior probability, inputting random samples into the segmenter to obtain the prediction result, and performing repeated sampling. M Multiple prediction results are generated, and an uncertainty graph is calculated based on the prediction results. The uncertainty diagram Calculated using the following formula: Given an input image The random sample is input into the encoder to obtain the posterior probability; The input is fed into the segmenter to obtain the predicted value. Repeat sampling process M Next, to generate a series of predictions Finally, uncertainty diagram Estimate using the following formula: in, Represents the prediction entropy, Indicates repeated use sequence The first category The predicted value of each pixel obtained by the segmenter; Indicates repeated use sequence The first category The predicted value of each pixel obtained by the segmenter; Indicates the first k Sub-sampling prediction; The model's focus area is adjusted based on the category diversity information and the uncertainty graph information, and the active label selection result is obtained through the final acquisition function, which is expressed as follows: in, This indicates that the final output result of the function is obtained. It is element-matrix multiplication. For category diversity information, This is information about the uncertainty graph.

Images