Meibomian gland image semi-supervised segmentation method based on registration in patient

Abstract

According to the meibomian gland image semi-supervised segmentation method based on registration in the patient, sampling is carried out by means of the registration technology in the patient, vivid and diversified meibomian gland images are generated, generation of pseudo label data does not depend on a segmentation network, and the situation that data quality is interfered by the performance of the segmentation network is fundamentally avoided. Compared with an existing method, the method has the advantages that hidden priori knowledge in clinical data is fully mined and applied, significant differences of meibomian gland images among individuals are considered, high-quality pseudo-label data are generated through registration and sampling in patients and are used for segmentation, dependence of a model on manual annotation data is effectively reduced, and segmentation accuracy is improved. The segmentation precision and generalization ability of the model are effectively improved, and a brand new and effective solution is provided for the field of meibomian gland image segmentation.

Description

Technical Field

[0001] The present invention relates to the technical field of infrared meibomian gland image segmentation, in particular to a meibomian gland image semi-supervised segmentation method based on intra-patient registration. Background Art

[0002] Meibomian glands located in the eyelids maintain a healthy ocular surface by secreting lipids. Abnormal meibomian gland function can cause the tear film to evaporate too quickly, which in turn causes eye diseases such as dry eyes. In clinical practice, obtaining information such as the morphology and quantity of meibomian glands is extremely critical for disease diagnosis and treatment. Infrared meibomian gland images can intuitively present their morphology. Automatic segmentation of glands is the basis for quantitative analysis, which can provide doctors with a basis for diagnosis and promote precision diagnosis and treatment. The development of artificial intelligence has promoted the advancement of medical image segmentation technology, and a large number of fully supervised medical image segmentation methods that rely on labeled data have emerged. However, the high cost of accurate labeling data for infrared meibomian gland images limits their clinical application. In recent years, semi-supervised segmentation methods that use a small amount of labeled and a large number of unlabeled samples for training have achieved good results in the application of meibomian gland image segmentation.

[0003] In actual clinical practice, patients will take multiple infrared meibomian gland images during treatment to detect the condition and treatment effect, which makes the clinical data contain the prior knowledge of multiple images of the same patient. Image data from the same patient not only share similar anatomical structures and pathological features, but may also contain specific physiological change patterns. Fully mining and utilizing the rich information and correlations contained in these patient internal data is particularly critical to help the model deeply understand the evolution of the disease and improve the learning and generalization capabilities of the model. However, most of the existing semi-supervised medical image segmentation methods simply divide the data into labeled and unlabeled data. This method destroys the potential correlation between the data to a certain extent, ignores the prior knowledge of the patient to whom the image belongs, and reduces the value of the data.

[0004] As a semi-supervised method, the easy-to-implement pseudo-labeling method generates pseudo-labels to expand the labeled dataset, thereby improving model performance. However, the quality of pseudo-labels generated by this method is highly dependent on the performance of the segmentation model. Once the pseudo-labels generated by the segmentation model contain a lot of noise, lack authenticity and diversity, it will have a negative impact on the improvement of model performance and hinder the effective training of the model.

[0005] In addition, some methods combine the registration task with the segmentation task, and have achieved certain results by creating an atlas and performing registration and alignment semantics to assist segmentation. However, due to the significant individual differences in the morphology of the meibomian glands, it is extremely challenging and impractical to build a universal clinical atlas. Summary of the invention

[0006] In view of this, the purpose of the present invention is to provide a semi-supervised segmentation method for meibomian gland images based on intra-patient registration. In the training segmentation, by introducing the intra-patient registration technology and designing a special sampling generation method, the pixel intensity and spatial distribution characteristics of the unlabeled data image of the same patient are fused into the labeled image, and pseudo-label data that is diverse and realistic and not affected by the performance fluctuation of the segmentation model is generated online, thereby reducing the model's dependence on a large amount of manually labeled data, improving the performance of the segmentation model under limited annotation, and providing an efficient and reliable new solution for infrared meibomian gland image segmentation.

[0007] To achieve the above object, the present invention adopts the following technical solution: a semi-supervised segmentation method of meibomian gland images based on intra-patient registration, comprising the following steps:

[0008] Step S1, data set processing: dividing and preprocessing the infrared meibomian gland image data set, which contains multiple images of different patients and corresponding labels; using the medical image registration library ANTsPy to perform affine registration on the infrared meibomian gland data of the same patient, and obtaining the affine displacement field and the image data for training the registration model;

[0009] Step S2, training a model for registration of infrared meibomian gland images in patients: using the image data after affine registration in step S1 to train a model for registration of infrared meibomian gland images in patients;

[0010] Step S3, registration sampling generation module design: The registration sampling generation module converts a labeled data (x l ,y), two unlabeled data x uk and x uj As input, pseudo-label data is generated by sampling, thereby providing training samples for the training of the segmentation network and enhancing the generalization ability and segmentation accuracy of the model used for the registration of infrared meibomian gland images inside patients;

[0011] Step S4, segmentation model training: The network is trained using labeled data and unlabeled data of the same patient, and the segmentation model is supervised through labeled images. At the same time, pseudo-label supervision is performed on the two pseudo-label data generated online by the data input registration sampling generation module, thereby completing the training of the segmentation network.

[0012] In a preferred embodiment, the step S1 processes the data set; divides the data set according to the patients into a training set, a test set and a validation set, and ensures that the data of each patient only appears in one of the sets; saves the obtained deformed image and the corresponding affine displacement field to be used in steps S3 and S4.

[0013] In a preferred embodiment, the registration model used in step S2 is composed of a registration network of VoxelMorph and a spatial transformation function; the input moving image and fixed image are superimposed in the channel dimension and then input into the registration network to estimate the deformation field; the spatial transformation function performs a deformation operation on the moving image according to the deformation field, and uses the obtained deformation field and the deformed image for model training; the training uses structural consistency loss and normalized cross-correlation loss to constrain the similarity between the deformed image and the fixed image to ensure that the two have high consistency in structure and grayscale distribution; at the same time, the deformation field is constrained by the L2 norm to suppress local exaggerated deformation, ensure the smoothness and rationality of the deformation field, and thus ensure the accuracy and stability of the registration.

[0014] In a preferred embodiment: the registration sampling generation module in step S3 is used online during the training of the segmentation network. The registration sampling generation module utilizes the intrinsic connection of the patient's own data and integrates the pixel intensity characteristics and spatial distribution characteristics of the unlabeled data into the labeled data, so that the generated pseudo-label data is more consistent with the real meibomian gland image characteristics in terms of pixel intensity and spatial distribution. Different from the traditional use of Beta peak distribution, the sampling parameters α and β are obtained with equal probability between the set [0,1] to ensure that the images generated by sampling have more diversity.

[0015] In a preferred embodiment, the sampling generation process of step S3 includes the following steps:

[0016] S31: Obtain the intensity difference characteristics of the gland; l is a fixed image, x uk To move the image, we first perform affine coarse registration using ANTsPy to obtain the affine image A k2l , then together with the fixed image x l Send it to the registration network for precise registration and use the spatial deformation function to obtain the deformed image Warp k2l , calculate the intensity difference feature And perform random sampling to obtain the intensity difference characteristics of the sampling The sampling parameter is α:

[0017]

[0018] Among them, Affine(·,·) means using ANTsPy for affine registration, ST(·,·) means using the spatial transformation function to deform the image according to the deformation field; Reg(·,·) means using the registration network for precise registration to obtain the deformation field;

[0019] S32: Obtain the spatial distribution characteristics of glandular structure; uj is a fixed image, xl For the moving image, ANTsPy affine coarse registration is performed to obtain the affine image A l2j and the corresponding affine displacement field ADF l2j , then together with the fixed image x uj Send it to the registration network for precise registration, get the deformation field, and superimpose the ADF l2j , and finally get the moving image x l Transform to a fixed image x uj Spatial distribution characteristics of glandular structures l2j And random sampling is performed to obtain the spatial distribution characteristics of the sampled glandular structure The sampling parameter is β;

[0020]

[0021] S33: Generate pseudo-label image data with new glandular structure spatial distribution features; according to the glandular structure spatial distribution features obtained by sampling in S32 For image x l And its label y is transformed to obtain the new pseudo-label image data g 1 :

[0022]

[0023] S34: Generate pseudo-label image data with new pixel intensity features and glandular structure spatial distribution features; image x l Superimpose the pixel intensity difference features obtained by S31 sampling And according to the spatial distribution characteristics of glandular structure obtained by S32 Use the spatial transformation function to deform it and get another new pseudo-label data g 2 :

[0024]

[0025] In a preferred embodiment, the overall loss of the segmentation network in step S4 is composed of the supervision loss of the label and the pseudo-label supervision loss of generating pseudo-label data, as follows:

[0026]

[0027] where p l , Represent the label data and the segmentation prediction results of two pseudo-label data respectively; y,y g Represent the label of the label data and the pseudo label corresponding to the pseudo label data; L ls represents the loss of label supervision; L pls represents the pseudo-label supervision loss of the two pseudo-label data generated by calculation; Lseg Represents the calculation of the linear combination of DICE and cross entropy; λ and μ represent the weights of the two loss terms, respectively.

[0028] Compared with the prior art, the present invention has the following beneficial effects: the method proposed in the present invention effectively mines the potential correlation between clinical data through intra-patient registration technology, and generates diverse and realistic pseudo-label data under limited annotation. In the process of generating pseudo-label data, the present invention does not rely on the segmentation network, which fundamentally avoids the quality of the pseudo-label data being affected by the performance of the segmentation network.

[0029] Compared with existing methods, the present invention reduces the model's dependence on manually labeled data by virtue of more diverse and realistic pseudo-label data, and effectively improves the segmentation accuracy and generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the principle of a preferred embodiment of the present invention.

[0031] Figure 2 It is a schematic diagram of the method steps of a preferred embodiment of the present invention.

[0032] Figure 3 It is a schematic diagram of a registration sampling generation module according to a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0033] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0034] It should be noted that the following detailed descriptions are illustrative and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present application belongs.

[0035] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.

[0036] like Figure 1-3 As shown, this embodiment provides a semi-supervised segmentation method of meibomian gland images based on intra-patient registration, including the following four steps:

[0037] Step S1: Dataset processing: The infrared meibomian gland image dataset consisting of multiple images of different patients and corresponding labels is divided into training set, validation set and test set in a ratio of 7:1:2 based on the patients to whom the images belong, to ensure that the data of each patient only appears in one set. After the division, the infrared meibomian gland data of the same patient are affinely registered pairwise using the medical image registration library ANTsPy to obtain the affine displacement field and deformed image data for use in steps S3 and S4.

[0038] Step S2: Training of infrared meibomian gland image registration model for patients: using the image data after preliminary affine registration in step S1, a model for infrared meibomian gland image registration for patients is trained. The network is used to form a subsequent registration sampling generation module.

[0039] Step S3 is as follows Figure 3 As shown in the figure, the registration sampling generation module converts the labeled data (x l ,y), unlabeled data x uk and x uj As input, the specific sampling generation process consists of the following 4 steps:

[0040] S31: Obtain the intensity difference characteristics of the gland; l is a fixed image, x uk To move the image, we first perform affine coarse registration using ANTsPy to obtain the affine image A k2l , followed by x l Send it to the registration network for precise registration and use the spatial deformation function to obtain the deformed image Warp k2l , calculate the intensity difference feature And perform random sampling

[0041]

[0042] Among them, Affine(·,·) means using ANTsPy for affine registration, ST(·,·) means using the spatial transformation function to deform the image according to the deformation field. Reg(·,·) means using the registration network to perform precise registration to obtain the deformation field. The sampling parameter is α.

[0043] S32: Obtain the spatial distribution characteristics of glandular structure; uj is a fixed image, x l For moving images, ANTsPy affine coarse registration is performed to obtain A l2j and the corresponding affine displacement field ADF l2j , then together with the fixed image x uj Send it to the registration network for precise registration, get the deformation field, and superimpose the ADF l2j, and finally get the image x l to image x uj Spatial distribution characteristics of glandular structures l2j And perform random sampling with the sampling parameter β.

[0044]

[0045] S33: Generate pseudo-label image data with new glandular structure spatial distribution features; according to the structural spatial distribution features obtained by S32 sampling For image x l And its label y is transformed to obtain new pseudo-label image data:

[0046]

[0047] S34: Generate pseudo-label image data with new pixel intensity features and glandular structure spatial distribution features; image x l Superimpose the pixel intensity difference features obtained by S31 sampling And according to the spatial distribution characteristics of glandular structure obtained by S32 Use the spatial transformation function to deform it and get another new pseudo-label data:

[0048]

[0049] Step S4 segmentation model training: Figure 2 As shown, for the input data of a certain patient, the two pseudo-label data obtained by the registration sampling generation module participate in the training of the segmentation model together with the labeled data of the patient.

[0050] In this embodiment, step S1 divides the data set according to the patient in the ratio of training set: test set: validation set = 7:1:2, where the data of each patient will only appear in one of the sets; the saved deformed image and the corresponding affine displacement field will be used in steps S3 and S4.

[0051] In this embodiment, the registration model used in step S2 is composed of the registration network of VoxelMorph and the spatial transformation function. The input moving image and the fixed image are superimposed in the channel dimension and then input into the registration network to estimate the deformation field. The spatial transformation function performs a deformation operation on the moving image according to the deformation field, and uses the obtained deformation field and the deformed image for model training. The training uses structural consistency loss and normalized cross-correlation loss to constrain the similarity between the deformed image and the fixed image to ensure that the two have high consistency in structure and grayscale distribution; at the same time, the L2 norm is used to constrain the deformation field to suppress local exaggerated deformation, ensure the smoothness and rationality of the deformation field, and thus ensure the accuracy and stability of the registration. The weight ratio of structural consistency loss, normalized cross-correlation loss and L2 norm loss is set to 1:1:0.1.

[0052] In this embodiment, the registration sampling generation module in step S3 is used online during the training of the segmentation network (step S4). Different from the traditional use of Beta peak distribution, we let the sampling parameters α and β be obtained with equal probability between the set [0,1] to ensure that the images generated by sampling have more diversity.

[0053] In this embodiment, the segmentation network in step S4 adopts a group-normalized U-Net network to ensure better stability and performance. The overall loss of the segmentation network is composed of the supervision loss of the label and the pseudo-label supervision loss of generating pseudo-label data, which is as follows:

[0054]

[0055] where p l , Represent the label data and the segmentation prediction results of two pseudo-label data respectively; y,y g Represent the label of the label data and the pseudo label corresponding to the pseudo label data; L ls Calculate label supervision loss; L pls Calculate the pseudo-label supervision loss of the two pseudo-label data generated; L seg Calculate the linear combination of DICE and cross entropy. λ and μ represent the weights of the two loss terms, respectively. λ is 1 and μ is 0.5.

[0056] The present invention has the following beneficial effects: the method proposed in the present invention can mine potential correlations between clinical data, generate diverse and realistic pseudo-label data under limited annotation, reduce the model's dependence on manually labeled data, and effectively improve the model's segmentation accuracy and generalization ability.

[0057] In summary, the present invention uses intra-patient registration technology to explore potential correlations between clinical data, integrate unlabeled and labeled data of the same patient, and generate diverse and realistic pseudo-label data under limited annotation for segmentation model training. The pseudo-label data is generated without relying on the segmentation network, which fundamentally avoids the data quality being affected by the performance of the segmentation network. Compared with existing methods, the pseudo-label data generated by the present invention is more diverse and realistic, effectively reducing the model's dependence on manually labeled data, and significantly improving the segmentation accuracy and generalization ability of the model.

[0058] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the present invention.

Claims

1. A semi-supervised segmentation method for meibomian gland images based on intra-patient registration, characterized in that: The following steps are involved: Step S1, data set processing: dividing and preprocessing the infrared meibomian gland image data set, the infrared meibomian gland image data set includes multiple images of different patients and corresponding labels; The medical image registration library ANTsPy is used to perform affine registration on the infrared meibomian gland data of the same patient, and the affine displacement field and image data for training the registration model are obtained; Step S2, training a model for registration of infrared meibomian gland images in patients: using the image data after affine registration in step S1 to train a model for registration of infrared meibomian gland images in patients; Step S3, registration sampling generation module design: The registration sampling generation module converts a labeled data (x l ,y), two unlabeled data x uk and x uj As input, pseudo-label data is generated by sampling, thereby providing training samples for the training of the segmentation network and enhancing the generalization ability and segmentation accuracy of the model used for the registration of infrared meibomian gland images inside patients; Step S4, segmentation model training: The network is trained using labeled data and unlabeled data of the same patient, and the segmentation model is supervised through labeled images. At the same time, pseudo-label supervision is performed on the two pseudo-label data generated online by the data input registration sampling generation module, thereby completing the training of the segmentation network.

2. The semi-supervised segmentation method of meibomian gland images based on intra-patient registration according to claim 1, characterized in that: The step S1 processes the data set; divides the data set into a training set, a test set and a validation set according to the patients, and ensures that the data of each patient only appears in one of the sets; saves the obtained deformed image and the corresponding affine displacement field to be used in steps S3 and S4.

3. The semi-supervised segmentation method of meibomian gland images based on intra-patient registration according to claim 1, characterized in that: The registration model used in step S2 is composed of the registration network of VoxelMorph and the spatial transformation function; the input moving image and fixed image are superimposed in the channel dimension and then input into the registration network to estimate the deformation field; the spatial transformation function performs a deformation operation on the moving image according to the deformation field, and uses the obtained deformation field and the deformed image for model training; the training uses structural consistency loss and normalized cross-correlation loss to constrain the similarity between the deformed image and the fixed image to ensure that the two have high consistency in structure and grayscale distribution; at the same time, the L2 norm is used to constrain the deformation field to suppress local exaggerated deformation, ensure the smoothness and rationality of the deformation field, and thus ensure the accuracy and stability of the registration.

4. The semi-supervised segmentation method of meibomian gland images based on intra-patient registration according to claim 1, characterized in that: The registration sampling generation module in step S3 is used online during the training of the segmentation network. The registration sampling generation module utilizes the intrinsic connection of the patient's own data and integrates the pixel intensity characteristics and spatial distribution characteristics of the unlabeled data into the labeled data, so that the generated pseudo-label data is more consistent with the real meibomian gland image characteristics in terms of pixel intensity and spatial distribution. Different from the traditional use of Beta peak distribution, the sampling parameters α and β are obtained with equal probability between the set [0,1] to ensure that the images generated by sampling have more diversity.

5. The semi-supervised segmentation method of meibomian gland images based on intra-patient registration according to claim 1, characterized in that: The sampling generation process of step S3 includes the following steps: S31: Obtain the intensity difference characteristics of the gland; l is a fixed image, x uk To move the image, we first perform affine coarse registration using ANTsPy to obtain the affine image A k2l , then together with the fixed image x l Send it to the registration network for precise registration and use the spatial deformation function to obtain the deformed image Warp k2l , calculate the intensity difference feature And perform random sampling to obtain the intensity difference characteristics of the sampling The sampling parameter is α: Among them, Affine(·,·) means using ANTsPy for affine registration, ST(·,·) means using the spatial transformation function to deform the image according to the deformation field; Reg(·,·) means using the registration network for precise registration to obtain the deformation field; S32: Obtain the spatial distribution characteristics of glandular structure; uj is a fixed image, x l For moving images, ANTsPy affine coarse registration is performed to obtain x l Affine to x uj Affine image A of l2j and the corresponding affine displacement field ADF l2j , then together with the fixed image x uj Send it to the registration network for precise registration, get the deformation field, and superimpose the ADF l2j , and finally get the moving image x l Transform to a fixed image x uj Spatial distribution characteristics of glandular structures l2j And random sampling is performed to obtain the spatial distribution characteristics of the sampled glandular structure The sampling parameter is β; S33: Generate pseudo-label image data with new glandular structure spatial distribution features; according to the glandular structure spatial distribution features obtained by sampling in S32 For image x l And its label y is transformed to obtain the new pseudo-label image data g1: S34: Generate pseudo-label image data with new pixel intensity features and glandular structure spatial distribution features; image x l Superimpose the pixel intensity difference features obtained by S31 sampling And according to the spatial distribution characteristics of glandular structure obtained by S32 Use the spatial transformation function to transform it and get another new pseudo-label data g2:

6. The semi-supervised segmentation method of meibomian gland images based on intra-patient registration according to claim 1, characterized in that: The overall loss of the segmentation network in step S4 is composed of the supervision loss of the label and the pseudo-label supervision loss of generating pseudo-label data, as follows: where p l , Represent the label data and the segmentation prediction results of two pseudo-label data respectively; y,y g Represent the label of the label data and the pseudo label corresponding to the pseudo label data; L ls represents the loss of label supervision; L pls represents the pseudo-label supervision loss of the two pseudo-label data generated by calculation; L seg Represents the calculation of the linear combination of DICE and cross entropy; λ and μ represent the weights of the two loss terms, respectively.

Images