Eye socket image segmentation identification method based on deep learning

Abstract

The invention discloses an orbit image segmentation identification method based on deep learning, and the method is characterized in that an attention mechanism is introduced into a Unet network structure, model training is carried out through employing a conventional eye CT / MRI image, an eye region segmentation network model is constructed, the to-be-processed eye CT / MRI image is segmented into different regions through the segmentation network model, and the to-be-processed eye CT / MRI image is identified. And respectively measuring corresponding data of different areas to obtain an eye CT / MRI image segmentation result. The orbit image segmentation recognition method based on deep learning solves the problems that in the prior art, orbit MRI image segmentation is not clear, and image recognition and data measurement are not accurate.

Description

Technical Field

[0001] The present invention belongs to the technical field of medical image processing methods, and specifically relates to an orbital image segmentation and recognition method based on deep learning. Background Art

[0002] In the medical field, medical imaging examinations can provide imaging information of internal lesions in the body, such as the shape, size, presence or absence of capsule, blood supply, etc. Doctors read the films and integrate the information provided by imaging with their own knowledge to diagnose and treat patients.

[0003] With the continuous development of image processing, computer vision, deep learning and other technologies, orbital images have also been widely studied. Imaging examinations can intuitively and accurately display the internal structure and subtle lesions of the orbit, including thickening of the extraocular muscles, ocular fat infiltration, lacrimal gland enlargement, protrusion of the eyeball, etc., and are non-invasive and time-saving. They play an irreplaceable and important role in the diagnosis of TAO, judgment of disease activity, and monitoring of treatment response. MRI (Magnetic Resonance Imaging) has a high soft tissue resolution and has unique advantages in judging TAO orbital tissue inflammation.

[0004] The orbit includes the eyeball, optic nerve, extraocular muscles and orbital fat. The existing orbital image processing methods mainly identify each organ through organ segmentation and measure the corresponding data, and output and display the data, measure the corresponding data and display it on the system interface. The existing technology used for the recognition and input of orbital MRI image data is difficult to accurately display the various parts of the human orbit, and the interface is difficult to clearly reflect the evaluation content of the structure, etc., which is of limited help to clinical practice. Summary of the invention

[0005] The purpose of the present invention is to provide an orbital image segmentation and recognition method based on deep learning, which solves the problems of unclear orbital MRI image segmentation, inaccurate image recognition and data measurement in the prior art.

[0006] The technical solution adopted by the present invention is to introduce the attention mechanism into the Unet network structure based on the deep learning orbital image segmentation and recognition method, use the previous eye CT / MRI images for model training, construct an eye area segmentation network model, and divide the eye CT / MRI images to be processed into different areas through the segmentation network model, and then measure the corresponding data of different areas respectively to obtain the eye CT / MRI image segmentation results.

[0007] The technical solution adopted by the present invention is also characterized in that:

[0008] Furthermore, the orbital image segmentation and recognition method based on deep learning is specifically implemented as follows:

[0009] Step 1: Collect a data set containing eye CT / MRI images, annotate the eyeball, optic nerve, eye muscle, orbit and lacrimal gland structure in each image, and build an orbital image segmentation network based on deep learning according to the data set;

[0010] Step 2, using the segmentation network in step 1 to segment the orbital MRI image to be processed into different structural regions;

[0011] Step 3, identifying different structural regions in step 2 in different MRI sequences;

[0012] Step 4, measuring corresponding data of different structural regions identified in step 3, including distance length, area or volume;

[0013] Step 5: Input the data obtained in step 4 into a computer for visual display.

[0014] Furthermore, the orbital image segmentation network in step 1 is an Attention Unet model that introduces an attention mechanism based on the Unet network structure, which specifically includes the following steps:

[0015] Step 1.1: Collect a dataset containing eye CT / MRI images and divide it into a training set and a test set. Each image needs to be labeled with the eyeball, optic nerve, eye muscle, orbit, and lacrimal gland structures.

[0016] Step 1.2: Convert the images and annotations into the nii format required by Attention Unet.

[0017] Step 1.3, configure the training related json files;

[0018] Step 1.4: Run the Attention Unet training script to train the model and complete data enhancement, cross-validation, and multi-scale prediction.

[0019] Step 1.5: After each round of training, evaluate the Dice score of each category of the model on the validation set. If it is lower than 90%, adjust the training hyperparameters to continue training, including: learning rate, momentum, and weight decay, until Dice reaches 90% or above;

[0020] Step 1.6: Run the trained model on the test set to generate prediction results, set values ​​greater than 0.5 to 1, values ​​less than 0.5 to 0, fill holes, remove small areas, retain the two largest areas on the left and right, and get the final prediction results.

[0021] Furthermore, in step 2, the different regions specifically include: the optic nerve region of the eyeball, the zygomatic apex region, the eye muscle region, the eye gland region, the optic nerve region, and the fat body region.

[0022] Furthermore, different structural regions in step 3 are identified in different MRI sequences, and the specific correspondence is as follows: the eyeball region, zygomatic apex region, and fat body region are in the axial T2WI sequence, the eye muscle region is in the coronal T1WI sequence, and the lacrimal gland region and optic nerve region are in the coronal T2WI fat-suppressed sequence.

[0023] Furthermore, the measurements of different structural regions in step 4 are as follows:

[0024] (1) Optic nerve area: Use the Unet network to generate labels for two eyeballs. Analyze the output results to find the most prominent point and the farthest point corresponding to it. Then, find the center position of the left and right eyeballs based on the maximum connected area to separate the left and right eyeballs. Based on the segmentation, the volume of the optic nerve area is calculated by calculating the size of the xyz pixel points.

[0025] (2) Zygomatic apex area: Based on the position of the eye sockets, all shapes with clear boundaries on the left and right eye sockets are found. Then, the shapes are screened based on their perimeter and area-to-perimeter ratio to select the zygomatic apex with the smallest area.

[0026] (3) Eye muscle area: All the results obtained by the segmentation network in step 1 are divided into eight eye muscle shapes through the maximum connected area, and then the major and minor diameters are found by fitting the circumscribed rectangle of each layer;

[0027] (4) Lacrimal gland area: Determine the area of ​​the lacrimal gland area based on the segmentation network result in step 2;

[0028] (5) Optic nerve area: Based on the results obtained in the segmentation network in step 2, find the layer closest to the circle by elliptical fitting.

[0029] (6) Fat body area: Based on the results obtained by the segmentation network in step 2, the fat body result is obtained by screening according to the threshold, selecting a threshold greater than 90%, and subtracting the result of the eyeball.

[0030] The beneficial effects of the present invention are:

[0031] 1. Use deep learning and other technologies to identify and display orbital structures (eyeball, optic nerve, eye muscle, orbital fat body and lacrimal gland) in different MRI sequences;

[0032] 2. Attention Unet is an open source medical image segmentation model based on the Unet network structure. Its core idea is to introduce an attention mechanism, which allows the model to focus on a smaller eye area. The accuracy of image segmentation is greatly improved, making it suitable for eye image segmentation scenarios.

[0033] 3. Measure the corresponding distance, length, area, volume, etc. of the identified structure, output the results, and design a human-computer interactive page to better adapt to and assist clinical practice. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a simple flow chart of the method of the present invention;

[0035] Figure 2 This is a schematic diagram of the eight extraocular muscles of both eyes on the coronal T1WI plain scan image of the method of the present invention;

[0036] Figure 3 Schematic diagram of the eyelid glands of both eyes hooked on the coronal T2WI fat-suppressed image of the method of the present invention;

[0037] Figure 4 Schematic diagram of the model for identifying extraocular muscles according to the method of the present invention;

[0038] Figure 5 is a schematic diagram of the lacrimal gland identification by the model of the method of the present invention;

[0039] Figure 6 It is a schematic diagram of the eyeball and optic nerve recognition by the model of the method of the present invention;

[0040] Figure 7 is a schematic diagram of the model of the method of the present invention for identifying orbital fat bodies;

[0041] Figure 8 It is a schematic diagram of measuring the long and short diameters of the extraocular muscles according to the method of the present invention. DETAILED DESCRIPTION

[0042] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0043] The present invention provides an orbital image segmentation and recognition method based on deep learning, introduces the attention mechanism into the Unet network structure, uses previous eye CT / MRI images for model training, constructs an eye region segmentation network model, and divides the eye CT / MRI images to be processed into different regions through the segmentation network model. Then, the corresponding data of different regions are measured respectively to obtain the eye CT / MRI image segmentation results.

[0044] like Figure 1 The specific implementation is as follows:

[0045] Step 1: Collect a data set containing eye CT / MRI images, annotate the eyeball, optic nerve, eye muscle, orbit and lacrimal gland structure in each image, and build an orbital image segmentation network based on deep learning according to the data set;

[0046] Step 1.1: Collect a dataset containing eye CT / MRI images and divide it into a training set and a test set. Each image needs to be labeled with the eyeball, optic nerve, eye muscle, orbit, and lacrimal gland structures.

[0047] Step 1.2: Convert the images and annotations into the nii format required by Attention Unet.

[0048] Step 1.3, configure the training related json files;

[0049] Step 1.4: Run the Attention Unet training script to train the model and complete data enhancement, cross-validation, and multi-scale prediction.

[0050] Step 1.5: After each round of training, evaluate the Dice score of each category of the model on the validation set. If it is lower than 90%, adjust the training hyperparameters to continue training, including: learning rate, momentum, and weight decay, until Dice reaches 90% or above;

[0051] Step 1.6: Run the trained model on the test set to generate prediction results, set values ​​greater than 0.5 to 1, and values ​​less than 0.5 to 0, fill holes, remove small areas, retain the two largest areas on the left and right, and get the final prediction results;

[0052] Step 2, using the segmentation network of step 1 to segment the orbital MRI image to be processed into different structural regions; the different regions specifically include: eyeball optic nerve region, zygomatic apex region, eye muscle region, eye gland region, optic nerve region, and fat body region;

[0053] Step 3, identifying different structural regions in step 2 in different MRI sequences; the specific corresponding relationships are as follows: the eyeball region, zygomatic apex region, and fat body region in the axial T2WI sequence, the eye muscle region in the coronal T1WI sequence, and the lacrimal gland region and optic nerve region in the coronal T2WI fat-suppressed sequence;

[0054] Step 4, measuring corresponding data of different structural regions identified in step 3, including distance length, area or volume;

[0055] The measurements of different structural areas in step 4 are as follows:

[0056] (1) Optic nerve area: Use the Unet network to generate labels for two eyeballs. Analyze the output results to find the most prominent point and the farthest point corresponding to it. Then, find the center position of the left and right eyeballs based on the maximum connected area to separate the left and right eyeballs. Based on the segmentation, the volume of the optic nerve area is calculated by calculating the size of the xyz pixel points.

[0057] (2) Zygomatic apex area: Based on the position of the eye sockets, all shapes with clear boundaries on the left and right eye sockets are found. Then, the shapes are screened based on their perimeter and area-to-perimeter ratio to select the zygomatic apex with the smallest area.

[0058] (3) Eye muscle area: All the results obtained by the segmentation network in step 1 are divided into eight eye muscle shapes through the maximum connected area, and then the major and minor diameters are found by fitting the circumscribed rectangle of each layer;

[0059] (4) Lacrimal gland area: Determine the area of ​​the lacrimal gland area based on the segmentation network result in step 2;

[0060] (5) Optic nerve area: Based on the results obtained in the segmentation network in step 2, find the layer closest to the circle by elliptical fitting.

[0061] (6) Fat body area: Based on the results obtained by the segmentation network in step 2, the fat body result is obtained by screening according to the threshold, selecting a threshold greater than 90%, and subtracting the result of the eyeball;

[0062] Step 5: Input the data obtained in step 4 into a computer for visual display.

[0063] The technical solution of the present invention is further described below through the accompanying drawings and embodiments.

[0064] Example 1

[0065] The orbital image segmentation and recognition method based on deep learning introduces the attention mechanism into the Unet network structure, uses the previous eye CT / MRI images for model training, and constructs the eye region segmentation network model. The eye CT / MRI images to be processed are segmented into different regions through the segmentation network model, and then the corresponding data of different regions are measured respectively to obtain the eye CT / MRI image segmentation results; the specific implementation method is as follows:

[0066] Step 1: Collect a data set containing eye CT / MRI images, annotate the eyeball, optic nerve, eye muscle, orbit and lacrimal gland structure in each image, and build an orbital image segmentation network based on deep learning according to the data set;

[0067] Step 2, using the segmentation network in step 1 to segment the orbital MRI image to be processed into different structural regions;

[0068] Step 3, identifying different structural regions in step 2 in different MRI sequences;

[0069] Step 4, measuring corresponding data of different structural regions identified in step 3, including distance length, area or volume;

[0070] Step 5: Input the data obtained in step 4 into a computer for visual display.

[0071] Example 2

[0072] The orbital image segmentation and recognition method based on deep learning introduces the attention mechanism into the Unet network structure, uses the previous eye CT / MRI images for model training, and constructs the eye region segmentation network model. The eye CT / MRI images to be processed are segmented into different regions through the segmentation network model, and then the corresponding data of different regions are measured respectively to obtain the eye CT / MRI image segmentation results; the specific implementation method is as follows:

[0073] Step 1: Collect a data set containing eye CT / MRI images, annotate the eyeball, optic nerve, eye muscle, orbit and lacrimal gland structure in each image, and build an orbital image segmentation network based on deep learning according to the data set;

[0074] Step 1.1: Collect a dataset containing eye CT / MRI images and divide it into a training set and a test set. Each image needs to be labeled with the eyeball, optic nerve, eye muscle, orbit, and lacrimal gland structures.

[0075] Step 1.2: Convert the images and annotations into the nii format required by Attention Unet.

[0076] Step 1.3, configure the training related json files;

[0077] Step 1.4: Run the Attention Unet training script to train the model and complete data enhancement, cross-validation, and multi-scale prediction.

[0078] Step 1.5: After each round of training, evaluate the Dice score of each category of the model on the validation set. If it is lower than 90%, adjust the training hyperparameters to continue training, including: learning rate, momentum, and weight decay, until Dice reaches 90% or above;

[0079] Step 1.6: Run the trained model on the test set to generate prediction results, set values ​​greater than 0.5 to 1, and values ​​less than 0.5 to 0, fill holes, remove small areas, retain the two largest areas on the left and right, and get the final prediction results;

[0080] Step 2, using the segmentation network in step 1 to segment the orbital MRI image to be processed into different structural regions;

[0081] Step 3, identifying different structural regions in step 2 in different MRI sequences;

[0082] Step 4, measuring corresponding data of different structural regions identified in step 3, including distance length, area or volume;

[0083] Step 5: Input the data obtained in step 4 into a computer for visual display.

[0084] Example 3

[0085] An orbital image segmentation and recognition method based on deep learning was proposed. The attention mechanism was introduced into the Unet network structure. The previous eye CT / MRI images were used for model training to construct an eye region segmentation network model. The eye CT / MRI images to be processed were segmented into different regions through the segmentation network model. Then the corresponding data of different regions were measured respectively to obtain the eye CT / MRI image segmentation results.

[0086] The specific implementation method is as follows:

[0087] Step 1: Collect a data set containing eye CT / MRI images, annotate the eyeball, optic nerve, eye muscle, orbit and lacrimal gland structure in each image, and build an orbital image segmentation network based on deep learning according to the data set;

[0088] Step 1.1, collect a dataset containing eye CT / MRI images and divide it into a training set and a test set in a ratio of 7:1:2. Each image needs to be labeled with the eyeball, optic nerve, eye muscle, orbit and lacrimal gland structure;

[0089] Step 1.2: Convert the images and annotations into the nii format required by Attention Unet.

[0090] Step 1.3, configure the training related json files;

[0091] Step 1.4: Run the Attention Unet training script to train the model and complete data enhancement, cross-validation, and multi-scale prediction.

[0092] Step 1.5: After each round of training, evaluate the Dice score of each category of the model on the validation set. If it is lower than 90%, adjust the training hyperparameters to continue training, including: learning rate, momentum, and weight decay, until Dice reaches 90% or above;

[0093] Step 1.6: Run the trained model on the test set to generate prediction results, set values ​​greater than 0.5 to 1, and values ​​less than 0.5 to 0, fill holes, remove small areas, retain the two largest areas on the left and right, and get the final prediction results;

[0094] Step 2, using the segmentation network of step 1 to segment the orbital MRI image to be processed into different structural regions; the different regions specifically include: eyeball optic nerve region, zygomatic apex region, eye muscle region, eye gland region, optic nerve region, and fat body region;

[0095] Step 3: Identify the different structural regions in step 2 in different MRI sequences; the specific correspondence is as follows: the eyeball region, zygomatic apex region, and fat body region in the axial T2WI sequence, and the eye muscle region in the coronal T1WI sequence, such as Figure 2 As shown, the lacrimal gland area and optic nerve area are in the coronal T2WI fat-suppressed sequence, as shown in Figure 3 As shown;

[0096] Step 4, measuring corresponding data of different structural regions identified in step 3, including distance length, area or volume;

[0097] The measurements of different structural areas in step 4 are as follows:

[0098] (1) Optic nerve area: Use the Unet network to generate labels for two eyeballs. Analyze the output results to find the most prominent point and the farthest point corresponding to it. Then, find the center position of the left and right eyeballs based on the maximum connected area to separate the left and right eyeballs. Based on the segmentation, the volume of the optic nerve area is calculated by calculating the size of the xyz pixel points.

[0099] (2) Zygomatic apex area: Based on the position of the eye sockets, all shapes with clear boundaries on the left and right eye sockets are found. Then, the shapes are screened based on their perimeter and area-to-perimeter ratio to select the zygomatic apex with the smallest area.

[0100] (3) Eye muscle area: All the results obtained by the segmentation network in step 1 are divided into the shapes of the eight eye muscles obtained by the maximum connected area, such as Figure 4 As shown, the long diameter and short diameter are found by fitting the circumscribed rectangle of each layer, as shown in Figure 8 As shown;

[0101] (4) Lacrimal gland area: The area of ​​the lacrimal gland area is determined based on the segmentation network result in step 2, such as Figure 5 As shown;

[0102] (5) Optic nerve area: Based on the results obtained in the segmentation network in step 2, find the layer closest to the circle by fitting an ellipse, such as Figure 6 As shown;

[0103] (6) Fat body area: Based on the results obtained by the segmentation network in step 2, the fat body result is obtained by screening according to the threshold, selecting a threshold greater than 90%, and subtracting the result of the eyeball, such as Figure 7 As shown;

[0104] Step 5: Input the data obtained in step 4 into a computer for visual display.

Claims

1. A deep learning-based orbital image segmentation and recognition method, characterized in that: The attention mechanism is introduced into the Unet network structure, and the previous eye CT / MRI images are used for model training to construct an eye region segmentation network model. The eye CT / MRI images to be processed are segmented into different regions through the segmentation network model, and then the corresponding data of different regions are measured respectively to obtain the eye CT / MRI image segmentation results.

2. The orbital image segmentation and recognition method based on deep learning according to claim 1, characterized in that: The specific implementation method is as follows: Step 1: Collect a data set containing eye CT / MRI images, annotate the eyeball, optic nerve, eye muscle, orbit and lacrimal gland structure in each image, and build an orbital image segmentation network based on deep learning according to the data set; Step 2, using the segmentation network in step 1 to segment the orbital MRI image to be processed into different structural regions; Step 3, identifying different structural regions in step 2 in different MRI sequences; Step 4, measuring corresponding data of different structural regions identified in step 3, including distance length, area or volume; Step 5: Input the data obtained in step 4 into a computer for visual display.

3. The orbital image segmentation and recognition method based on deep learning according to claim 1, characterized in that: The orbital image segmentation network in step 1 is an Attention Unet model that introduces an attention mechanism based on the Unet network structure, and specifically includes the following steps: Step 1.1: Collect a dataset containing eye CT / MRI images and divide it into a training set and a test set. Each image needs to be labeled with the eyeball, optic nerve, eye muscle, orbit, and lacrimal gland structures. Step 1.2: Convert the images and annotations into the nii format required by Attention Unet. Step 1.3, configure the training related json files; Step 1.4: Run the Attention Unet training script to train the model and complete data enhancement, cross-validation, and multi-scale prediction. Step 1.5: After each round of training, evaluate the Dice score of each category of the model on the validation set. If it is lower than 90%, adjust the training hyperparameters to continue training, including: learning rate, momentum, and weight decay, until Dice reaches 90% or above; Step 1.6: Run the trained model on the test set to generate prediction results, set values ​​greater than 0.5 to 1, values ​​less than 0.5 to 0, fill holes, remove small areas, retain the two largest areas on the left and right, and get the final prediction results.

4. The orbital image segmentation and recognition method based on deep learning according to claim 2, characterized in that: In step 2, the different regions specifically include: the optic nerve region of the eyeball, the zygomatic apex region, the eye muscle region, the eye gland region, the optic nerve region, and the fat body region.

5. The orbital image segmentation and recognition method based on deep learning according to claim 2, characterized in that: In step 3, different structural regions are identified in different MRI sequences, and the specific corresponding relationships are as follows: the eyeball region, zygomatic apex region, and fat body region are identified in the axial T2WI sequence, the eye muscle region is identified in the coronal T1WI sequence, and the lacrimal gland region and optic nerve region are identified in the coronal T2WI fat-suppressed sequence.

6. The orbital image segmentation and recognition method based on deep learning according to claim 2, characterized in that: The specific measurement of different structural regions in step 4 is as follows: (1) Optic nerve area: Use the Unet network to generate labels for two eyeballs. Analyze the output results to find the most prominent point and the farthest point corresponding to it. Then, find the center position of the left and right eyeballs based on the maximum connected area to separate the left and right eyeballs. Based on the segmentation, the volume of the optic nerve area is calculated by calculating the size of the xyz pixel points. (2) Zygomatic apex area: Based on the position of the eye sockets, all shapes with clear boundaries on the left and right eye sockets are found. Then, the shapes are screened based on their perimeter and area-to-perimeter ratio to select the zygomatic apex with the smallest area. (3) Eye muscle area: All the results obtained by the segmentation network in step 1 are divided into eight eye muscle shapes through the maximum connected area, and then the major and minor diameters are found by fitting the circumscribed rectangle of each layer; (4) Lacrimal gland area: Determine the area of ​​the lacrimal gland area based on the segmentation network result in step 2; (5) Optic nerve area: Based on the segmentation results obtained in step 2, find the layer closest to the circle by ellipse fitting; (6) Fat body area: Based on the results obtained by the segmentation network in step 2, the fat body result is obtained by screening according to the threshold, selecting a threshold greater than 90%, and subtracting the result of the eyeball.

Images