An OCT retinal automatic layering method based on a Mask2Former model
By combining the Mask2Former model with the Swing-transformer and JPU structure, the problems of detail loss and insufficient generalization in retinal layering of OCT images are solved, achieving higher layering accuracy and wider applicability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- NINGBO UNIVERSITY OF TECHNOLOGY
- Filing Date
- 2024-10-31
- Publication Date
- 2026-06-09
AI Technical Summary
In existing OCT image retinal layering techniques, the loss of detail is a problem that has not been effectively solved. The current technical problem is that existing techniques suffer from detail loss and lack of generalization characteristics during the retinal layering process, resulting in low layering accuracy.
We employ the Mask2Former model, combined with the Swin-transformer and JPU architecture, to automatically layer OCT images by constructing multi-scale feature maps and a Transformer decoder. We optimize the training process using a multi-attention mechanism and loss function, and train the model using various types of OCT image datasets.
It improves the accuracy of retinal layering, avoids loss of detail, and has better generalization properties, making it suitable for different types of OCT images.
Smart Images
Figure CN119417927B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of ophthalmology and image processing technology, and in particular to an automatic retinal layering method in OCT based on the Mask2Former model. Background Technology
[0002] OCT images are widely used by ophthalmologists to detect fundus diseases. For example, the analysis of two common blinding diseases, AMD and DME, requires first stratifying the retinal region in the image, and then analyzing the progression of fundus lesions by calculating the thickness between each layer and the changes in thickness. A common stratification method is manual annotation, using specialized annotation tools to divide each OCT image into layers. Obviously, when dealing with tens of millions of OCT images, this method is time-consuming, labor-intensive, and prone to errors.
[0003] In existing technologies, such as the "Automatic Layering Method for Retinal OCT Images" proposed by the Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, and the "Training Method, Device, Computer Equipment and Storage Medium for Retinal OCT Image Layering Model" proposed by Shenzhen University, they all employ similar methods. They first load training data, then select a deep learning algorithm to generate a retinal layering model, and finally input an unlayered OCT image into the model to output the layering result. The shortcomings of this type of method are as follows: (1) It uses older algorithms such as VGGNet and UNet. Although they can also achieve the goal of image segmentation, they are prone to losing details due to the reduction in the size of the feature image during downsampling, which affects the accuracy of the model; (2) The dataset used in this type of technology during training only focuses on one type of OCT and lacks generalization characteristics. Summary of the Invention
[0004] The technical problem to be solved by the present invention is to provide an automatic retinal layering method in OCT based on the Mask2Former model that can avoid loss of detail, improve layering accuracy, and has generalization characteristics.
[0005] The technical solution adopted in this invention is an automatic retinal layering method in OCT based on the Mask2Former model, which includes the following steps:
[0006] S1. Construct the original OCT image dataset, which includes several images of AMD symptoms, several images of DME symptoms, and several images without symptoms;
[0007] S2. Preprocess the original OCT image dataset to obtain the preprocessed original OCT image dataset;
[0008] S3. Perform manual layer annotation on each original OCT image in the original OCT image dataset to obtain the corresponding manually annotated images and form a manually annotated image dataset; wherein, the specific process of performing manual layer annotation on the original OCT images is as follows: divide the retinal region in the original OCT image into five layers from top to bottom, the five layers divide the space into six regions in the vertical direction, and fill the six regions with pixels of different pixel values in sequence to obtain manually annotated images;
[0009] S4. Construct a Mask2Former network model; the Mask2Former network model includes a backbone structure, a neck structure, and a decoder head; the backbone structure adopts the Swin-transformer feature extraction model, the neck structure adopts joint pyramid upsampling, the decoder head includes a pixel decoder and a Transformer decoder, the Transformer decoder includes N groups of decoding blocks, each group of decoding blocks includes three decoding blocks, namely the first decoding block, the second decoding block, and the third decoding block;
[0010] S5. Input the preprocessed original OCT image dataset and the manually labeled image dataset into the Mask2Former network model for training to obtain the trained Mask2Former network model; the specific training process is as follows:
[0011] S5.1 After the preprocessed original OCT image dataset is input into the Mask2Former network model, the backbone structure processes the OCT images and generates four sets of feature maps. The size of each set of feature maps is half that of the previous set, and the number of channels is doubled. The backbone structure extracts the last three sets of feature maps and inputs them into the neck structure for fusion. The neck structure generates a set of synthetic feature maps, the size of which is equal to the first set of feature maps generated by the backbone structure.
[0012] S5.2. Input the first set of synthesized feature maps and the last two sets of feature maps generated by the backbone structure into the pixel decoder. The pixel decoder processes each set of feature maps to generate corresponding multi-scale features, namely the first multi-scale feature, the second multi-scale feature, and the third multi-scale feature. Fuse all the multi-scale features with the first set of feature maps generated by the backbone structure to generate mask features. At the same time, input the first multi-scale feature, the second multi-scale feature, the third multi-scale feature, and the mask feature into the Transformer decoder. The N sets of decoding blocks in the Transformer decoder are used for training to complete N rounds of training. In each round of training, the Transformer decoder outputs the pixel filling map and the mask' feature, and uses a loss function to calculate the loss between the pixel filling map and the corresponding manually labeled image obtained in step S3, to obtain the trained Mask2Former network model.
[0013] S6. Input the OCT image to be layered into the trained Mask2Former network model, and the trained Mask2Former network model outputs the corresponding pixel filling map.
[0014] S7. Convert the pixel-filled image obtained in step S6 into a layered image of the retinal region.
[0015] The beneficial effects of this invention are as follows: This invention adopts the Mask2Former framework and the Swin-transformer structure, combined with the JPU structure, which can retain more upper-layer features, avoid loss of details, improve the layering accuracy, and have better retinal layering effect; in addition, the dataset used in this invention focuses on three types of OCT: AMD, DME, and normal retina, and has better generalization characteristics; before model training, this invention preprocesses the input raw OCT image to enable better feature extraction.
[0016] Preferably, in step S2, the specific process of preprocessing the original OCT image dataset is as follows: the size and format of all original OCT images in the original OCT image dataset are unified; then, each pixel of each original OCT image is standardized, specifically as follows: Where mean represents the average value of all pixels in the original OCT image, std represents the standard deviation of all pixels in the original OCT image, and P represents the pixel value of a single pixel in the original OCT image.
[0017] Preferably, in step S5.2, the first multi-scale feature, the second multi-scale feature, and the third multi-scale feature are input into the Transformer decoder, and the specific process of training by the N groups of decoding blocks in the Transformer decoder includes the following steps:
[0018] S5.21. In each round of training, the first multi-scale feature, the second multi-scale feature, and the third multi-scale feature are input into the current set of decoding blocks of the Transformer decoder, and the query vector is calculated by the first decoding block, the second decoding block, and the third decoding block in the current set of decoding blocks.
[0019] S5.22, combine the query vector, the first multi-scale feature, and the mask output from the previous set of decoding blocks. ！ The features serve as the input to the first decoding block of the current decoding block group. After passing through cross attention, self attention, and feedforward network layers in the first decoding block, the first result is output.
[0020] S5.23. The first result and the second multi-scale feature are used as the input of the second decoding block of the current decoding block group. After passing through cross attention, self attention and feedforward network layers in the second decoding block, the second result is output.
[0021] S5.24. The second result and the third multi-scale feature are used as the input to the third decoding block of the current decoding block group. After passing through cross attention, self attention and feedforward network layers in the third decoding block, the third result is output.
[0022] S5.25. Perform a multiplication operation on the third result and the mask feature generated by the pixel decoder to generate a pixel fill map and a mask' feature;
[0023] S5.26. Use the mask′ feature obtained in step S5.25 as the input of the first decoding block of the next decoding block group, and then use the next decoding block group to perform the next round of training until N rounds of training are completed.
[0024] Preferably, in step S5.2, the loss function is expressed as: Loss = λ1L cls +λ2L mask +λ3L Dice ;
[0025] Where λ1, λ2, and λ3 all represent coefficients, L cls L represents the cross-entropy loss function based on Softmax; mask L represents the Sigmoid-based cross-entropy loss function; Dice This represents the Dice loss function.
[0026] Preferably, in step S7, the specific process of converting the pixel-filled image obtained in step S6 into a layered image of the retinal region includes the following steps:
[0027] Step S7.1: Construct an M×N matrix, where M represents the width of the corresponding OCT image to be layered, and N represents the number of layers in the retinal region, N=5;
[0028] Step S7.2: The pixel filling image obtained in step S6 contains six regions in the vertical space. The pixel filling image is traversed pixel by pixel column by column to find two different pixels in the vertical space. These pixels are regarded as the vertical coordinates of the boundary position and recorded in the matrix.
[0029] Step S7.3: Connect each pixel in the same layer sequentially to obtain a layered image of the retinal region. Attached Figure Description
[0030] Figure 1 This is a block diagram illustrating the principle of an automatic retinal layering method in OCT based on the Mask2Former model according to the present invention.
[0031] Figure 2 This is a schematic diagram comparing the effects of the original OCT image, the pixel-filled image, and the retinal layered image obtained using the method of this invention. Detailed Implementation
[0032] The invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can implement it based on the description. The scope of protection of the invention is not limited to these specific embodiments.
[0033] This invention relates to an automatic retinal layering method in OCT based on the Mask2Former model, such as... Figure 1 As shown, the method includes the following steps:
[0034] Step 1: Construct the original OCT image dataset, which contains 1672 original OCT images, including 739 images related to AMD, 403 images related to DME, and 530 images without symptoms;
[0035] Among them, OCT (Optical Coherence Tomography) is an ophthalmic examination method that uses light waves as imaging. It has the characteristics of being non-contact, non-invasive, highly sensitive, and having high resolution.
[0036] AMD (Age-related Macular Degeneration): An ophthalmic disease caused by pathological aging changes in the structure of the macular region;
[0037] DME (Diabetic Macular Edema): A retinopathy caused by diabetes, it is a common complication of diabetes.
[0038] Step 2: Preprocess the original OCT image dataset to obtain the preprocessed original OCT image dataset;
[0039] Furthermore, the specific preprocessing process for the original OCT image dataset is as follows: The size and format of all original OCT images in the original OCT image dataset are unified; then, each pixel of each original OCT image is standardized, specifically as follows: Where mean represents the average value of all pixels in the original OCT image, std represents the standard deviation of all pixels in the original OCT image, and P represents the pixel value of a single pixel in the original OCT image.
[0040] Step 3: Perform manual layer-by-layer annotation on each original OCT image in the original OCT image dataset to obtain the corresponding manually annotated images, and combine all the manually annotated images to form the manually annotated image dataset;
[0041] Furthermore, the specific process of manually layering and annotating the original OCT image is as follows: the retinal region in the original OCT image is divided into five layers from top to bottom, namely ILM, OPL, IS / OS, OBRPE, and IBRPE; the five layers divide the space into six regions in the vertical direction, and the six regions are filled sequentially with pixels of different pixel values to obtain the manually layered and annotated image; for example, the region above ILM is filled with 0, the region between ILM and OPL is filled with 1, the region between OPL and IS / OS is filled with 2, the region between S / OS and OBRPE is filled with 3, the region between OBRPE and IBRPE is filled with 4, and the region below IBRPE is filled with 5.
[0042] Among them, ILM (Internal Limiting Membrane) is a thin membrane located between the retina and the vitreous body;
[0043] OPL (Outer Plexiform Layer): This is a loose, fleshy structure that forms the synaptic sites where the terminal spheres of cone and rod cells connect with the dendrites of bipolar cells and the processes of horizontal cells. It is located below the ILM layer.
[0044] IS / OS (Inner and Outer Segments): The junction between the inner and outer segments of the photoreceptor.
[0045] OBRPE (Outer Boundary of Retinal Pigment Epithelia): The upper boundary of the retinal pigment epithelial cell region;
[0046] IBRPE (Inner Boundary of Retinal Pigment Epithelia): The lower boundary of the retinal pigment epithelial cell region.
[0047] Step 4: Construct the Mask2Former network model; the Mask2Former network model includes a backbone structure, a neck structure, and a decoder head; the backbone structure adopts the Swin-transformer feature extraction model, the neck structure adopts joint pyramid upsampling, and the decoder head includes a pixel decoder and a Transformer decoder. The Transformer decoder includes N groups of decoding blocks, and each group of decoding blocks includes three decoding blocks, namely the first decoding block, the second decoding block, and the third decoding block.
[0048] In step four, the backbone and neck structures are used to obtain multi-level features; the pixel decoder is used for feature extraction; and the Transformer decoder is used to generate prediction results and calculate the loss between the prediction and the true target.
[0049] Mask2Former is an image segmentation model based on Transformer that supports semantic segmentation, instance segmentation, and panoptic segmentation tasks.
[0050] Swin-transformer: An image feature extraction model based on Transformer, featuring hierarchical feature representation and a self-attention mechanism based on shift windows, aiming to improve computational efficiency while maintaining high performance, and is used as the backbone structure in this invention;
[0051] JPU (Joint Pyramid Upsampling): The last three feature maps of the backbone are used as input and fused into a high-resolution feature map, which is used as the neck structure in this invention.
[0052] Step 5: Input the preprocessed original OCT image dataset and the manually labeled image dataset into the Mask2Former network model for training to obtain the trained Mask2Former network model; the specific training process is as follows:
[0053] S5.1 After each OCT image in the preprocessed original OCT image dataset is input into the Mask2Former network model, the backbone structure processes the OCT image and generates four sets of feature maps. The size of each set of feature maps is half that of the previous set, and the number of channels is doubled. The backbone structure extracts the last three sets of feature maps and inputs them into the neck structure for fusion. The neck structure generates a synthetic feature map to retain more detailed information and reduce the loss caused by upsampling. The size of the synthetic feature map is equal to the size of the first set of feature maps generated by the backbone structure.
[0054] S5.2. Input the first set of synthesized feature maps and the last two sets of feature maps generated by the backbone structure into the pixel decoder. The pixel decoder processes each set of feature maps to generate corresponding multi-scale features, namely the first multi-scale feature, the second multi-scale feature, and the third multi-scale feature. Fuse all the multi-scale features with the first set of feature maps generated by the backbone structure to generate the mask feature. At the same time, input the first multi-scale feature, the second multi-scale feature, the third multi-scale feature, and the mask feature into the Transformer decoder. The Transformer decoder performs training using N sets of decoding blocks. Here, N=9, so 9 rounds of training are required. In each round of training, the Transformer decoder outputs the pixel fill map and mask corresponding to the OCT image described in step S5.1. ！ The features are identified, and a loss function is used to calculate the loss between the pixel-filled image and the corresponding manually labeled image obtained in step S3; finally, the trained Mask2Former network model is obtained.
[0055] Furthermore, the preprocessed original OCT image dataset is input into the Mask2Former network model. The specific process of training the Transformer decoder with 9 decoding block groups for any given OCT image from the preprocessed original OCT image dataset includes the following steps:
[0056] S5.21. In each round of training, the first multi-scale feature, the second multi-scale feature, and the third multi-scale feature are input into the current set of decoding blocks of the Transformer decoder, and the query vector is calculated by the first decoding block, the second decoding block, and the third decoding block in the current set of decoding blocks.
[0057] S5.22, combine the query vector, the first multi-scale feature, and the mask output from the previous set of decoding blocks. ！ The features serve as the input to the first decoding block of the current decoding block group. After passing through cross attention, self attention, and feedforward network layers in the first decoding block, the first result is output.
[0058] S5.23. The first result and the second multi-scale feature are used as the input of the second decoding block of the current decoding block group. After passing through cross attention, self attention and feedforward network layers in the second decoding block, the second result is output.
[0059] S5.24. The second result and the third multi-scale feature are used as the input to the third decoding block of the current decoding block group. After passing through cross attention, self attention and feedforward network layers in the third decoding block, the third result is output.
[0060] S5.25. Perform a multiplication operation on the third result and the mask feature generated by the pixel decoder to generate the pixel filling map and mask' feature corresponding to the OCT image described in step S5.1.
[0061] S5.26. Use the mask′ feature obtained in step S5.25 as the input of the first decoding block of the next decoding block group, and the next decoding block group will carry out the next round of training until 9 rounds of training are completed.
[0062] Furthermore, the loss function is expressed as: Loss = λ1L cls +λ2L mask +λ3L Dice ;
[0063] Among them, λ1=0.2, λ2=0.5, λ3=0.5, L cls L represents the cross-entropy loss function based on Softmax; mask L represents the Sigmoid-based cross-entropy loss function; Dice This represents the Dice loss function.
[0064] Step 6: Test the trained Mask2Former network model. Input the OCT image to be layered into the trained Mask2Former network model, and the trained Mask2Former network model will output the corresponding pixel filling map.
[0065] Step 7: Convert the pixel-filled image obtained in step S6 into a layered image of the retinal region.
[0066] Furthermore, the specific process of converting the pixel-filled image obtained in step S6 into a layered image of the retinal region includes the following steps:
[0067] Step S7.1: Construct a 512×5 matrix;
[0068] Step S7.2: The pixel filling image obtained in step S6 contains six regions in the vertical space. The pixel filling image is traversed pixel by pixel column by column to find two different pixels in the vertical space. These pixels are regarded as the vertical coordinates of the boundary position and recorded in the matrix.
[0069] Step S7.3: Connect each pixel in the same layer sequentially to obtain a layered image of the retinal region.
[0070] This invention applies a semantic segmentation model to achieve automatic stratification of OCT images, dividing the retinal region into five layers: ILM, ONL, IS / OS, OBRPE, and IBRPE. This solves the problem of low efficiency in manual stratification and lays a good foundation for further analysis and diagnosis of fundus diseases.
[0071] This invention uses a Swin-transformer as the backbone, a JPU as the neck structure, and a Mask2Former as the decoder head, employing a multi-attention mechanism. This combination of techniques has not yet been applied to retinal layering in OCT images. Compared to existing deep learning models, the current architecture utilizes an attention mechanism that adaptively focuses on key parts of the image, improving pixel segmentation accuracy and thus reducing errors in retinal layering. This invention can layer retinal regions in AMD, DME, and normal OCT images, exhibiting better generalization characteristics.
Claims
1. An automatic retinal layering method in OCT based on the Mask2Former model, characterized in that: The method includes the following steps: S1. Construct the original OCT image dataset, which includes several images of AMD symptoms, several images of DME symptoms, and several images without symptoms; S2. Preprocess the original OCT image dataset to obtain the preprocessed original OCT image dataset; S3. Perform manual layer annotation on each original OCT image in the original OCT image dataset to obtain the corresponding manually annotated images and form a manually annotated image dataset; wherein, the specific process of performing manual layer annotation on the original OCT images is as follows: divide the retinal region in the original OCT image into five layers from top to bottom, the five layers divide the space into six regions in the vertical direction, and fill the six regions with pixels of different pixel values in sequence to obtain manually annotated images; S4. Construct a Mask2Former network model; the Mask2Former network model includes a backbone structure, a neck structure, and a decoder head; the backbone structure adopts the Swin-transformer feature extraction model, the neck structure adopts joint pyramid upsampling, the decoder head includes a pixel decoder and a Transformer decoder, the Transformer decoder includes N groups of decoding blocks, each group of decoding blocks includes three decoding blocks, namely the first decoding block, the second decoding block, and the third decoding block; S5. Input the preprocessed original OCT image dataset and the manually labeled image dataset into the Mask2Former network model for training to obtain the trained Mask2Former network model; the specific training process is as follows: S5.1 After each OCT image in the preprocessed original OCT image dataset is input into the Mask2Former network model, the backbone structure processes the OCT image and generates four sets of feature maps in succession. The size of each set of feature maps is half that of the previous set of feature maps, and the number of channels is doubled. The backbone structure extracts the last three sets of feature maps and inputs them into the neck structure for fusion. The neck structure generates a set of synthetic feature maps, the size of which is equal to the first set of feature maps generated by the backbone structure. S5.
2. Input the first set of synthesized feature maps and the last two sets of feature maps generated by the backbone structure into the pixel decoder. The pixel decoder processes each set of feature maps to generate corresponding multi-scale features, namely the first multi-scale feature, the second multi-scale feature, and the third multi-scale feature. Fuse all the multi-scale features with the first set of feature maps generated by the backbone structure to generate mask features. At the same time, input the first multi-scale feature, the second multi-scale feature, the third multi-scale feature, and the mask feature into the Transformer decoder. The N sets of decoding blocks in the Transformer decoder are used for training to complete N rounds of training. In each round of training, the Transformer decoder outputs the pixel fill map and mask' feature of the OCT image described in step S5.1, and uses a loss function to calculate the loss between the pixel fill map and the corresponding manually labeled image obtained in step S3, to obtain the trained Mask2Former network model. S6. Input the OCT image to be layered into the trained Mask2Former network model, and the trained Mask2Former network model outputs the corresponding pixel filling map. S7. Convert the pixel-filled image obtained in step S6 into a layered image of the retinal region.
2. The automatic retinal stratification method in OCT based on the Mask2Former model according to claim 1, characterized in that: In step S2, the specific process of preprocessing the original OCT image dataset is as follows: The size and format of all original OCT images in the original OCT image dataset are unified; then, each pixel of each original OCT image is standardized, specifically as follows: Where mean represents the average value of all pixels in the original OCT image, std represents the standard deviation of all pixels in the original OCT image, and P represents the pixel value of a single pixel in the original OCT image.
3. The automatic retinal stratification method in OCT based on the Mask2Former model according to claim 1 or 2, characterized in that: In step S5.2, the first multi-scale feature, the second multi-scale feature, and the third multi-scale feature are input into the Transformer decoder, and the specific process of training by the N groups of decoding blocks in the Transformer decoder includes the following steps: S5.
21. In each round of training, the first multi-scale feature, the second multi-scale feature, and the third multi-scale feature are input into the current set of decoding blocks of the Transformer decoder, and the query vector is calculated by the first decoding block, the second decoding block, and the third decoding block in the current set of decoding blocks. S5.
22. The query vector, the first multi-scale feature, and the mask! feature output from the previous decoding block group are used as the input to the first decoding block of the current decoding block group. After passing through cross attention, self attention, and feedforward network layers in the first decoding block, the first result is output. S5.
23. The first result and the second multi-scale feature are used as the input of the second decoding block of the current decoding block group. After passing through cross attention, self attention and feedforward network layers in the second decoding block, the second result is output. S5.
24. The second result and the third multi-scale feature are used as the input to the third decoding block of the current decoding block group. After passing through cross attention, self attention and feedforward network layers in the third decoding block, the third result is output. S5.
25. Perform a multiplication operation on the third result and the mask feature generated by the pixel decoder to generate the pixel fill map and mask' feature of the OCT image described in step S5.
1. S5.
26. Use the mask′ feature obtained in step S5.25 as the input of the first decoding block of the next decoding block group, and then use the next decoding block group to perform the next round of training until N rounds of training are completed.
4. The automatic retinal stratification method in OCT based on the Mask2Former model according to claim 3, characterized in that: In step S5.2, the loss function is expressed as: Loss = λ1L cls +λ2L mask +λ3L Dice ; Where λ1, λ2, and λ3 all represent coefficients, L cls L represents the cross-entropy loss function based on Softmax; mask L represents the Sigmoid-based cross-entropy loss function; Dice This represents the Dice loss function.
5. The automatic retinal stratification method in OCT based on the Mask2Former model according to claim 4, characterized in that: In step S7, the specific process of converting the pixel-filled image obtained in step S6 into a layered image of the retinal region includes the following steps: Step S7.1: Construct an M×N matrix, where M represents the width of the corresponding OCT image to be layered, and N represents the number of layers in the retinal region, N=5; Step S7.2: The pixel filling image obtained in step S6 contains six regions in the vertical space. The pixel filling image is traversed pixel by pixel column by column to find two different pixels in the vertical space. These pixels are regarded as the vertical coordinates of the boundary position and recorded in the matrix. Step S7.3: Connect each pixel in the same layer sequentially to obtain a layered image of the retinal region.