Automatic retina layering method in OCT (Optical Coherence Tomography) based on Mask2Former model

Abstract

The invention relates to a Mask2Former model-based retina automatic layering method in OCT (Optical Coherence Tomography). The method comprises the following steps of: constructing an original OCT image data set; carrying out manual layered labeling on the original OCT image data set to obtain a corresponding manual layered labeling image data set; a Mask2Former network model is constructed; the Mask2Former network model comprises a backbone structure, a neck structure and a decoder head; the backbone structure adopts a Swinin-transform feature extraction model, the neck structure adopts combined pyramid up-sampling, and the decoder head comprises a pixel decoder and a transform decoder; and training is carried out in the Mask2Former network model. According to the method, a Mask2Former framework and a Swindow-transformer structure are adopted, and a JPU structure is combined, so that more upper-layer features can be reserved, detail loss is avoided, the layering accuracy is improved, and a better retina layering effect is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of ophthalmology and image processing, and in particular to an automatic retinal layering method in OCT based on a Mask2Former model. Background Art

[0002] OCT images are widely used by ophthalmologists to detect fundus diseases. For example, the analysis of two common blinding diseases, AMD and DME, requires first stratifying the retinal area in the image, and then analyzing the progression of fundus lesions by calculating the thickness between the layers and the thickness changes. The commonly used stratification method is to use manual annotation, that is, to use special annotation tools to divide each OCT image into layers. Obviously, faced with tens of millions of OCT images, this method is time-consuming, labor-intensive, and prone to errors.

[0003] In the existing technology, for example, the "Automatic stratification method of retinal OCT images" proposed by the Suzhou Institute of Biomedical Engineering and Technology of the Chinese Academy of Sciences and the "A retinal OCT image stratification model training method, device, computer equipment and storage medium" proposed by Shenzhen University, the methods they adopt are similar, first loading the training data, then selecting a deep learning algorithm to generate a retinal stratification model, and finally inputting the unstratified OCT image into the model to output the stratification result. The defects of this type of method are as follows: (1) Using older algorithms such as VGGNet and UNet, although they can also achieve the goal of image segmentation, due to the reduction of the feature image size during the downsampling process, details are easily lost, affecting the accuracy of the model; (2) The data set used in the training of this type of technology only focuses on one type of OCT and lacks generalization characteristics. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an automatic retinal stratification method in OCT based on the Mask2Former model, which can avoid detail loss, improve stratification accuracy and have generalization characteristics.

[0005] The technical solution adopted by the present invention is a method for automatic retinal layering in OCT based on the Mask2Former model, which comprises the following steps:

[0006] S1. constructing an original OCT image dataset, wherein the original OCT image dataset includes a plurality of images related to AMD symptoms, a plurality of images related to DME symptoms, and a plurality of asymptomatic images;

[0007] S2, preprocessing the original OCT image data set to obtain a preprocessed original OCT image data set;

[0008] S3, manually layering and annotating each original OCT image in the original OCT image data set to obtain a corresponding manually layered and annotated image, and forming a manually layered and annotated image data set; wherein the specific process of manually layering and annotating the original OCT image is: dividing the retinal area in the original OCT image into five layers from top to bottom, the five layers divide the space into six areas in the vertical direction, and using pixel points with different pixel values ​​to fill the six areas in turn to obtain a manually layered and annotated image;

[0009] S4, constructing a Mask2Former network model; the Mask2Former network model includes a backbone structure, a neck structure and a decoder head; the backbone structure adopts a Swin-transformer feature extraction model, the neck structure adopts a joint pyramid upsampling, the decoder head includes a pixel decoder and a Transformer decoder, the Transformer decoder includes N groups of decoding block groups, each group of decoding block groups includes three decoding blocks, namely a first decoding block, a second decoding block and a third decoding block;

[0010] S5. Input the preprocessed original OCT image dataset and the manually layered annotated image dataset into the Mask2Former network model for training to obtain a trained Mask2Former network model; wherein the specific training process is:

[0011] S5.1. After the OCT images in the preprocessed original OCT image data set are input into the Mask2Former network model, the backbone structure processes the OCT images and generates four sets of feature maps in succession, wherein the size of each set of feature maps is half of the previous set of feature maps, and the number of channels is doubled; the backbone structure extracts the last three sets of feature maps and transmits them to the neck structure for fusion, and the neck structure generates a set of synthetic feature maps, whose size is equal to the first set of feature maps generated by the backbone structure;

[0012] S5.2, input the set of synthetic feature maps and the last two sets of feature maps generated by the backbone structure into the pixel decoder, and the pixel decoder processes each set of feature maps to generate corresponding multi-scale features, namely, first multi-scale features, second multi-scale features and third multi-scale features, and fuses all multi-scale features with the first set of feature maps generated by the backbone structure to generate mask features; at the same time, input the first multi-scale features, the second multi-scale features, the third multi-scale features and the mask features into the Transformer decoder, and train them by N sets of decoding block groups in the Transformer decoder to complete N rounds of training; in each round of training, the Transformer decoder outputs a pixel filling map and a mask′ feature, and uses a loss function to calculate the loss between the pixel filling map and the corresponding artificial layered annotated image obtained in step S3, to obtain a trained Mask2Former network model;

[0013] S6, inputting the OCT image to be layered into the trained Mask2Former network model, and the trained Mask2Former network model outputs a corresponding pixel filling map;

[0014] S7. Convert the pixel filling image obtained in step S6 into a layered image of the retinal area.

[0015] The beneficial effects of the present invention are as follows: the present invention adopts the Mask2Former framework and the Swin-transformer structure, and combines with the JPU structure, which can retain more upper-layer features, avoid detail loss, improve the stratification accuracy, and have a better retinal stratification effect; and the data set used by the present invention focuses on three types of OCT, namely AMD disease, DME disease and normal retina, and has better generalization characteristics; before model training, the present invention preprocesses the input original OCT image so that it can better extract features.

[0016] Preferably, in step S2, the specific process of preprocessing the original OCT image data set is: unifying the size and format of all original OCT images in the original OCT image data set; and then standardizing each pixel point of each original OCT image, which is specifically expressed as: Among them, mean represents the average value of all pixels on the original OCT image, std represents the standard deviation of all pixels on the original OCT image, and P represents the pixel value of a pixel on the original OCT image.

[0017] Preferably, in step S5.2, the first multi-scale feature, the second multi-scale feature and the third multi-scale feature are input into a Transformer decoder, and the specific process of training by the N groups of decoding blocks in the Transformer decoder includes the following steps:

[0018] S5.21. In each round of training, the first multi-scale feature, the second multi-scale feature, and the third multi-scale feature are input into a current set of decoding blocks of the Transformer decoder, and the query vector is calculated by the first decoding block, the second decoding block, and the third decoding block in the current set of decoding blocks;

[0019] S5.22, the query vector, the first multi-scale feature, and the mask output by the previous set of decoding blocks ！ The feature is used as the input of the first decoding block of the current set of decoding blocks, and the first result is output after the cross attention, self-attention and feed-forward network layers in the first decoding block;

[0020] S5.23, using the first result and the second multi-scale feature as inputs of a second decoding block of the current set of decoding blocks, and outputting a second result after the cross attention, self-attention and feed-forward network layers in the second decoding block;

[0021] S5.24, using the second result and the third multi-scale feature as inputs of a third decoding block of the current set of decoding blocks, and outputting a third result after passing through the cross attention, self-attention and feedforward network layers in the third decoding block;

[0022] S5.25, multiplying the third result and the mask feature generated by the pixel decoder to generate a pixel filling map and a mask′ feature;

[0023] S5.26. Use the mask′ feature obtained in step S5.25 as the input of the first decoding block of the next set of decoding block groups, and the next set of decoding block groups performs the next round of training until N rounds of training are completed.

[0024] Preferably, in step S5.2, the loss function is expressed as: Loss = λ1L cls +λ2L mask +λ3L Dice ;

[0025] Among them, λ1, λ2 and λ3 are coefficients, L cls represents the cross entropy loss function based on Softmax; L mask Represents the cross entropy loss function based on Sigmoid; L Dice Represents the Dice loss function.

[0026] Preferably, in step S7, the specific process of converting the pixel filling image obtained in step S6 into the layered image of the retinal area includes the following steps:

[0027] Step S7.1, constructing an M×N matrix, wherein M represents the width of the corresponding OCT image to be layered, and N represents the number of layers of the retinal area, N=5;

[0028] Step S7.2: The pixel filling map obtained in step S6 includes six areas in the vertical space. The pixel filling map is traversed pixel by pixel in columns to find two different pixel points before and after the pixel filling map in the vertical space, which are regarded as the ordinates of the boundary positions and recorded in the matrix;

[0029] Step S7.3: Connect each pixel point in the same layer in sequence to obtain a layered image of the retinal area. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 The present invention is a block diagram of the principle of an automatic retinal layering method in OCT based on the Mask2Former model;

[0031] Figure 2 It is a schematic diagram for comparing the effects of the original OCT image, the pixel filling map and the retinal layered image obtained by the method of the present invention. DETAILED DESCRIPTION

[0032] The invention will be further described below with reference to the accompanying drawings and in combination with specific implementations, so that those skilled in the art can implement the invention with reference to the description. The protection scope of the invention is not limited to the specific implementations.

[0033] The present invention relates to an automatic retinal layering method in OCT based on the Mask2Former model. Figure 1 As shown, the method comprises the following steps:

[0034] Step 1: construct an original OCT image dataset, wherein the original OCT image dataset includes 1672 original OCT images, including 739 images of AMD, 403 images of DME, and 530 images without symptoms;

[0035] Among them, OCT (Optical Coherence Tomography) is an ophthalmic examination method that uses light waves as imaging, and has the characteristics of non-contact, non-invasive, highly sensitive and high-resolution;

[0036] AMD (Age-related Macular Degeneration): An eye disease caused by pathological aging changes in the structure of the macula;

[0037] DME (Diabetic Macular Edema) is a common diabetic complication.

[0038] Step 2: preprocessing the original OCT image data set to obtain a preprocessed original OCT image data set;

[0039] Furthermore, the specific process of preprocessing the original OCT image data set is: unifying the size and format of all original OCT images in the original OCT image data set; and then standardizing each pixel point of each original OCT image, which is specifically expressed as: Among them, mean represents the average value of all pixels on the original OCT image, std represents the standard deviation of all pixels on the original OCT image, and P represents the pixel value of a pixel on the original OCT image.

[0040] Step 3: manually layer-label each original OCT image in the original OCT image dataset to obtain a corresponding manually layer-labeled image, and all the manually layer-labeled images form a manually layer-labeled image dataset;

[0041] Furthermore, the specific process of manually layering and labeling the original OCT image is as follows: the retinal area in the original OCT image is divided into five layers from top to bottom, namely ILM, OPL, IS / OS, OBRPE and IBRPE; the five layers divide the space into six regions in the vertical direction, and the six regions are filled in sequence with pixels of different pixel values ​​to obtain a manually layered and labeled image; for example, the area above ILM is filled with 0, the area between ILM and OPL is filled with 1, the area between OPL and IS / OS is filled with 2, the area between S / OS and OBRPE is filled with 3, the area between OBRPE and IBRPE is filled with 4, and the area below IBRPE is filled with 5;

[0042] Among them, ILM (Internal Limiting Membrane): a thin film between the retina and the vitreous body;

[0043] OPL (Outer Plexiform Layer): It is a loose fleshy structure. It is the synaptic site where the end balls of cones and rods connect with the dendrites of bipolar cells and the processes of horizontal cells. It is located below the ILM layer.

[0044] IS / OS (Inner and Outer Segments): The junction between the inner and outer segments of the photoreceptor.

[0045] OBRPE (Outer Boundary of Retinal Pigment Epithelia): the upper boundary of the retinal pigment epithelial cell area;

[0046] IBRPE (Inner Boundary of Retinal Pigment Epithelia): The lower boundary of the retinal pigment epithelium cell area.

[0047] Step 4, constructing a Mask2Former network model; the Mask2Former network model includes a backbone structure, a neck structure and a decoder head; the backbone structure adopts a Swin-transformer feature extraction model, the neck structure adopts a joint pyramid upsampling, the decoder head includes a pixel decoder and a Transformer decoder, the Transformer decoder includes N groups of decoding block groups, each group of decoding block groups includes three decoding blocks, namely a first decoding block, a second decoding block and a third decoding block;

[0048] In step 4, the backbone structure and the neck structure are used to obtain multi-level features; the pixel decoder is used for feature extraction; the Transformer decoder is used to generate prediction results and calculate the loss between the prediction results and the true target;

[0049] Among them, Mask2Former: a Transformer-based image segmentation model that supports semantic segmentation, instance segmentation, and panoptic segmentation tasks;

[0050] Swin-transformer: A Transformer-based image feature extraction model with hierarchical feature expression and a self-attention mechanism based on a shifted window, which aims to improve computational efficiency while maintaining high performance. It is used as the backbone structure in this paper.

[0051] JPU (Joint Pyramid Upsampling): The last three feature maps of the backbone are taken as input and fused into a high-resolution feature map, which is used as the neck structure in this invention.

[0052] Step 5: Input the preprocessed original OCT image dataset and the manually layered annotated image dataset into the Mask2Former network model for training to obtain a trained Mask2Former network model; wherein the specific training process is:

[0053] S5.1. After each OCT image in the preprocessed original OCT image data set is input into the Mask2Former network model, the backbone structure processes the OCT image and generates four sets of feature maps in succession. The size of each set of feature maps is half of the previous set of feature maps, and the number of channels is doubled. The backbone structure extracts the last three sets of feature maps and transmits them to the neck structure for fusion. The neck structure generates a set of synthetic feature maps so as to retain more detail information and reduce the loss caused by upsampling. The size of the synthetic feature map is equal to the size of the first set of feature maps generated by the backbone structure.

[0054] S5.2. Input the set of synthetic feature maps and the last two sets of feature maps generated by the backbone structure into the pixel decoder, and the pixel decoder processes each set of feature maps to generate corresponding multi-scale features, namely, the first multi-scale features, the second multi-scale features, and the third multi-scale features. All the multi-scale features are fused with the first set of feature maps generated by the backbone structure to generate mask features. At the same time, the first multi-scale features, the second multi-scale features, the third multi-scale features, and the mask features are input into the Transformer decoder, and trained by N sets of decoding block groups in the Transformer decoder. Here, N=9 is taken, so 9 rounds of training need to be completed. In each round of training, the Transformer decoder outputs the pixel filling map and mask corresponding to the OCT image described in step S5.1. ！ Features, and use a loss function to calculate the loss between the pixel filling image and the corresponding manually layered annotated image obtained in step S3; finally, a trained Mask2Former network model is obtained;

[0055] Furthermore, the preprocessed original OCT image dataset is input into the Mask2Former network model. For any OCT image in the preprocessed original OCT image dataset, the specific process of training by the 9 groups of decoding blocks in the Transformer decoder includes the following steps:

[0056] S5.21. In each round of training, the first multi-scale feature, the second multi-scale feature, and the third multi-scale feature are input into a current set of decoding blocks of the Transformer decoder, and the query vector is calculated by the first decoding block, the second decoding block, and the third decoding block in the current set of decoding blocks;

[0057] S5.22, the query vector, the first multi-scale feature, and the mask output by the previous set of decoding blocks ！ The feature is used as the input of the first decoding block of the current set of decoding blocks, and the first result is output after the cross attention, self-attention and feed-forward network layers in the first decoding block;

[0058] S5.23, using the first result and the second multi-scale feature as inputs of a second decoding block of the current set of decoding blocks, and outputting a second result after the cross attention, self-attention and feed-forward network layers in the second decoding block;

[0059] S5.24, using the second result and the third multi-scale feature as inputs of a third decoding block of the current set of decoding blocks, and outputting a third result after passing through the cross attention, self-attention and feedforward network layers in the third decoding block;

[0060] S5.25, multiplying the third result and the mask feature generated by the pixel decoder to generate a pixel filling map and a mask′ feature corresponding to the OCT image described in step S5.1;

[0061] S5.26. Use the mask′ feature obtained in step S5.25 as the input of the first decoding block of the next set of decoding block groups, and the next set of decoding block groups performs the next round of training until 9 rounds of training are completed.

[0062] Furthermore, the loss function is expressed as: Loss = λ1L cls +λ2L mask +λ3L Dice ;

[0063] Among them, λ1=0.2, λ2=0.5, λ3=0.5, L cls represents the cross entropy loss function based on Softmax; L mask Represents the cross entropy loss function based on Sigmoid; L Dice Represents the Dice loss function.

[0064] Step 6: Test the trained Mask2Former network model, input the OCT image to be layered into the trained Mask2Former network model, and the trained Mask2Former network model outputs the corresponding pixel filling map.

[0065] Step 7: Convert the pixel filling image obtained in step S6 into a layered image of the retinal area.

[0066] Furthermore, the specific process of converting the pixel filling image obtained in step S6 into the layered image of the retinal area includes the following steps:

[0067] Step S7.1, construct a 512×5 matrix;

[0068] Step S7.2: The pixel filling map obtained in step S6 includes six areas in the vertical space. The pixel filling map is traversed pixel by pixel in columns to find two different pixel points before and after the pixel filling map in the vertical space, which are regarded as the ordinates of the boundary positions and recorded in the matrix;

[0069] Step S7.3: Connect each pixel point in the same layer in sequence to obtain a layered image of the retinal area.

[0070] The present invention applies a semantic segmentation model to realize automatic stratification of OCT images, and divides the retinal area into five levels: ILM, ONL, IS / OS, OBRPE, and IBRPE, thereby solving the problem of low efficiency of manual stratification and laying a good foundation for further analysis and diagnosis of fundus diseases.

[0071] The present invention uses Swin-transformer as the backbone structure, JPU as the neck structure, Mask2Former as the decoder head, and uses a multi-attention mechanism. This technology combination has not been applied to retinal stratification in OCT images. Compared with the existing deep learning model, the current architecture uses an attention mechanism, which can adaptively focus on the key parts of the image, improve the accuracy of pixel segmentation, and thus reduce the error of retinal stratification. The present invention can stratify the retinal areas of AMD, DME and normal OCT images, and has better generalization characteristics.

Claims

1. A method for automatic retinal layering in OCT based on the Mask2Former model, characterized by: The method comprises the following steps: S1. constructing an original OCT image dataset, wherein the original OCT image dataset includes a plurality of images related to AMD symptoms, a plurality of images related to DME symptoms, and a plurality of asymptomatic images; S2, preprocessing the original OCT image data set to obtain a preprocessed original OCT image data set; S3, manually layering and annotating each original OCT image in the original OCT image data set to obtain a corresponding manually layered and annotated image, and forming a manually layered and annotated image data set; wherein the specific process of manually layering and annotating the original OCT image is: dividing the retinal area in the original OCT image into five layers from top to bottom, the five layers divide the space into six areas in the vertical direction, and using pixel points with different pixel values ​​to fill the six areas in turn to obtain a manually layered and annotated image; S4, constructing a Mask2Former network model; the Mask2Former network model includes a backbone structure, a neck structure and a decoder head; the backbone structure adopts a Swin-transformer feature extraction model, the neck structure adopts a joint pyramid upsampling, the decoder head includes a pixel decoder and a Transformer decoder, the Transformer decoder includes N groups of decoding block groups, each group of decoding block groups includes three decoding blocks, namely a first decoding block, a second decoding block and a third decoding block; S5. Input the preprocessed original OCT image dataset and the manually layered annotated image dataset into the Mask2Former network model for training to obtain a trained Mask2Former network model; wherein the specific training process is: S5.

1. After each OCT image in the preprocessed original OCT image data set is input into the Mask2Former network model, the backbone structure processes the OCT image and generates four sets of feature maps in succession. The size of each set of feature maps is half of the previous set of feature maps, and the number of channels is doubled. The backbone structure extracts the last three sets of feature maps and transmits them to the neck structure for fusion. The neck structure generates a set of synthetic feature maps, whose size is equal to the first set of feature maps generated by the backbone structure. S5.2, input the set of synthetic feature maps and the last two sets of feature maps generated by the backbone structure into the pixel decoder, and the pixel decoder processes each set of feature maps to generate corresponding multi-scale features, namely the first multi-scale features, the second multi-scale features and the third multi-scale features, and fuses all the multi-scale features with the first set of feature maps generated by the backbone structure to generate mask features; at the same time, input the first multi-scale features, the second multi-scale features, the third multi-scale features and the mask features into the Transformer decoder, and train them by N sets of decoding block groups in the Transformer decoder to complete N rounds of training; in each round of training, the Transformer decoder outputs the pixel filling map and mask′ features of the OCT image described in step S5.1, and uses the loss function to calculate the loss between the pixel filling map and the corresponding artificial layered annotation image obtained in step S3, to obtain the trained Mask2Former network model; S6, inputting the OCT image to be layered into the trained Mask2Former network model, and the trained Mask2Former network model outputs a corresponding pixel filling map; S7. Convert the pixel filling image obtained in step S6 into a layered image of the retinal area.

2. The method for automatic retinal layering in OCT based on the Mask2Former model according to claim 1, characterized in that: In step S2, the specific process of preprocessing the original OCT image data set is: unifying the size and format of all original OCT images in the original OCT image data set; and then standardizing each pixel point of each original OCT image, which is specifically expressed as: Among them, mean represents the average value of all pixels on the original OCT image, std represents the standard deviation of all pixels on the original OCT image, and P represents the pixel value of a pixel on the original OCT image.

3. The automatic retinal layering method in OCT based on the Mask2Former model according to claim 1 or 2, characterized in that: In step S5.2, the first multi-scale feature, the second multi-scale feature, and the third multi-scale feature are input into a Transformer decoder, and the specific process of training by the N groups of decoding blocks in the Transformer decoder includes the following steps: S5.

21. In each round of training, the first multi-scale feature, the second multi-scale feature, and the third multi-scale feature are input into a current set of decoding blocks of the Transformer decoder, and the query vector is calculated by the first decoding block, the second decoding block, and the third decoding block in the current set of decoding blocks; S5.22, taking the query vector, the first multi-scale feature and the mask! feature output by the previous set of decoding blocks as the input of the first decoding block of the current set of decoding blocks, and outputting the first result after the cross attention, self-attention and feed-forward network layers in the first decoding block; S5.23, using the first result and the second multi-scale feature as inputs of a second decoding block of the current set of decoding blocks, and outputting a second result after the cross attention, self-attention and feed-forward network layers in the second decoding block; S5.24, using the second result and the third multi-scale feature as inputs of a third decoding block of the current set of decoding blocks, and outputting a third result after passing through the cross attention, self-attention and feedforward network layers in the third decoding block; S5.25, multiplying the third result and the mask feature generated by the pixel decoder to generate the pixel filling map and mask′ feature of the OCT image described in step S5.1; S5.

26. Use the mask′ feature obtained in step S5.25 as the input of the first decoding block of the next set of decoding block groups, and the next set of decoding block groups performs the next round of training until N rounds of training are completed.

4. The method for automatic retinal layering in OCT based on the Mask2Former model according to claim 3, characterized in that: In step S5.2, the loss function is expressed as: Loss = λ1L cls +λ2L mask +λ3L Dice ; Among them, λ1, λ2 and λ3 are coefficients, L cls represents the cross entropy loss function based on Softmax; L mask Represents the cross entropy loss function based on Sigmoid; L Dice Represents the Dice loss function.

5. The method for automatic retinal layering in OCT based on the Mask2Former model according to claim 4, characterized in that: In step S7, the specific process of converting the pixel filling image obtained in step S6 into the layered image of the retinal area includes the following steps: Step S7.1, constructing an M×N matrix, wherein M represents the width of the corresponding OCT image to be layered, and N represents the number of layers of the retinal area, N=5; Step S7.2: The pixel filling map obtained in step S6 includes six areas in the vertical space. The pixel filling map is traversed pixel by pixel in columns to find two different pixel points before and after the pixel filling map in the vertical space, which are regarded as the ordinates of the boundary positions and recorded in the matrix; Step S7.3: Connect each pixel point in the same layer in sequence to obtain a layered image of the retinal area.

Images