Three-dimensional segmentation method and device for DME focus in human fundus OCT image

Abstract

The invention provides a three-dimensional segmentation method and device for a DME focus in a human fundus OCT image, and relates to the technical field of OCT medical image segmentation, and the method comprises the steps: obtaining an initial fundus optical coherence tomography image outputted by an optical coherence tomography imaging system; pre-training a three-dimensional deep learning neural network model which is used for segmenting a diabetic macular edema focus area in the fundus OCT image and is based on a transformer architecture; inputting the fundus OCT image into a trained three-dimensional deep learning neural network model to carry out real-time full-automatic three-dimensional segmentation of a DME focus area, and obtaining a stereoscopic three-dimensional focus image with good continuity in a space area; according to the method, the three-dimensional lesion image with better continuity in the space area can be obtained, a doctor is better helped to obtain three-dimensional information in the lesion, the manual segmentation burden of the doctor is relieved, meanwhile, the doctor is helped to analyze the morphology and change of the lesion more visually and clearly, and convenience is provided for clinical diagnosis of diabetic macular edema.

Description

Technical Field

[0001] The present application relates to the technical field of OCT medical image segmentation, and in particular to a three-dimensional segmentation method and device for DME lesions in human fundus OCT images. Background Art

[0002] Diabetic macular edema (DME) is a serious and common complication that can occur at any stage of diabetic retinopathy. Characterized by fluid accumulation in the macula due to increased vascular permeability, it is currently the leading cause of blindness and related functional impairment in working-age populations in most developed countries. DME can trigger progressive retinal dysfunction, leading to irreversible and permanent vision loss. To effectively prevent progression to blindness, quantitative assessment of the shape and size of DME lesions is crucial for appropriate diagnosis and treatment of DME.

[0003] Optical coherence tomography (OCT) imaging is a non-invasive three-dimensional imaging technology that is currently widely used to monitor the progression and assess damage of DME. Manual segmentation of DME lesion areas in fundus OCT images by professional ophthalmologists is time-consuming and inefficient, so automated and accurate segmentation methods are of great clinical significance. Currently, most segmentation methods for DME lesion areas in fundus OCT images are limited to two-dimensional layers, so they cannot fully utilize the three-dimensional segmentation information in OCT images. In addition, the segmentation results have poor continuity in the spatial layer and cannot accurately display the three-dimensional morphology of the lesions. Therefore, there is an urgent need for a three-dimensional fully automatic segmentation method for DME lesion area segmentation in fundus OCT images to fully utilize the three-dimensional information in OCT images, improve segmentation performance, and facilitate the diagnosis and monitoring of DME in clinical practice. Summary of the Invention

[0004] In order to address the shortcomings of the existing technology, the present application provides a three-dimensional segmentation method and device for DME lesions in human fundus OCT images, which can fully utilize the spatial information in the OCT images, improve the spatial continuity of the segmentation results, and achieve high-precision, fully automatic three-dimensional segmentation of DME lesions in human fundus OCT images.

[0005] To achieve the above objectives, the technical solutions adopted in this application are as follows:

[0006] In a first aspect, the present application provides a three-dimensional segmentation method for DME lesions in human fundus OCT images, comprising:

[0007] Acquire human fundus OCT images;

[0008] The preprocessed human fundus OCT image is input into a pre-trained three-dimensional deep learning neural network model based on the transformer architecture, and the OCT image is segmented to obtain the three-dimensional segmentation result of the DME lesion.

[0009] Furthermore, the three-dimensional deep learning neural network model includes an encoder based on a transformer structure and a decoder based on a convolutional structure. A skip-layer connection based on a 3D global channel-spatial attention mechanism is set between the encoder and the decoder, and a 3D multi-scale feature aggregator is set after the decoder to perform multi-scale feature aggregation on the output of each level of the decoder.

[0010] Specifically, the encoder has a 12-layer structure, including 12 transformer structures connected in sequence. The 0th layer output, 3rd layer output, 6th layer output, 9th layer output and 12th layer output of the encoder are connected to the 3D multi-scale non-local feature extractor. The output of the 3D multi-scale non-local feature extractor and the output of the 12th layer of the encoder are connected to the 3D semantic aggregator and then to the decoder. The output of the 0th layer of the encoder is the output of the OCT image data input to the encoder after passing through two convolutional layers, and each convolutional layer is also followed by a BN layer and a ReLU layer. The 3rd layer output, 6th layer output, 9th layer output and 12th layer output are the outputs of the 3rd transformer structure, the 6th transformer structure, the 9th transformer structure and the 12th transformer structure, respectively.

[0011] The decoder includes five layers, namely the first decoding layer, the second decoding layer, the third decoding layer, the fourth decoding layer and the fifth decoding layer, each decoding layer is composed of a single or multiple convolutional modules; the output of the 3D semantic aggregator is connected to the first decoding layer; the output of each decoding layer of the decoder is commonly connected to the 3D multi-scale feature aggregator;

[0012] In addition to the 12th layer of the encoder, there are also skip-layer connection modules between the 0th layer output, 3rd layer output, 6th layer output, and 9th layer output of the encoder and the corresponding layers of the decoder, namely the first skip-layer connection module, the second skip-layer connection module, the third skip-layer connection module, and the fourth skip-layer connection module.

[0013] Specifically, the transformer structure in the encoder includes a linear projection layer, an embedding patch layer, a first normalization layer, a multi-head attention layer, a first element-by-element addition layer, a second normalization layer, a multi-layer perceptron layer, and a second element-by-element addition layer, which are arranged in sequence. The output of the embedding patch layer is also input into the first element-by-element addition layer, and the input of the first element-by-element addition layer is also input into the second element-by-element addition layer.

[0014] Specifically, the first decoding layer is a single convolutional layer, the second decoding layer, the third decoding layer, the fourth decoding layer and the third decoding layer are all two convolutional layers, and each convolutional layer includes a BN layer and a ReLU layer.

[0015] Specifically, the first skip connection module is a 3D multi-scale global channel-spatial attention mechanism layer, the second skip connection module, the third skip connection module, and the fourth skip connection module have the same structure, each including a convolutional layer and a 3D multi-scale global channel-spatial attention mechanism layer; the third layer output, the sixth layer output, and the ninth layer output of the encoder are connected to the skip connection modules of the corresponding layers after passing through one or more 2×2×2 deconvolution layers;

[0016] The output of the 3D semantic aggregator is connected to the 3D multi-scale global channel-spatial attention mechanism layer of the fourth skip connection module through a 2×2×2 deconvolution operation, and the output of the 3D multi-scale global channel-spatial attention mechanism layer of the fourth skip connection module and the output of the 3D semantic aggregator are input into the second decoding layer after a channel connection operation; the output of the second decoding layer is connected to the 3D multi-scale global channel-spatial attention mechanism layer of the third skip connection module through a 2×2×2 deconvolution operation, and the output of the 3D multi-scale global channel-spatial attention mechanism layer of the third skip connection module and the output of the second decoding layer are input into the third decoding layer after a channel connection operation. layer; the output of the third decoding layer is connected to the 3D multi-scale global channel-spatial attention mechanism layer of the second skip connection module through a 2×2×2 deconvolution operation, and the output of the 3D multi-scale global channel-spatial attention mechanism layer of the second skip connection module and the output of the third decoding layer are input into the fourth decoding layer after performing a channel connection operation; the output of the fourth decoding layer is connected to the 3D multi-scale global channel-spatial attention mechanism layer of the first skip connection module through a 2×2×2 deconvolution operation, and the output of the 3D multi-scale global channel-spatial attention mechanism layer of the first skip connection module and the output of the fourth decoding layer are input into the fifth decoding layer after performing a channel connection operation.

[0017] Specifically, the 3D multi-scale feature aggregator includes a first feature aggregator, a second feature aggregator, a third feature aggregator and a fourth feature aggregator with the same structure. The structure of the feature aggregator includes a first sub-convolutional layer, a channel connection layer, a second sub-convolutional layer and a third sub-convolutional layer connected in sequence. The second sub-convolutional layer is also connected to a sigmoid layer, and the output of the sigmoid layer and the output of the first sub-convolutional layer are connected to the third sub-convolutional layer after an element-by-element multiplication operation; the channel connection layer of the first feature aggregator is used for the channel connection between the first decoding layer and the first sub-convolutional layer of the first feature aggregation layer, the channel connection layer of the second feature aggregator is used for the channel connection between the output of the first feature aggregator and the first sub-convolutional layer of the second feature aggregator, the channel connection layer of the third feature aggregator is used for the channel connection between the output of the second feature aggregator and the first sub-convolutional layer of the third feature aggregator, and the channel connection layer of the fourth feature aggregator is used for the channel connection between the output of the third feature aggregator and the first sub-convolutional layer of the fourth feature aggregator;

[0018] The output of the fourth feature aggregator passes through a convolutional layer and a sigmoid layer in sequence, and the output is the output of the entire three-dimensional deep learning neural network model based on the transformer architecture.

[0019] Preferably, the loss function of the three-dimensional deep learning neural network model is L total , L total By Tversky loss function T(α,β) and binary entropy loss function L BCE The formula is as follows:

[0020] L total =λT(α,β)+(1-λ)L BCE

[0021]

[0022] Among them, α, β are hyperparameters; λ is the weight; N is the total number of pixels in the image; p 0i is the probability that pixel i is a lesion in the prediction result; p 1i is the probability that pixel i is not a lesion in the prediction result; when the actual result of pixel i is a lesion, g 0i =1, indicating non-lesion g 0i is 0; when the true result of pixel i is a lesion, g 1i When it is 0, it is non-lesion. 1i is 1; y i Indicates the probability that pixel i is a lesion in the true result, p(y i ) is the probability that pixel i is a lesion in the prediction result.

[0023] In the second aspect, an embodiment of the present application also provides a three-dimensional segmentation device for DME lesions in human fundus OCT images, including a memory and a processor, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, it implements the above-mentioned method for three-dimensional segmentation of DME lesions in human fundus OCT images.

[0024] The beneficial effects of this application are:

[0025] The present application provides a three-dimensional segmentation method and device for DME lesions in human fundus OCT images. The method obtains an initial human fundus OCT image containing a DME lesion area output by an optical coherence tomography system; the initial human fundus OCT image containing the DME lesion area is input into a pre-trained three-dimensional segmentation network model based on a transformer architecture, and the initial human fundus OCT image containing the DME lesion area is three-dimensionally segmented to obtain a three-dimensional segmented image containing only the DME lesions. Compared with the two-dimensional segmentation method based on a pure convolutional structure, the pre-trained three-dimensional segmentation network model based on the transformer architecture can more fully utilize the spatial information in the OCT volume image, better capture the long-distance dependencies in the sequence, and fully automatically obtain a three-dimensional segmentation image of the DME lesion with better spatial continuity and higher accuracy. Applying this model to the clinical diagnosis and monitoring of DME can provide convenience and reliability in assisting professional doctors in diagnosis, reduce the probability of misdiagnosis or missed diagnosis, and improve diagnostic efficiency. It has good development prospects in the medical field. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0027] Figure 1 A schematic diagram of the application scenario structure of a transformer-based three-dimensional segmentation method for DME lesions in human fundus OCT images provided in an embodiment of the present application.

[0028] Figure 2 A flowchart of a three-dimensional segmentation method for DME lesions in human fundus OCT images based on a transformer architecture is provided in an embodiment of the present application.

[0029] Figure 3A flowchart of another transformer-based three-dimensional segmentation method for DME lesions in human fundus OCT images provided in an embodiment of the present application.

[0030] Figure 4 Schematic diagram of the initial network model structure of a transformer-based three-dimensional segmentation method for DME lesions in human fundus OCT images provided in an embodiment of the present application.

[0031] Figure 5 Schematic diagram of the specific structures of 3D-MsGCS, 3D-MsNLFE and 3D-SCA provided in the embodiments of the present application.

[0032] Figure 6 A schematic flow chart of a method for constructing a human fundus OCT image dataset provided in an embodiment of the present application.

[0033] Figure 7 This is a comparison chart of the segmentation samples in the test set of a three-dimensional segmentation method for DME lesions in human fundus OCT images based on the transformer architecture of the present invention.

[0034] Figure 8 A schematic diagram of the system structure for initial model training provided in an embodiment of the present application.

[0035] Figure 9 This is a structural block diagram of a three-dimensional segmentation device for DME lesions in human fundus OCT images provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of illustration and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can be implemented out of sequence, and steps without logical context can be reversed or implemented simultaneously. In addition, those skilled in the art, under the guidance of the contents of this application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0037] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0038] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0039] Figure 1 This is a schematic diagram of the application scenario structure of the three-dimensional segmentation method of DME lesions in fundus OCT images provided in the embodiment of the present application. Figure 1 As shown, the present application is applicable to the scenario of obtaining an OCT image of any sample, which may be a biological sample, etc. This scenario involves an optical coherence tomography (OCT) system and an electronic device. First, the fundus of any patient is scanned by the OCT system to output an initial fundus OCT image, and then the initial diseased fundus OCT image is input into the above-mentioned electronic device, which uses the trained three-dimensional segmentation network model based on the transformer architecture to perform three-dimensional segmentation on the DME lesions in the fundus OCT image, and finally obtain a three-dimensional target lesion area result.

[0040] Figure 2 This is a flow chart of a three-dimensional segmentation method for DME lesions in fundus OCT images provided in an embodiment of the present application. Figure 2 As shown, the method is applied to the above-mentioned electronic device, which can be, for example, a terminal device with computing and display capabilities such as a desktop computer or a notebook computer, or a server. The method includes:

[0041] S101: Acquire an initial human fundus OCT image output by an optical coherence tomography system and preprocess it. The preprocessing operations include but are not limited to cropping, flipping, rotating, and denoising operations.

[0042] Optionally, the OCT system may include a light source, a coupler, a reference mirror, a sample arm, a reference arm, and a spectrometer. The specific principle is: the system may use a broadband light source as the light source, for example, a broadband light source with a central wavelength of 850 nm and a full width at half maximum of 165 nm. The light emitted by the broadband light source passes through the coupler through the sample arm and the reference arm to irradiate the sample and the reference mirror respectively. The reflected light in the two optical paths converges in the coupler to generate an interference signal under certain conditions, and the interference signal is output to the spectrometer. The spectrometer transmits the detected interference signal to the computer, and the computer performs grayscale imaging and outputs an initial OCT image. The initial fundus OCT image contains a DME lesion area.

[0043] S102: Input the pre-processed initial fundus OCT image into a pre-trained three-dimensional segmentation network model based on the transformer architecture, and use the network model to perform three-dimensional segmentation on the initial fundus OCT image to obtain a three-dimensional DME lesion area image.

[0044] The trained three-dimensional segmentation network model based on the transformer architecture is combined with the OCT system in the above step S101. The initial fundus OCT image obtained in the above step S01 can be input into the trained three-dimensional segmentation network model based on the transformer architecture. The trained three-dimensional segmentation network model based on the transformer architecture is used to perform three-dimensional segmentation on the initial fundus OCT image to obtain a three-dimensional DME lesion area.

[0045] This embodiment obtains an initial fundus optical coherence layer image output by an optical coherence tomography system; inputs the initial fundus optical coherence tomography image into a pre-trained three-dimensional segmentation network model based on a transformer architecture, and performs three-dimensional segmentation on the initial fundus optical coherence tomography image by the transformer architecture-based three-dimensional segmentation network model to obtain a three-dimensional segmentation result of the DME lesion area. Compared with the two-dimensional segmentation method based on a pure convolutional structure, the pre-trained three-dimensional segmentation network model based on the transformer architecture can more fully utilize the spatial information in the OCT volume image, better capture the long-distance dependencies in the sequence, and fully automatically obtain a three-dimensional segmentation result of the DME lesion with better spatial continuity and higher accuracy.

[0046] Figure 4 This is a schematic diagram of the initial network model structure of a three-dimensional segmentation method for DME lesions in human fundus OCT images based on a transformer architecture provided in an embodiment of the present application. Figure 4As shown, the three-dimensional deep learning neural network model includes an encoder based on a transformer structure and a decoder based on a convolutional structure, with a jump connection between the encoder and the decoder through a multi-scale global channel-spatial attention mechanism (3D-MsGCS); at the network bottleneck, a multi-scale non-local feature extractor (3D-MsNLFE) and a semantic aggregator (3D-SCA) are connected; at the end of the network, the output of each level of the decoder is passed through a multi-scale feature aggregator (3D-MsFA). The human fundus OCT image is input into the pre-trained three-dimensional deep learning neural network model, the encoder extracts the features in the input image, and then its output is sequentially upsampled and transmitted to the decoder of the model through a 3D global channel-spatial attention mechanism with a jump connection between the encoder and the decoder. Finally, the output of each level of the decoder is subjected to multi-scale feature aggregation to obtain a three-dimensional segmentation result of the DME lesion area in the target fundus OCT image.

[0047] Furthermore, the encoder has a 12-layer structure, including 12 transformer structures connected in sequence. The 0th layer output, 3rd layer output, 6th layer output, 9th layer output and 12th layer output of the encoder are jointly connected to the 3D multi-scale non-local feature extractor, and the output of the 3D multi-scale non-local feature extractor and the output of the 12th layer of the encoder are jointly connected to the 3D semantic aggregator and then connected to the decoder; the output of the 0th layer of the encoder is the output of the OCT image data input to the encoder after passing through 2 convolution layers and each convolution layer is also followed by a BN layer and a Relu layer. The 3rd layer output, 6th layer output, 9th layer output and 12th layer output are the outputs of the 3rd transformer structure, the 6th transformer structure, the 9th transformer structure and the 12th transformer structure respectively.

[0048] The decoder consists of 5 layers, namely the first decoding layer, the second decoding layer, the third decoding layer, the fourth decoding layer and the fifth decoding layer, each decoding layer is composed of a single or multiple convolution modules; the output of the 3D semantic aggregator is connected to the first decoding layer; the input image layer can be regarded as the 0th layer, not f0 in the figure, and the output of the third transformer structure is not shown in the figure. f3, f6, f9, f 12 The meaning of is similar; the output of each decoding layer of the decoder is commonly connected to the 3D multi-scale feature aggregator.

[0049] In addition to the 12th layer of the encoder, skip-layer connection modules are provided between the encoder's 0th, 3rd, 6th, and 9th layer outputs and the corresponding layers of the decoder. These are the first, second, third, and fourth skip-layer connection modules, respectively. The 9th layer output of the encoder corresponds to the second decoding layer, the 6th layer output corresponds to the third decoding layer, the 3rd layer output corresponds to the fourth decoding layer, and the encoder's input image layer corresponds to the fifth decoding layer.

[0050] The transformer structure in the encoder includes a linear projection layer, an embedding patch layer, a first normalization layer, a multi-head attention layer, a first element-by-element addition layer, a second normalization layer, a multi-layer perceptron layer, and a second element-by-element addition layer, which are arranged in sequence. The output of the embedding patch layer is also input into the first element-by-element addition layer, and the input of the first element-by-element addition layer is also input into the second element-by-element addition layer.

[0051] The first decoding layer is a single convolutional layer, the second decoding layer, the third decoding layer, the fourth decoding layer and the third decoding layer are two convolutional layers, and each convolutional layer includes a BN layer and a ReLU layer.

[0052] The first skip connection module is a 3D multi-scale global channel-spatial attention mechanism layer, and the second skip connection module, the third skip connection module and the fourth skip connection module have the same structure, each including a convolution layer and a 3D multi-scale global channel-spatial attention mechanism layer; the 3rd layer output, the 6th layer output and the 9th layer output of the encoder are connected to the skip connection module of the corresponding layer after passing through one or more 2×2×2 deconvolution layers; the 3rd layer output of the encoder is connected to three 2×2×2 deconvolution layers, the 6th layer output is connected to two 2×2×2 deconvolution layers, and the 9th layer output is connected to one 2×2×2 deconvolution layer.

[0053] The output of the 3D semantic aggregator is connected to the 3D multi-scale global channel-spatial attention mechanism layer of the fourth skip connection module through a 2×2×2 deconvolution operation, and the output of the 3D multi-scale global channel-spatial attention mechanism layer of the fourth skip connection module and the output of the 3D semantic aggregator are input into the second decoding layer after a channel connection operation; the output of the second decoding layer is connected to the 3D multi-scale global channel-spatial attention mechanism layer of the third skip connection module through a 2×2×2 deconvolution operation, and the output of the 3D multi-scale global channel-spatial attention mechanism layer of the third skip connection module and the output of the second decoding layer are input into the third decoding layer after a channel connection operation. layer; the output of the third decoding layer is connected to the 3D multi-scale global channel-spatial attention mechanism layer of the second skip connection module through a 2×2×2 deconvolution operation, and the output of the 3D multi-scale global channel-spatial attention mechanism layer of the second skip connection module and the output of the third decoding layer are input into the fourth decoding layer after performing a channel connection operation; the output of the fourth decoding layer is connected to the 3D multi-scale global channel-spatial attention mechanism layer of the first skip connection module through a 2×2×2 deconvolution operation, and the output of the 3D multi-scale global channel-spatial attention mechanism layer of the first skip connection module and the output of the fourth decoding layer are input into the fifth decoding layer after performing a channel connection operation.

[0054] The 3D multi-scale feature aggregator (3D-MsFA) includes a first feature aggregator, a second feature aggregator, a third feature aggregator and a fourth feature aggregator with the same structure. The structure of the feature aggregator includes a first sub-convolutional layer, a channel connection layer, a second sub-convolutional layer and a third sub-convolutional layer connected in sequence. The second sub-convolutional layer is also connected to a sigmoid layer. The output of the sigmoid layer and the output of the first sub-convolutional layer are connected to the third sub-convolutional layer after an element-by-element multiplication operation; the channel connection layer of the first feature aggregator is used for channel connection between the first decoding layer and the first sub-convolutional layer of the first feature aggregation layer, the channel connection layer of the second feature aggregator is used for channel connection between the output of the first feature aggregator and the first sub-convolutional layer of the second feature aggregator, the channel connection layer of the third feature aggregator is used for channel connection between the output of the second feature aggregator and the first sub-convolutional layer of the third feature aggregator, and the channel connection layer of the fourth feature aggregator is used for channel connection between the output of the third feature aggregator and the first sub-convolutional layer of the fourth feature aggregator.

[0055] The output of the fourth feature aggregator passes through a convolutional layer and a sigmoid layer in sequence, and the output is the output of the entire three-dimensional deep learning neural network model based on the transformer architecture. Here, the convolutional layer compresses the channel to 1.

[0056] 3D-MsGCS, 3D-MsNLFE, and 3D-SCA all adopt a computational strategy containing a global learnable heat map.

[0057] Specifically, if Figure 5 As shown, the specific operation in the 3D global channel-spatial attention mechanism layer (3D-MsGCS) is: receiving the feature map f from the low-scale up (the convolutional layer output in the fourth skip-connection module, the convolutional layer output in the third skip-connection module, the convolutional layer output in the second skip-connection module, or the convolutional layer output in the first skip-connection module) and the feature map f from the high-scale H (output of the 3D semantic aggregation module, output of the second decoding layer, output of the third decoding layer or output of the fourth decoding layer), firstly fuse the high- and low-scale feature maps through channel connection, and then capture the multi-semantic global information of the channel dimension through 3×3×3 convolution operation. The 3×3×3 convolution operation here can be performed once or twice. Each convolution operation is followed by a BN layer and a ReLU layer, and then the high- and low-scale fused feature map f is obtained after the channel normalization operation. M , the high- and low-scale fusion feature map f M and the global learnable heatmap F which is formed by summing the learning vectors in vertical, horizontal and depth (X, Y and Z) directions G Multiply to capture the multi-semantic global information of the spatial dimension. Finally, apply batch normalization and sigmoid activation function to obtain the attention weight F att , the attention weight F att With the input high-scale feature map f H Multiply them together to get the output f of the 3D global channel-spatial attention mechanism layer Final .

[0058] like Figure 5 As shown, the specific operation of the 3D non-local feature extractor (3D-MsNLFE) is as follows: receiving feature maps output from the transformer structure of different levels of the encoder (encoder layer 0, layer 3, layer 6, layer 9, corresponding to Figure 4 f0, f3, f6 and f9), each of which first undergoes a 3×3×3 convolution, and then performs element-by-element superposition to obtain a multi-scale feature map F A , the multi-scale feature map F A Compared with the feature map from the highest level of the encoder (encoder layer 12, corresponding to Figure 4 f in 12 ) Input multi-head attention mechanism, where the multi-scale feature map F AAs the query of the multi-head attention mechanism, the feature map from the lowest level of the encoder is used as the key and value of the multi-head attention mechanism; when calculating the attention weight, a calculation strategy containing a global learnable heat map is adopted. Specifically, the feature map F is obtained by multiplying the value and the query. E , the query is combined with the global learnable heat map F formed by adding the learning vectors in the vertical, horizontal and depth (X, Y and Z) directions G Multiply to get the feature map F EP , the feature map F E and feature map F EP After adding, the attention weight F is obtained through the Softmax activation function att , the attention weight F att Multiply by Key to get feature map F M , the feature map F M and the feature map f at the highest level of the encoder 12 Add together to get the output F of the 3D non-local feature extractor En , the parameter γ in the figure is used to adjust the feature map F M The weight of .

[0059] like Figure 5 As shown, the specific operation in the 3D semantic aggregation layer (3D-SCA) is: receiving the feature map F output by the 3D non-local feature extractor En And the feature map f output by the 12th layer of the encoder 12 , which is input into the multi-head attention mechanism, where the feature map f output by the 12th layer of the encoder 12 As the query of the multi-head attention mechanism, the feature map F output by the 3D non-local feature extractor En As the key and value of the multi-head attention mechanism; when calculating the attention weight, a calculation strategy containing a global learnable heat map is adopted. Specifically, the feature map F is obtained by multiplying the value and the query. ED , the query is combined with the global learnable heat map F formed by adding the learning vectors in the vertical, horizontal and depth (X, Y and Z) directions G Multiply to get the feature map F EPD , the feature map F ED and feature map F EPD After adding, the attention weight F is obtained through the Softmax activation function Datt , the attention weight F Datt Multiply by Value to get the feature map F MD , the feature map F MD and the feature map f at the highest level of the encoder 12 Add together to get the output f of the 3D non-local feature extractor Final, the parameter γ in the figure is used to adjust the feature map F M The weight of .

[0060] The present invention uses a multi-scale global channel-spatial attention mechanism (3D-MsGCS) jump connection between the encoder and decoder to improve the model's ability to learn multi-semantic global context features in the channel and spatial dimensions, while suppressing irrelevant local features. The multi-scale non-local feature extractor (3D-MsNLFE) and semantic aggregator (3D-SCA) connected at the end of the encoder can alleviate the impact of changes in the size and shape of DME lesions and shot noise in OCT images on the segmentation results. At the end of the decoder, the output of each level of the decoder is passed through a multi-scale feature aggregator (3D-MsFA) to mine more feature information and improve the accuracy of segmentation.

[0061] The following examples illustrate the training process of the above-mentioned three-dimensional deep learning neural network model based on the transformer architecture.

[0062] Figure 3 This is a flow chart of another method for 3D segmentation of DME lesions in human fundus OCT images based on transformer architecture provided in an embodiment of the present application. Figure 3 As shown, in the above step S102, the initial human fundus OCT image is input into the pre-trained three-dimensional deep learning neural network model based on the transformer architecture. Before the three-dimensional deep learning neural network model based on the transformer architecture performs three-dimensional segmentation on the initial human fundus OCT image, the process of training the three-dimensional deep learning neural network model based on the transformer architecture may include:

[0063] S201. Construct a sample dataset. The sample dataset includes multiple image pairs, each of which includes an original fundus OCT image and a manually annotated lesion image. The fundus OCT image dataset is preprocessed; the preprocessing includes, but is not limited to, cropping, flipping, resizing, and contrast adjustments. Finally, the preprocessed dataset is divided into a training set, a test set, and a validation set in a ratio of 8:1:1.

[0064] S202: Construct an initial model. The specific structure of the model has been described in detail above.

[0065] S203, inputting the fundus OCT images of the training set into the three-dimensional deep learning neural network model to obtain a segmentation result;

[0066] S204, calculating the segmentation loss based on the segmentation result and the label value; the loss function of the three-dimensional deep learning neural network model is L total , Ltotal By Tversky loss function T(α,β) and binary entropy loss function L BCE The formula is as follows:

[0067] L total =λT(α,β)+(1-λ)L BCE

[0068]

[0069] Among them, α, β are hyperparameters; λ is the weight; N is the total number of pixels in the image; p 0i is the probability that pixel i is a lesion in the prediction result; p 1i is the probability that pixel i is not a lesion in the prediction result; when the actual result of pixel i is a lesion, g 0i =1, indicating non-lesion g 0i is 0; when the true result of pixel i is a lesion, g 1i When it is 0, it is non-lesion. 1i is 1; y i Indicates the probability that pixel i is a lesion in the true result, p(y i ) is the probability that pixel i is a lesion in the prediction result.

[0070] By adjusting the hyperparameters α and β of the Tversky loss function, we can control the balance between false positives, true negatives, and true positives and false negatives. Adjusting the hyperparameters α and β based on the ratio of false positives to false negatives in the predicted results can improve segmentation accuracy and reduce outliers. The binary cross entropy loss function can further improve segmentation by eliminating data imbalance caused by a large difference between the lesion and background volumes.

[0071] Backpropagate and update the network parameters according to the segmentation loss. When the segmentation loss is minimized, save the network weight parameters.

[0072] S205. Calculate the Dice coefficient of the segmentation result on the validation set. When the Dice coefficient is greater than the Dice coefficient calculated last time, save the optimal model parameters to obtain a trained three-dimensional deep learning neural network model.

[0073] S206. Input the test set data into the trained three-dimensional deep learning neural network model, and the resulting image is the three-dimensional segmentation result of the fundus DME lesion.

[0074] Figure 6A flow chart of a method for constructing a sample data set provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the above step S201 constructs a sample data set, which may include:

[0075] S301 , scanning any sample using a modified optical coherence tomography system to obtain an initial OCT image corresponding to the sample object, and combining each initial OCT image with a manually annotated reference image into an image pair.

[0076] S302. Each initial fundus OCT image is used as the initial fundus OCT image in the image pair (referred to as the initial three-dimensional volume image). According to the selected fundus DME target segmentation area, the DME target segmentation area on each initial image is manually marked as the reference image in the image pair, and each initial three-dimensional volume image is combined with the corresponding true value image into an image pair.

[0077] Optionally, for each sample, multiple corresponding initial three-dimensional volume images and the true value image corresponding to the initial object are selected and combined to obtain an image pair. By using this method, other initial three-dimensional volume images are combined with their corresponding reference images to obtain multiple image pairs.

[0078] For example, if there are 100 initial three-dimensional volume images corresponding to any DME patient sample, then the sample may have 100 image pairs.

[0079] S303, cropping the initial OCT image; since most areas of the initial OCT image have no valid information, the initial three-dimensional volume image with a size of 512×1044×512 pixels (X×Y×Z) is cropped into a three-dimensional volume image with a size of 224×224×512 pixels (X×Y×Z); in order to fully utilize the computing power of the GPU, each cropped image is divided into 8 three-dimensional volume images with a size of 224×224×64 pixels (X×Y×Z) along the Z-axis; then the cropped OCT images are divided into a training set, a validation set, and a test set, where the ratio of the data in the training set, validation set, and test set is 8:1:1; data augmentation is performed on the data in the training set and validation set, that is, they are randomly rotated 90° and repeated 0 to 3 times.

[0080] For example, there are a total of 100 patient image pairs, and the image pairs used for network model training and testing are 80 and 10 respectively, and the image pairs used to verify the practicality of the network model are 10.

[0081] Figure 7The three-dimensional segmentation method for DME lesions in human fundus OCT images based on the transformer architecture provided in the embodiment of this application is compared in the test set segmentation sample diagram, as shown in the figure. Figure 7 As shown in the figure, from left to right are the original fundus OCT image (left), the 3D lesion image manually annotated by an ophthalmologist (center), and the predicted 3D lesion image (right). It can be seen intuitively that these 3D prediction results can not only be close to the actual morphology of the lesion, but also retain surface details, which is beneficial for DME diagnosis.

[0082] Figure 8 A schematic diagram of the system structure for initial model training provided in an embodiment of the present application.

[0083] Figure 9 This is a structural block diagram of a three-dimensional segmentation device 300 for DME lesions in fundus OCT images provided in an embodiment of the present application. Figure 8 As shown, the device may include: a processor 301 and a memory 302.

[0084] Optionally, a bus 303 may also be included, wherein the memory 302 is used to store machine-readable instructions executable by the processor 301. When the electronic device 300 is running, the processor 301 communicates with the memory 302 through the bus 303. When the machine-readable instructions are executed by the processor 601, the method steps in the above method embodiment are performed.

[0085] An embodiment of the present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method steps of the above-mentioned three-dimensional deep learning neural network model method embodiment based on the transformer architecture are executed.

[0086] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0087] In addition, the functional units in the various embodiments of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the various embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0088] The above is only a specific implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the protection scope of the present application.

Claims

1. A three-dimensional segmentation method for DME lesions in human fundus OCT images, characterized in that: include: Acquire human fundus OCT images; The preprocessed human fundus OCT image is input into a pre-trained three-dimensional deep learning neural network model based on the transformer architecture, and the OCT image is segmented to obtain the three-dimensional segmentation result of the DME lesion.

2. The three-dimensional segmentation method for DME lesions in human fundus OCT images according to claim 1, characterized in that: The three-dimensional deep learning neural network model includes an encoder based on a transformer structure and a decoder based on a convolutional structure. A skip-layer connection based on a 3D global channel-spatial attention mechanism is set between the encoder and the decoder. A 3D multi-scale feature aggregator is set after the decoder to perform multi-scale feature aggregation on the output of each level of the decoder.

3. The three-dimensional segmentation method for DME lesions in human fundus OCT images according to claim 2, characterized in that: The encoder has a 12-layer structure, including 12 transformer structures connected in sequence. The output of the 0th layer, the 3rd layer, the 6th layer, the 9th layer and the 12th layer of the encoder are connected to the 3D multi-scale non-local feature extractor. The output of the 3D multi-scale non-local feature extractor and the output of the 12th layer of the encoder are connected to the 3D semantic aggregator and then to the decoder. The output of the 0th layer of the encoder is the output of the OCT image data input to the encoder after passing through two convolutional layers, and each convolutional layer is further provided with a BN layer and a ReLU layer. The output of the 3rd layer, the output of the 6th layer, the output of the 9th layer and the output of the 12th layer are the outputs of the 3rd transformer structure, the 6th transformer structure, the 9th transformer structure and the 12th transformer structure respectively. The decoder includes five layers, namely the first decoding layer, the second decoding layer, the third decoding layer, the fourth decoding layer and the fifth decoding layer, each decoding layer is composed of a single or multiple convolutional modules; the output of the 3D semantic aggregator is connected to the first decoding layer; the output of each decoding layer of the decoder is commonly connected to the 3D multi-scale feature aggregator; In addition to the 12th layer of the encoder, there are also skip-layer connection modules between the 0th layer output, 3rd layer output, 6th layer output, and 9th layer output of the encoder and the corresponding layers of the decoder, namely the first skip-layer connection module, the second skip-layer connection module, the third skip-layer connection module, and the fourth skip-layer connection module.

4. The three-dimensional segmentation method for DME lesions in human fundus OCT images according to claim 3, characterized in that: The transformer structure in the encoder includes a linear projection layer, an embedding patch layer, a first normalization layer, a multi-head attention layer, a first element-by-element addition layer, a second normalization layer, a multi-layer perceptron layer, and a second element-by-element addition layer, which are arranged in sequence. The output of the embedding patch layer is also input into the first element-by-element addition layer, and the input of the first element-by-element addition layer is also input into the second element-by-element addition layer.

5. The three-dimensional segmentation method for DME lesions in human fundus OCT images according to claim 3, characterized in that: The first decoding layer is a single convolutional layer, the second decoding layer, the third decoding layer, the fourth decoding layer and the third decoding layer are two convolutional layers, and each convolutional layer includes a BN layer and a ReLU layer.

6. The three-dimensional segmentation method for DME lesions in human fundus OCT images according to claim 3, characterized in that: The first skip connection module is a 3D multi-scale global channel-spatial attention mechanism layer. The second, third, and fourth skip connection modules have the same structure, each including a convolutional layer and a 3D multi-scale global channel-spatial attention mechanism layer. The third, sixth, and ninth layer outputs of the encoder are connected to the skip connection modules of the corresponding layers after passing through one or more 2×2×2 deconvolution layers. The output of the 3D semantic aggregator is connected to the 3D multi-scale global channel-spatial attention mechanism layer of the fourth skip connection module through a 2×2×2 deconvolution operation, and the output of the 3D multi-scale global channel-spatial attention mechanism layer of the fourth skip connection module and the output of the 3D semantic aggregator are input into the second decoding layer after a channel connection operation; the output of the second decoding layer is connected to the 3D multi-scale global channel-spatial attention mechanism layer of the third skip connection module through a 2×2×2 deconvolution operation, and the output of the 3D multi-scale global channel-spatial attention mechanism layer of the third skip connection module and the output of the second decoding layer are input into the third decoding layer after a channel connection operation. layer; the output of the third decoding layer is connected to the 3D multi-scale global channel-spatial attention mechanism layer of the second skip connection module through a 2×2×2 deconvolution operation, and the output of the 3D multi-scale global channel-spatial attention mechanism layer of the second skip connection module and the output of the third decoding layer are input into the fourth decoding layer after performing a channel connection operation; the output of the fourth decoding layer is connected to the 3D multi-scale global channel-spatial attention mechanism layer of the first skip connection module through a 2×2×2 deconvolution operation, and the output of the 3D multi-scale global channel-spatial attention mechanism layer of the first skip connection module and the output of the fourth decoding layer are input into the fifth decoding layer after performing a channel connection operation.

7. The three-dimensional segmentation method for DME lesions in human fundus OCT images according to claim 3, characterized in that: The 3D multi-scale feature aggregator includes a first feature aggregator, a second feature aggregator, a third feature aggregator and a fourth feature aggregator with the same structure. The structure of each feature aggregator includes a first sub-convolutional layer, a channel connection layer, a second sub-convolutional layer and a third sub-convolutional layer connected in sequence. The second sub-convolutional layer is also connected to a sigmoid layer. The output of the sigmoid layer and the output of the first sub-convolutional layer are connected to the third sub-convolutional layer after an element-by-element multiplication operation; the channel connection layer of the first feature aggregator is used for channel connection between the first decoding layer and the first sub-convolutional layer of the first feature aggregation layer, the channel connection layer of the second feature aggregator is used for channel connection between the output of the first feature aggregator and the first sub-convolutional layer of the second feature aggregator, the channel connection layer of the third feature aggregator is used for channel connection between the output of the second feature aggregator and the first sub-convolutional layer of the third feature aggregator, and the channel connection layer of the fourth feature aggregator is used for channel connection between the output of the third feature aggregator and the first sub-convolutional layer of the fourth feature aggregator; The output of the fourth feature aggregator passes through a convolutional layer and a sigmoid layer in sequence, and the output is the output of the entire three-dimensional deep learning neural network model based on the transformer architecture.

8. The three-dimensional segmentation method for DME lesions in human fundus OCT images according to claim 3, characterized in that: The loss function of the three-dimensional deep learning neural network model is L total , L total By Tversky loss function T(α,β) and binary entropy loss function L BCE The formula is as follows: L total =λT(α,β)+(1-λ)L BCE Among them, α, β are hyperparameters; λ is the weight; N is the total number of pixels in the image; p 0i is the probability that pixel i is a lesion in the prediction result; p 1i is the probability that pixel i is not a lesion in the prediction result; when the actual result of pixel i is a lesion, g 0i =1, indicating non-lesion g 0i is 0; when the true result of pixel i is a lesion, g 1i When it is 0, it is non-lesion. 1i is 1; y i Indicates the probability that pixel i is a lesion in the true result, p(y i ) is the probability that pixel i is a lesion in the prediction result.

9. A three-dimensional segmentation device for DME lesions in human fundus OCT images, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, the method for three-dimensional segmentation of DME lesions in human fundus OCT images as described in any one of claims 1 to 8 is implemented.

Images