A method for converting FFA imaging images based on deep learning-based OCT and OCTA imaging.

Abstract

This invention provides a method for converting OCT and OCTA imaging into FFA angiography images based on deep learning, belonging to the field of medical image processing technology. The method includes the following steps: S1: acquiring OCT and OCTA images of the target location of the patient's eye; S2: layering the OCT B-Scan to obtain OCT Enface images and OCTA Enface images for each layer; S3: acquiring FFA angiography of the target location of the patient's eye, registering the OCT Enface images and OCTA Enface images with any frame of the FFA angiography, and recording the injection volume of contrast agent and the time since the initial injection; S4: preprocessing the input path of the neural network to obtain a feature map, and performing feature fusion on the feature map. This method can enhance angiography of the patient's fundus vessels.

Description

Technical Field

[0001] This invention relates to the field of medical image processing technology, and more specifically, to a method for converting OCT and OCTTA imaging into FFA angiography images based on deep learning. Background Technology

[0002] Fundus photography is a widely used diagnostic procedure in ophthalmology. The blood vessels in the fundus are the only blood vessels in the human body that can be directly observed through the skin surface. Changes in the optic nerve, retina, and their blood vessels in the fundus not only greatly aid in the diagnosis and treatment of eye diseases themselves, but also provide significant guidance for the examination of conditions such as cerebral infarction, cerebral hemorrhage, cerebral arteriosclerosis, brain tumors, diabetes, kidney disease, and hypertension.

[0003] Optical coherence tomography (OCT) is a non-contact, non-invasive, high-resolution optical tomographic imaging technique for living tissue. While similar in principle to ultrasound tomography, OCT uses broadband near-infrared light as its detection medium, leveraging the low coherence of light to accurately detect the delay of reflected light waves at different depths within the tissue. This allows for non-contact acquisition of tomographic information about the tissue's structure, achieving a resolution approaching that of micrometers, comparable to microscopic observation of pathological tissue sections.

[0004] FFA angiography is the standard in ophthalmology, but it is invasive. OCT and OCTA imaging are non-invasive, but there is still room for improvement in observing small hemangiomas. Summary of the Invention

[0005] The problem addressed by this invention is how to enhance angiographic imaging of the blood vessels in the patient's fundus.

[0006] To address the aforementioned problems, this invention provides a method for converting OCT and OCTA imaging into FFA angiographic images based on deep learning, comprising the following steps:

[0007] S1: Acquire OCT and OCTA images of the target location on the patient's eye;

[0008] S2: Divide the OCT B-Scan into layers to obtain the OCT Enface image and OCTA Enface image of each layer;

[0009] S3: Acquire FFA angiography of the target location of the patient's eye, register any frame of the FFA angiography with the OCT Enface image and OCTA Enface image, and record the injection volume of contrast agent and the time since the initial injection.

[0010] S4: Preprocess the input path of the neural network to obtain feature maps, and then perform feature fusion on the feature maps;

[0011] S5: Input the fused feature map into the Decoder and upsample it step by step. After the feature map is upsampled, it is fused with the feature map of the Enface image in the Encoder through channel concatenation. After multiple upsampling to restore the original resolution, the output is an angiogram of the same size as the input Enface image.

[0012] In the above method, OCT and OCTA images are acquired using equipment. The OCT B-Scan is then layered to obtain OCT and OCTA Enface images for each layer. Simultaneously, an FFA (fine line aspiration) video of the same area on the patient needs to be acquired. Registration is then performed between the OCT / OCTA Enface images and any specific frame of the FFA image; registration methods include rigid and non-rigid registration.

[0013] Furthermore, the registration method in step S3 includes rigid registration and non-rigid registration, and a training pair is formed through registration;

[0014] OCT and OCTA are the inputs, and FFA imaging is the output.

[0015] Furthermore, in step S3, the imaging of a set of FFAs includes multiple frames, forming a multi-frame training pair, with each training pair corresponding to a different time.

[0016] Furthermore, the input path in step S4 includes:

[0017] Three-dimensional volumetric data of OCT and OCTA, two-dimensional projection image data of OCT Enface and OCTA Enface, and contrast agent injection volume and time from initial injection in the neck of the U-Net input.

[0018] Furthermore, the preprocessing in step S4 includes:

[0019] S41: Input the 3D volume data to obtain the N*C*H*W*D feature map;

[0020] S42: Compress the original feature map into an N*C'*H*W target feature map by using 1x1 convolution in the depth direction or average pooling or max pooling in the depth direction.

[0021] S43: After unifying the number of channels by performing a 1x1 convolution between the target feature map and the feature map of the Enface image, add them together for fusion processing.

[0022] Furthermore, the preprocessing in step S4 includes:

[0023] S44: The features fused in step S43 are processed by multiple two-dimensional convolutional layers, pooling layers, and normalization layers to obtain the feature map of the neck.

[0024] S45: Integrate information on injection time and contrast agent injection volume into the feature map of the U-Net neck;

[0025] S46: The injection time and contrast agent injection volume are passed through a Linear layer and projected into a feature map with the same width and height as the feature map. This feature map is then fused with the feature map of the neck through channel stitching.

[0026] The present invention employing the above technical solution has the following beneficial effects:

[0027] This invention can generate realistic FFA images based on OCT and OCTA images through neural network conversion, thereby achieving enhanced angiographic imaging of the patient's fundus vessels without invasive procedures. Attached Figure Description

[0028] Figure 1 The method flow for converting OCT and OCTA imaging into FFA angiographic images based on deep learning provided in this embodiment of the invention. Figure 1 ;

[0029] Figure 2 The method flow for converting OCT and OCTA imaging into FFA angiographic images based on deep learning provided in this embodiment of the invention. Figure 2 ;

[0030] Figure 3 This is a schematic diagram of the preprocessing process in the method for converting OCT and OCT imaging into FFA angiographic images based on deep learning, as provided in an embodiment of the present invention. Detailed Implementation

[0031] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0032] The following are specific embodiments of the present invention, which are described in conjunction with the accompanying drawings. However, the present invention is not limited to these embodiments.

[0033] Example

[0034] This embodiment provides a method for converting OCT and OCTA imaging into FFA angiography images based on deep learning, such as... Figure 1 and Figure 2 As shown, this method includes the following steps:

[0035] S1: Acquire OCT and OCTA images of the target location on the patient's eye;

[0036] S2: Divide the OCT B-Scan into layers to obtain the OCT Enface image and OCTA Enface image of each layer;

[0037] S3: Acquire FFA angiography of the target location of the patient's eye, register any frame of the FFA angiography with the OCT Enface image and OCTA Enface image, and record the injection volume of contrast agent and the time since the initial injection.

[0038] S4: Preprocess the input path of the neural network to obtain feature maps, and then perform feature fusion on the feature maps;

[0039] S5: Input the fused feature map into the Decoder and upsample it step by step. After the feature map is upsampled, it is fused with the feature map of the Enface image in the Encoder through channel concatenation. After multiple upsampling to restore the original resolution, the output is an angiogram of the same size as the input Enface image.

[0040] Specifically, the registration methods in step S3 include rigid registration and non-rigid registration, which form a training pair.

[0041] OCT and OCTA are the inputs, and FFA imaging is the output.

[0042] Specifically, in step S3, a set of FFA imaging contains multiple frames, forming a multi-frame training pair, with each training pair corresponding to a different time.

[0043] Specifically, the input path in step S4 includes:

[0044] Three-dimensional volumetric data of OCT and OCTA, two-dimensional projection image data of OCT Enface and OCTA Enface, and contrast agent injection volume and time from initial injection in the neck of the U-Net input.

[0045] The neural network in this design is similar to the standard U-Net encoder-decoder structure. It has three input paths: the first is the 3D volumetric data from OCT / OCTA, processed using several 3D convolutional layers, pooling layers, normalization layers, and activation layers. The second is the 2D image data projected onto the OCT and OCTA Enfaces, processed using several 2D convolutional layers, pooling layers, normalization layers, and activation layers. These two encoders can also contain attention modules, such as channel attention, spatial attention, or self-attention modules. The third input path is the Q,t information input to the U-Net neckline.

[0046] See Figure 2 Specifically, the preprocessing in step S4 includes:

[0047] S41: Input the 3D volume data to obtain the N*C*H*W*D feature map;

[0048] S42: Compress the original feature map into an N*C'*H*W target feature map by using 1x1 convolution in the depth direction or average pooling or max pooling in the depth direction.

[0049] S43: After unifying the number of channels by performing a 1x1 convolution between the target feature map and the feature map of the Enface image, add them together for fusion processing.

[0050] See Figure 2 Specifically, the preprocessing in step S4 includes:

[0051] S44: The features fused in step S43 are processed by multiple two-dimensional convolutional layers, pooling layers, and normalization layers to obtain the feature map of the neck.

[0052] S45: Integrate information on injection time and contrast agent injection volume into the feature map of the U-Net neck;

[0053] S46: The injection time and contrast agent injection volume are passed through a Linear layer and projected into a feature map with the same width and height as the feature map. This feature map is then fused with the feature map of the neck through channel stitching.

[0054] See Figure 3 Specifically, the fused features are processed through multiple 2D convolutional layers, pooling layers, and normalization layers to obtain the neck feature map. The U-Net neck feature map needs to incorporate injection time and dosage information. Time t and dosage Q are passed through a Linear layer and projected into a feature map with the same width and height as the original feature map. This feature map is then fused with the neck feature map using channel concatenation. Alternatively, t and Q can be directly expanded into a map with the same width and height as the original feature map, and then fused with either t or Q, and finally fused with the original neck feature map.

[0055] Specifically, the feature map after fusing time and dose information is input into the Decoder, and upsampling is performed step by step to restore the original image resolution. After each upsampling of the feature map, it is fused with the Enface feature map in the Encoder. The fusion method can be channel concatenation, or the two feature maps can be combined by performing a 1x1 convolution to unify the number of channels before adding them. After n upsampling operations to restore the original resolution, the final output is an angiography image of the same size as the input enface image.

[0056] Specifically, the reconstruction loss function used in training is either the L1 / L2 function or the L1 Smooth function. Additionally, a temporal consistency loss is added. Because the time interval between two adjacent FFA frames is relatively short, the images will appear very similar. Therefore, the temporal consistency loss measures the image difference between the network output FFA of adjacent frames. For example, the L1 / L2 / L1 Smooth loss function constrains the formation of abrupt changes in the image between two frames. By inputting OCT / OCTA data into the network along with the dose Q and the time intervals t,t-1 or t,t+1, the L1 / L2 / L1 Smooth loss function is calculated between the two outputs, multiplied by a certain coefficient, and then added to the reconstruction loss function.

[0057] This method can generate realistic FFA images based on OCT and OCTA images through neural network conversion, thereby achieving enhanced angiographic imaging of the patient's fundus vessels in a non-invasive manner.

[0058] While the disclosure is as stated above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of this disclosure, and all such changes and modifications will fall within the protection scope of this invention.

Claims

1. A method of converting FFA angiogram images based on deep learning OCT and OCTA imaging, characterized by, The method comprises the steps of: S1: obtaining OCT images and OCTA images of a target position of a patient's eye; S2: layering the OCT B-Scan to obtain OCT Enface images and OCTA Enface images of each layer; S3: obtaining FFA contrast of the target position of the patient's eye, registering any one frame of the OCT Enface images and the OCTA Enface images with the FFA contrast, and recording the injection amount of the contrast agent and the time from the initial injection; S4: preprocessing the input path of the neural network to obtain a feature map, and performing feature fusion on the feature map; The input path in step S4 comprises: OCT images and OCTA images of the target position, OCT Enface images and OCTA Enface images, and injection amount of the contrast agent and time from the initial injection at the neck of the U-Net; S5: inputting the fused feature map into the Decoder, performing upsampling step by step, and after upsampling of the feature map, fusing the feature map with the feature map of the OCT Enface images and the OCTA Enface images in the Encoder through channel splicing, and after multiple upsampling to restore the original resolution, outputting the FFA contrast image with the same size as the input OCT Enface images and OCTA Enface images.

2. The method of claim 1, wherein the deep learning-based OCT and OCTA imaging converts FFA angiogram images. The registration method in step S3 comprises rigid registration and non-rigid registration, and a training pair is formed through registration. Wherein, OCT and OCTA are inputs, and FFA contrast is output.

3. The deep learning-based OCT and OCTA imaging converted FFA angiogram image generating method of claim 1, wherein, In step S3, a plurality of frames of FFA contrast are included to form a plurality of frame groups of training pairs, and each training pair corresponds to different times.

4. The deep learning-based OCT and OCTA imaging converted FFA angiogram image generating method of claim 1, wherein, The preprocessing in step S4 comprises: S41: inputting the OCT images and the OCTA images of the target position to obtain a feature map of N, C, H, W, and D; S42: compressing the original feature map into a target feature map of N, C', H, and W through average pooling or maximum pooling in the depth direction; S43: adding and fusing the target feature map and the feature map of the Enface image after 1x1 convolution to unify the channel number.

5. The method of claim 4, wherein the deep learning-based OCT and OCTA imaging converts FFA angiogram images, and The preprocessing in step S4 comprises: S44: performing multiple two-dimensional convolution layers, pooling layers, and normalization layers on the fused feature of step S43 to obtain a feature map of the neck; S45: fusing the injection time and the injection amount of the contrast agent in the feature map of the U-Net neck; S46: projecting the injection time and the injection amount of the contrast agent into a feature map with the same width and height as the feature map through a Linear layer, and fusing the feature map with the feature map of the neck through channel splicing.

Images