Macular edema diagnosis system, method and device based on color fundus photography, medium and program product

Abstract

According to the macular oedema diagnosis system, method and device based on color fundus photography, the medium and the program product, the bright part enhancement and dark part enhancement of the CFP image are combined, the problem that diagnosis is difficult due to the fact that macular oedema features are not obvious in traditional CFP image diagnosis is solved, and the diagnosis accuracy and efficiency are greatly improved. Experimental results show that the accuracy rate of the method can reach 95% or above on a private data set, and the accuracy rate of multi-stage diagnosis of macular edema on a public macular degeneration and macular edema combined data set IDRID can reach 85%. Besides, a Mama layer in an SSM (Spatial-Sense-Modulation) framework is adopted to perform global feature modeling, so that the capability of capturing the global relationship of the image is further improved, the bottleneck problem encountered in actual clinical application due to the fact that a Transform model depends on computing resources is reduced, the requirement for high-performance computing equipment is reduced, and the method has the advantages of being high in practicability and easy to popularize. And possibility is provided for application in areas with limited resources.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a macular edema diagnosis system, method, device, medium and program product based on color fundus photography. Background Art

[0002] Macular edema is a common complication of many ophthalmic diseases, such as diabetic retinopathy and retinal vein occlusion, which may lead to severe vision impairment or even blindness. Timely detection and treatment of macular edema is essential to protect patients' vision. Currently, optical coherence tomography (OCT) is considered the gold standard for diagnosing macular edema because it can provide high-resolution cross-sectional images of the retina, from which the edema area can be clearly identified. However, OCT equipment is expensive and complicated to operate, making it difficult to be widely popularized in remote areas and areas with limited resources.

[0003] In contrast, color fundus photography (CFP) is a more economical and easy-to-use fundus imaging method. Its equipment cost is low and its operation is simple. It has been used in many primary medical institutions and remote areas. However, the characteristics of macular edema in CFP images are not as intuitive as those in OCT images. Traditional manual reading methods are difficult to identify early macular edema in a timely and accurate manner.

[0004] At present, most of the existing deep learning-based solutions focus more on ophthalmic diseases such as macular degeneration and glaucoma that are easier to detect in CFP modality, while there are relatively few studies on macular edema. The features of macular edema in CFP images are complex and difficult to identify, which increases the difficulty of diagnosis. Therefore, the present invention urgently needs a method that can solve the problem of identifying macular edema in CFP images. Summary of the invention

[0005] In view of the above-mentioned shortcomings of the prior art, an object of the present invention is to provide a macular edema diagnosis system, method, device, medium and program product based on color fundus photography, which is used to solve the problem of identifying macular edema on CFP images.

[0006] To achieve the above-mentioned purpose and other related purposes, the first aspect of the present invention provides a macular edema diagnosis system based on color fundus photography, comprising: an image preprocessing module, used to perform dark and bright enhancement processing on the original CFP image; the bright enhancement processing is used to enhance the part of the retinal blood vessels in the image, and the dark enhancement processing is used to enhance the edge features of the macular area in the image; a feature extraction module, used to perform feature extraction on the enhanced dark and bright images respectively based on a convolutional neural network; a multi-branch feature fusion module, used to downsample the original CFP image to the same size as the bright and dark feature matrices using deep separable convolution, and input the downsampled original CFP image, bright feature image, and dark feature image into a cascade of multiple adaptive multi-branch feature fusion modules for feature fusion; a global modeling module, used to perform global image modeling based on the fused features; a classification module, used to input the finally fused features into a pooling layer and a fully connected layer for classification according to the image global modeling results, and determine whether there is macular edema in the CFP image and grade it.

[0007] In some embodiments of the first aspect of the present invention, the feature extraction module extracts features of the bright image and the dark image respectively based on different strategies, including: using residual blocks to perform multi-scale deep feature extraction on the bright image; and using a super-dimensional block processing module to perform block processing and high-dimensional mapping on the dark image, and extracting dark edge information through an activation function.

[0008] In some embodiments of the first aspect of the present invention, the process of block processing and high-dimensional mapping by the super-dimensional block processing module includes: using a block encoder to divide the dark image into multiple image blocks that match the corresponding bright image feature scale; for each image block of the dark image, projecting its pixel information to a single high-dimensional pixel through high-dimensional mapping and performing a convolution operation to extract features in the image block; performing nonlinear activation function processing on the extracted feature information, and selecting the feature layer with the largest activation value as the representative feature of the image block; combining the representative features of each image block to form a dark feature image; and the final output feature map contains enhanced dark edge information.

[0009] In some embodiments of the first aspect of the present invention, the processing process of the multi-branch feature fusion module includes: selecting twice the highest dimension branch from the downsampled original CFP image, bright feature image, and dark feature image as the fusion dimension, and upgrading the feature dimensions of these three branches to the fusion dimension through a learnable convolutional layer, and then inputting them into the first adaptive multi-branch feature fusion module for feature fusion processing, and the output fusion features are downsampled to the buffer block as the pre-fusion features of the next adaptive multi-branch feature fusion module; the next adaptive multi-branch feature fusion module continues to perform feature fusion processing with the pre-fusion features, the further downsampled bright feature image and the dark feature image as three branches; repeating the above steps until the last adaptive multi-branch feature fusion module in the cascade structure completes the fusion processing.

[0010] In some embodiments of the first aspect of the present invention, the adaptive multi-branch feature fusion module includes: three convolution layer modules, a feature splicing module, a batch normalization module, an activation function module, and an output module; the original CFP image, the bright feature image, and the dark feature image are respectively input into the three convolution layer modules for convolution processing; the convolution processing results are all output to the feature splicing module for splicing; the spliced ​​features are input into the batch normalization module after convolution processing; the output results of the batch normalization module are input into the activation function module for nonlinear activation processing, and the activated features are output by the output module.

[0011] In some embodiments of the first aspect of the present invention, the global modeling module flattens the fused feature matrix into a one-dimensional vector and inputs it into multiple Mamba layers based on the state-space model; the Mamba layer is a deep learning module for global modeling of the image.

[0012] To achieve the above-mentioned purpose and other related purposes, the second aspect of the present invention provides a method for determining macular edema based on color fundus photography, comprising: performing dark and bright enhancement processing on the original CFP image; the bright enhancement processing is used to enhance the retinal blood vessel part in the image, and the dark enhancement processing is used to enhance the edge features of the macular area in the image; based on a convolutional neural network, feature extraction is performed on the enhanced dark and bright images respectively; the original CFP image is downsampled to the same size as the bright and dark feature matrices using a deep separable convolution, and the downsampled original CFP image, the bright feature image, and the dark feature image are input into a cascade of multiple adaptive multi-branch feature fusion modules for feature fusion; based on the fused features, global image modeling is performed to perform global feature enhancement, and the globally enhanced features are input into a pooling layer and a fully connected layer for classification, so as to determine whether there is macular edema in the CFP image and to grade it.

[0013] To achieve the above-mentioned object and other related objects, the third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for determining macular edema based on color fundus photography.

[0014] To achieve the above-mentioned purpose and other related purposes, the fourth aspect of the present invention provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer implements the method for determining macular edema based on color fundus photography.

[0015] To achieve the above-mentioned purpose and other related purposes, the fifth aspect of the present invention provides a computer device, including a memory, a processor and a computer program stored in the memory; the processor executes the computer program to implement the macular edema determination method based on color fundus photography.

[0016] As described above, the macular edema diagnosis system, method, device, medium and program product based on color fundus photography of the present invention have the following beneficial effects:

[0017] (1) The present invention solves the problem of diagnostic difficulty caused by the unclear characteristics of macular edema in traditional CFP image diagnosis by combining the enhancement of the bright and dark parts of the CFP image, and greatly improves the accuracy and efficiency of diagnosis. Experimental results show that this method can achieve an accuracy of more than 95% on private datasets, and an accuracy of 85% on multi-level diagnosis of macular edema on the public macular lesion and macular edema joint dataset IDRID.

[0018] (2) The present invention uses the Mamba layer in the SSM (Spatial-Semantic-Modulation) framework for global feature modeling, which further improves the ability to capture global image relationships, reduces the bottleneck problem encountered by the Transformer model in actual clinical applications due to its dependence on computing resources, reduces the demand for high-performance computing equipment, and provides the possibility for application in resource-limited areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Shown is a schematic structural diagram of a macular edema diagnosis system based on color fundus photography in one embodiment of the present invention.

[0020] Figure 2 Shown is a schematic diagram of the comparison of images before and after image enhancement in one embodiment of the present invention.

[0021] Figure 3It is a schematic diagram showing a process of enhancing the dark and bright parts of an original CFP image by an image preprocessing module in one embodiment of the present invention.

[0022] Figure 4 Shown is a schematic diagram of a framework structure of a macular edema diagnostic system according to an embodiment of the present invention.

[0023] Figure 5 Shown is another schematic diagram of the framework structure of a macular edema diagnostic system according to an embodiment of the present invention.

[0024] Figure 6 Shown is a schematic diagram of the performance results of a macular edema diagnostic system on a public dataset and a private dataset according to an embodiment of the present invention.

[0025] Figure 7 Shown is a flow chart of a method for determining macular edema based on color fundus photography in one embodiment of the present invention.

[0026] Figure 8 Shown is a schematic diagram of the structure of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0027] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0028] Before further describing the present invention in detail, the nouns and terms involved in the embodiments of the present invention are explained. The nouns and terms involved in the embodiments of the present invention are applicable to the following interpretations:

[0029] <1> Optical Coherence Tomography (OCT) is a non-invasive imaging technology used to obtain high-resolution cross-sectional images of the internal structure of biological tissues. OCT technology is based on light emitted by low-coherence light sources. When this light hits the tissue, part of the light is reflected back by the tissue surface and internal structure. By measuring the coherence between the reflected light and the reference light, an image of the internal structure of the tissue can be reconstructed. OCT is particularly important in the field of ophthalmology because it can provide detailed images of the retina, helping doctors diagnose and monitor retinal diseases such as macular degeneration and diabetic retinopathy.

[0030] <2> Color Fundus Photography (CFP) is an ophthalmic examination technology used to take color photos of the back of the eye (including the retina, optic nerve head and macular area). Through this technology, doctors can observe the blood vessels, nerves and other structures of the fundus to evaluate eye health and diagnose various fundus diseases. Color fundus photography is a fast and painless examination method, commonly used for screening and monitoring of fundus diseases such as diabetic retinopathy, glaucoma, and hypertensive retinopathy.

[0031] <3> The state space model (SSM) is a mathematical model used to describe the state representation of a sequence at each time step and predict its next state based on the input.

[0032] The present invention uses a large number of annotated CFP images to train the model, so that the model can automatically learn and extract features closely related to macular edema, thereby improving the diagnostic accuracy and efficiency of the disease. This method effectively makes up for the limitations of doctors in diagnosing macular edema based on CFP images, so that in the absence of OCT equipment, it can still provide reliable diagnostic support for macular edema. The present invention not only overcomes the shortcomings of expensive and difficult to popularize OCT equipment, but also provides a practical macular edema detection solution for remote and resource-poor areas, significantly improves the detection rate of the disease, and helps to provide targeted diagnosis and treatment plans to prevent further deterioration of patients' vision.

[0033] Figure 1 The schematic diagram of the structure of a macular edema diagnosis system based on color fundus photography in an embodiment of the present invention is shown. The technical solution of the present invention combines brightness and dark image enhancement, convolutional neural network (CNN), state space model (SSM) and feature fusion technology to improve the detection accuracy and efficiency of macular edema. Specifically, the light and dark contrast characteristics of macular edema in CFP images are utilized to enhance the visibility of vascular structures and edge details in CFP images. Then, the bright and dark images are extracted by convolutional neural network (CNN), and multi-scale features are combined to capture the characteristics of lesions at different scales. Finally, the adaptive multi-branch feature fusion (AMFS) module is used to fuse features from different branches, and global modeling is performed through the state space model (SSM), thereby improving the accuracy and robustness of diagnosis.

[0034] The macular edema diagnosis system based on color fundus photography in this embodiment mainly includes: an image preprocessing module 101, a feature extraction module 102, a multi-branch feature fusion module 103, a global modeling module 104, and a classification module 105.

[0035] In the embodiment of the present application, the image preprocessing module 101 is used to perform dark and bright enhancement processing on the original CFP image; the bright enhancement processing is used to enhance the retinal blood vessels in the image, and the dark enhancement processing is used to enhance the edge features of the macular area in the image. The images after bright enhancement and dark enhancement are helpful to improve the effectiveness of feature extraction.

[0036] For ease of understanding, Figure 2 The diagram shows the comparison of images before and after enhancement. The image on the left is the original CFP image, the image in the middle is the bright part enhanced image, and the image on the right is the dark part enhanced image. It is worth noting that the area of ​​effusion caused by macular edema usually shows a light-dark contrast different from the surrounding normal tissue. However, due to factors such as imaging conditions, individual differences, and the degree of lesions, this difference may not be obvious. In order to enhance the brightness and color feature differences in the edema area, the CFP image is first enhanced in the dark and bright parts (for example, the enhancement coefficient can be set to 0.5) to improve the effectiveness of subsequent feature extraction and classification. For the enhanced bright image, the retinal vascular structure is clearer than the original image. The morphology, direction, and branching pattern of the blood vessels are of great significance in the diagnosis of macular edema, because edema may compress the blood vessels in the macular area, resulting in abnormal, distorted, and deformed vascular morphology.

[0037] In some examples, the image preprocessing module 101 performs dark and bright enhancement processing on the original CFP image as follows: Figure 3 As shown, including the following:

[0038] Step S3a: Calculate the brightness histogram of the original CFP image to analyze the brightness distribution of the image and determine the brightness and dark areas that need to be enhanced.

[0039] The process of calculating the brightness histogram is: first, traverse each pixel in the original CFP image; second, determine the brightness value of each pixel (for grayscale images, the brightness value is the pixel value; for color images, it can be obtained by converting to grayscale or calculating the brightness channel); finally, increase the count at the corresponding brightness level in the histogram. It should be understood that the brightness histogram is an important tool in image processing, which shows the number of pixels at each brightness level in the image, for example, for 8-bit images, the brightness level is usually from 0 (black) to 255 (white). The histogram provides a visual representation of the image brightness distribution and can help users understand the overall brightness characteristics of the image.

[0040] Generally speaking, the peak of the histogram indicates where the brightness of the image is mainly concentrated; the width of the histogram can show the distribution range of the brightness values, that is, whether the image contains all brightness levels from black to white; the shape of the histogram can be used to determine whether the brightness distribution is uniform or whether there are any missing brightness levels. Dark areas usually correspond to the left side of the histogram (low brightness values), and these areas may lose details due to underexposure. Bright areas usually correspond to the right side of the histogram (high brightness values), and these areas may lose details due to overexposure.

[0041] Preferably, before analyzing the brightness distribution of the image, the original CFP image is smoothed using a filter to reduce Gaussian noise in the image. The Gaussian filter reduces the high-frequency components in the image through smoothing, which are usually associated with noise. The Gaussian filter is based on the Gaussian distribution (normal distribution), which is a smoothing technique widely used in image processing. The core idea of ​​the Gaussian filter is to replace each pixel value in the image with the weighted average of its neighboring pixels, with the weight determined by the Gaussian function.

[0042] Step S3b: performing equalization processing on the brightness histogram of the dark area to increase the contrast of the dark area, applying a preset low gamma value to the dark area to increase the brightness of the dark area, and enhancing local details based on an adaptive histogram equalization algorithm.

[0043] Histogram equalization is a method to improve the global contrast of an image. It redistributes the histogram of the image so that the histogram distribution of the output image is more uniform, thereby enhancing the contrast of the image. For dark areas, histogram equalization can increase the contrast of these areas, making the originally dark areas clearer. The process of histogram equalization includes: calculating the histogram of the original image; calculating the cumulative distribution function; applying the equalization transformation function to map the original pixel values ​​to new pixel values; and applying the transformed pixel values ​​to generate the enhanced image.

[0044] Gamma correction is a nonlinear operation used to adjust the brightness and contrast of an image. When the gamma value is less than 1, the image becomes brighter because the pixel values ​​in the dark area increase more, while the pixel values ​​in the bright area increase less. The specific implementation steps include: selecting a gamma value (for dark enhancement, a value less than 1 is usually selected, such as 0.5); applying the gamma correction formula (O = I γ , I represents the input pixel value, γ represents the gamma value, and O represents the output pixel value); the corrected pixel value is mapped to the valid pixel range (0 to 255).

[0045] Adaptive histogram equalization (AHE) is a computer image processing technique used to enhance the local contrast of an image. AHE changes the image contrast by calculating the local histogram of the image and then redistributing the brightness, thereby improving the local contrast of the image and obtaining more image details.

[0046] Step S3c: performing equalization processing on the brightness histogram of the bright area to increase the contrast of the bright area, applying a preset high gamma value to the bright area to reduce the low brightness of the bright area, and enhancing local details based on an adaptive histogram equalization algorithm.

[0047] For bright areas, histogram equalization can make the originally concentrated pixel values ​​more evenly distributed to the range of all possible values, thereby increasing the contrast and details of the bright areas. The implementation process includes: first, calculating the brightness histogram of the bright area. Then, applying the mapping function of the histogram equalization, the pixel values ​​of the bright area are mapped to a wider brightness range. Finally, an enhanced image is generated, in which the contrast of the bright area is improved. For the bright area, select a gamma value (for bright enhancement, usually a value greater than 1, such as 1.5); apply the gamma correction formula (O=I γ , I represents the input pixel value, γ represents the gamma value, and O represents the output pixel value); the corrected pixel value is mapped to the valid pixel range (0 to 255).

[0048] In an embodiment of the present application, the feature extraction module 102 is used to extract features of the enhanced dark image and the bright image respectively based on a convolutional neural network.

[0049] Preferably, the convolutional neural network extracts features of bright images and dark images respectively based on different strategies, including: using a residual block to perform multi-scale deep feature extraction on the bright image; and using a hyper dimensional block processing (HDBP) module to perform block processing and high-dimensional mapping on the dark image, and extracting dark edge information through an activation function.

[0050] It is worth noting that for the enhanced bright image, the retinal vascular structure is clearer than the original image. The morphology, direction and branching pattern of blood vessels are of great significance in the diagnosis of macular edema, because edema may compress the blood vessels in the macular area, resulting in abnormal, twisted and deformed vascular morphology. In order to effectively detect these potential changes, a convolutional neural network is used to perform deep feature extraction on the bright image. When the feature map size is one-quarter, one-eighth and one-sixteenth of the original image, the corresponding feature maps are extracted respectively. This multi-scale feature extraction method helps to capture vascular features at different scales and enhance the model's sensitivity to tiny vascular deformations.

[0051] The residual block is an important structure in convolutional neural networks. It solves the gradient vanishing problem in deep network training by introducing skip connections. The residual block can adaptively detect image features of different scales by using convolution kernels of different sizes, thereby capturing local details and global context information in the image. Skip connections are applied between features of different scales to achieve the sharing and reuse of feature information, make full use of the local features of the image, and enhance the network's ability to capture details. Exemplarily, the process of multi-scale feature extraction of bright images by the residual block includes:

[0052] (1) Within the residual block, convolution kernels of different sizes are used to perform convolution operations on the input features to extract features of different scales.

[0053] (2) Apply the ReLU activation function to the features after the convolution operation to enhance the nonlinear expression ability of the network.

[0054] (3) Features of different scales are fused through jump connections by weighted summation or concatenation.

[0055] (4) Use a 1×1 convolution block to process the features of the input residual block so that its number of channels and size are consistent with the output features of the residual block.

[0056] (5) Add the output features of the residual block to the input features processed in step (4) to form a residual connection, and then output it to the next residual block or other parts of the network.

[0057] (6) By stacking multiple residual blocks, a deeper network can be constructed to further improve the feature extraction capability.

[0058] (7) Finally, the multi-scale features output by the network can be used for subsequent image processing tasks such as image classification, object detection, or image segmentation.

[0059] The Hyper Dimensional Block Processor (HDBP) module is a technical solution specially designed to process dark images and extract edge feature information. The image after dark enhancement highlights the edge and detail features of the macular area, but may introduce a lot of invalid information, such as background noise and interference from non-lesion areas, so it is processed by the HDBP module. Exemplarily, the process of block processing and high-dimensional mapping by the HDBP module includes:

[0060] (1) Use a block encoder to segment the dark image into multiple image blocks that match the feature scale of the corresponding bright image to ensure size consistency in the subsequent fusion process.

[0061] (2) For each image block of the dark image, its pixel information is projected to a single high-dimensional pixel through high-dimensional mapping and a convolution operation is performed to extract the features in the image block. For example, each image block is projected to a 1×1 high-dimensional space pixel through high-dimensional mapping, and then a 1×1×3 channel convolution operation is performed on the pixel to integrate the feature vectors of the small block.

[0062] (3) The extracted feature information is processed by a nonlinear activation function (such as a ReLU function), and the feature layer with the largest activation value is selected as the representative feature of the image block.

[0063] (4) The representative features of each image block are combined to form a dark feature image. The final output feature map contains enhanced dark edge information and can be used for subsequent image processing tasks, such as image segmentation or recognition. This dark image processing method helps to highlight important edge information, suppress irrelevant noise, and improve the accuracy of feature representation.

[0064] In an embodiment of the present application, the multi-branch feature fusion module 103 is used to downsample the original CFP image to the same size as the bright and dark feature matrices using deep separable convolution, and input the downsampled original CFP image, bright feature image, and dark feature image into a cascade of multiple adaptive multi-branch feature fusion modules for feature fusion.

[0065] Specifically, twice the highest dimension branch is selected from the downsampled original CFP image, bright feature image, and dark feature image as the fusion dimension. The feature dimensions of these three branches are upgraded to the fusion dimension through a learnable convolutional layer and then input into the first AMFS module for feature fusion processing. The output fusion feature is downsampled to the buffer block (Buffer Block) to serve as the pre-fusion feature of the next AMFS module; the next AMFS module continues to perform feature fusion processing with the pre-fusion feature, the further downsampled bright feature image, and the dark feature image as three branches. This cycle is repeated until the last AMFS module in the cascade structure completes feature fusion. The feature matrix finally fused mainly integrates the local features of the original image and can effectively capture subtle lesions.

[0066] In the process of fusing dark features, bright features and the original image, in order to ensure the stability of the gradient and speed up the convergence of the model, the original CFP image is first downsampled to the same size as the brightness and dark feature matrices through depthwise separable convolution. Depthwise separable convolution is an efficient convolution operation that decomposes the standard convolution operation into two smaller operations: depthwise convolution and pointwise convolution. This decomposition can significantly reduce the number of parameters and computations of the model while maintaining similar or better performance. In the depthwise convolution stage, a convolution kernel is applied independently to each input channel, which means that if there are M input channels, there will be M convolution kernels, and each convolution kernel only operates on one corresponding channel. In this way, depthwise convolution can extract spatial features but does not combine information from different channels. Next, in the pointwise convolution stage, the output channels of the depthwise convolution are combined using a 1x1 convolution kernel to produce the final feature map. This step can be regarded as feature fusion in the channel dimension, which can extract channel features.

[0067] The structure of the AMFS module is shown in the figure, which includes three convolutional layer modules (Conv), a feature splicing module (Concat), a batch normalization module (BN), an activation function module (Relu), and an output module (Out). The three input branches are represented by F1, F2, and F3. Each branch is processed by a series of convolutional layer modules, which are used to extract features. The processed three branch features are input into the feature splicing module for splicing. The spliced ​​features are extracted through another convolutional layer module. The output features of the convolutional layer module are input into the batch normalization module, which helps to speed up the training process and improve the stability of the model. The features output by the batch normalization module are input into the activation function module to increase the nonlinearity of the network. Finally, the features activated by the Relu function are used as the output of the module.

[0068] In an embodiment of the present application, the global modeling module 104 is used to perform global image modeling based on the fused features. Specifically, the fused feature matrix is ​​flattened into a one-dimensional vector and input into multiple Mamba layers based on the state space model; the Mamba layer is a deep learning module for global image modeling.

[0069] It is worth noting that, in the process of feature fusion, the lack of global context association may limit the model's understanding of the complex relationship between global features. To solve the above problem, the global modeling module 104 flattens the fused feature matrix into a one-dimensional vector and inputs it into the Mamba layer based on the state space model (SSM). The Mamba layer is a deep learning module that efficiently models the global relationship of sequence data. By establishing dependencies between long-distance parameters in the vector, the model's ability to understand global features is enhanced. After being processed by multiple Mamba layers, the model can capture the complex association between each parameter in the feature vector and the global, and achieve effective modeling of the global information of the image.

[0070] In an embodiment of the present application, the classification module 105 is used to receive the output result of the global modeling module 104, input the finally fused features into the pooling layer and the fully connected layer for classification, and determine whether macular edema exists in the CFP image and grade it.

[0071] Combination Figure 4 and Figure 5 The framework structure of the technical solution of this application is described as follows:

[0072] Before proceeding, it is necessary to explain the symbols in the figure. Bright represents the bright part enhanced image, I DarkRepresents dark enhanced image, Bright Branch represents bright image branch, Dark Branch represents dark image branch. HDBP (Hyper Dimensional Block Processor) Module represents hyperdimensional block processing module, AMFS (Adaptive Multi-Branch Feature Synthesis) Module represents adaptive multi-branch feature fusion module, Buffer Block (B) represents buffer block. Feature Input represents feature input, Residual Block represents residual block, Patch Embedding represents block encoding, Pixelwise Activation represents pixel-wise activation, Depthwise Separable Convolution represents depthwise separable convolution. F1, F2, F3 represent three branches, Conv Layers represents convolution layer, Concat represents feature concatenation, Cov represents convolution, BN represents batch normalization layer, Relu represents activation function, and Out represents output.

[0073] First, the original CFP image is enhanced in the dark and bright parts to obtain the bright part enhanced image I Bright and dark enhanced image I Dark .

[0074] Secondly, the enhanced bright image is feature extracted based on the convolutional neural network. In the bright image branch, the residual block is used to extract multi-scale deep features of the bright image. The sizes of the extracted feature images are 4×256×256, 8×128×128, 16×128×128, 32×64×64, and 128×32×32.

[0075] Again, in the dark branch, the dark image is cut by the patch embedding to obtain 128×128 small blocks, 64×64 medium blocks, and 32×32 large blocks. The Hyper Dimensional Block Processor (HDBP) module is used to perform block processing and high-dimensional mapping on the dark image. The HDBP module projects the pixels of each image block to a 1×1 high-dimensional space pixel through high-dimensional mapping, and then performs a 1×1×3 convolution operation on the pixel, and then performs a Relu function activation process, selects the feature layer with the largest activation value as the representative feature of the image block, and combines the representative features of each image block to form a dark feature image.

[0076] Next, each of the multiple cascaded AMFS modules has three branch inputs. For the first AMFS module, the first branch input is the original CFP image, which is converted into a buffer block of 3×128×128 by depthwise separable convolution. The second branch input is the bright feature image of 8×128×128, and the third branch input is the dark feature image of 1×128×128. The fused feature image output by the first AMFS module is converted into a buffer block of 16×64×64 by depthwise separable convolution, which is used as the first branch input of the second AMFS module. The second branch input of the second AMFS module is the bright feature image of 32×64×64, and the third branch input is 1×64×64. The fused features output by the second AMFS module continue to be converted into a buffer block of 64×32×32 by depthwise separable convolution, which is used as the first branch input of the third AMFS module, and so on, until the last AMFS module in the cascade completes the fusion process. Among them, the AMFS module includes three convolutional layer modules (Conv), which input F1, F2, and F3 respectively. The output results of the three convolutional layer modules (Conv) are input into the feature splicing module (Concat). After splicing, convolution processing is performed, and then the convolution results are input into the batch normalization module (BN), and finally the Relu function is activated and output.

[0077] Finally, the fused feature matrix is ​​flattened into a one-dimensional vector and input into multiple Mamba layers based on the state-space model for global image modeling. The final fused features are then input into the pooling layer and the fully connected layer for classification to determine whether there is macular edema in the CFP image and to grade it.

[0078] In the embodiments of the present application, after constructing a macular edema diagnosis system based on color fundus photography, a data set formed by a large number of annotated CFP images is used for model training, so that the model can automatically learn and extract features closely related to macular edema, thereby improving the diagnostic accuracy and efficiency of the disease. For example, the performance of the macular edema diagnosis system based on color fundus photography provided by the present application on public and private data sets is as follows: Figure 6 As shown: On the IDRID public dataset, the validation set accuracy is 0.850; on a private dataset of a hospital, the validation set accuracy is 0.968.

[0079] It should be understood that the division of modules in the embodiments of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present invention may be integrated into a processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0080] Figure 7 The flowchart of a method for determining macular edema based on color fundus photography in an embodiment of the present invention is shown. It should be noted that the determination method in the embodiment of the present invention is applied to the macular edema diagnosis system in the above embodiment, and the method includes the following steps:

[0081] Step S71: performing dark and bright enhancement processing on the original CFP image; the bright enhancement processing is used to enhance the retinal blood vessels in the image, and the dark enhancement processing is used to enhance the edge features of the macular area in the image.

[0082] Step S72: performing feature extraction on the enhanced dark and bright images respectively based on a convolutional neural network.

[0083] Step S73: Use depthwise separable convolution to downsample the original CFP image to the same size as the bright and dark feature matrices, and input the downsampled original CFP image, bright feature image, and dark feature image into a cascade of multiple adaptive multi-branch feature fusion modules for feature fusion.

[0084] Step S74: Perform global image modeling based on the fused features to perform global feature enhancement, input the globally enhanced features into the pooling layer and the fully connected layer for classification, determine whether there is macular edema in the CFP image and grade it.

[0085] It should be noted that the implementation process and principle of the macular edema determination method based on color fundus photography in the embodiment of the present invention are similar to the macular edema diagnosis system based on color fundus photography mentioned above, and will not be repeated here.

[0086] It should be noted that, in the embodiments of the present invention, words such as "exemplary" or "for example" represent examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present invention should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0087] In the embodiments of the present invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B may be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c may represent: a, b, c, ab, ac, bc or abc, where a, b, c may be single or plural.

[0088] Figure 8 is a schematic block diagram of a computer device / equipment / system provided by an embodiment of the present invention. Figure 8 As shown in FIG. 8 , the computer device includes: at least one processor 801, a memory 802, at least one network interface 803, and a user interface 805. The various components in the device are coupled together via a bus system 804. It can be understood that the bus system 804 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 804 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 804 is not described in detail. Figure 8 In the specification, various buses are labeled as bus systems.

[0089] The user interface 805 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0090] It is understood that the memory 802 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.

[0091] The memory 802 in the embodiment of the present invention is used to store various categories of data to support the operation of the electronic terminal 800. Examples of these data include: any executable program used to operate on the electronic terminal 800, such as an operating system 8021 and an application 8022; the operating system 8021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 8022 can include various applications, such as a media player (Media Player), a browser (Browser), etc., for implementing various application services. The XX method provided by the embodiment of the present invention can be included in the application 8022.

[0092] The method disclosed in the above embodiment of the present invention can be applied to the processor 801, or implemented by the processor 801. The processor 801 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 801 or the instruction in the form of software. The above processor 801 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 801 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The general processor 801 can be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0093] In an exemplary embodiment, the electronic terminal 800 may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD) to execute the aforementioned method.

[0094] According to the method provided by an embodiment of the present invention, the present invention also provides a computer program product, which includes: computer program code, when the computer program code is run on a computer, the computer executes a macular edema determination method based on color fundus photography.

[0095] According to the method provided by an embodiment of the present invention, the present invention also provides a computer-readable storage medium, which stores a program code. When the program code runs on a computer, the computer executes the above method.

[0096] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program and / or a computer. By way of illustration, both applications running on a computing device and a computing device can be components. One or more components may reside in a process and / or an execution thread, and a component may be located on a computer and / or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. Components may, for example, communicate through local and / or remote processes according to signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system and / or a network, such as the Internet interacting with other systems through signals).

[0097] Those skilled in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0098] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0099] In the several embodiments provided by the present invention, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0100] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0101] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0102] In the above embodiments, the functions of each functional unit can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions (programs). When the computer program instructions (programs) are loaded and executed on a computer, the process or function according to the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrations. Available media may be magnetic media (e.g., floppy disks, hard disks, tapes), optical media (e.g., high-density digital video discs (DVDs), or semiconductor media (e.g., solid state disks (SSDs), etc.).

[0103] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store program codes.

[0104] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0105] In summary, the present invention provides a macular edema diagnosis system, method, device, medium and program product based on color fundus photography. The present invention solves the problem of diagnostic difficulty caused by the unclear characteristics of macular edema in traditional CFP image diagnosis by combining the enhancement of the bright and dark parts of the CFP image, and greatly improves the accuracy and efficiency of diagnosis. Experimental results show that the method can achieve an accuracy of more than 95% on a private data set, and can achieve an accuracy of 85% for multi-level diagnosis of macular edema on a public macular lesion and macular edema joint data set IDRID. The present invention uses the Mamba layer in the SSM (Spatial-Semantic-Modulation) framework for global feature modeling, further improves the ability to capture global image relationships, reduces the bottleneck problem encountered by the Transformer model in actual clinical applications due to its dependence on computing resources, reduces the demand for high-performance computing equipment, and provides possibilities for application in resource-limited areas. Therefore, the present invention effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.

[0106] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A macular edema diagnosis system based on color fundus photography, characterized in that: include: An image preprocessing module, used for performing dark and bright enhancement processing on the original CFP image; the bright enhancement processing is used for enhancing the retinal blood vessels in the image, and the dark enhancement processing is used for enhancing the edge features of the macular area in the image; A feature extraction module is used to extract features of the enhanced dark and bright images respectively based on a convolutional neural network; A multi-branch feature fusion module is used to downsample the original CFP image to the same size as the bright and dark feature matrices using a deep separable convolution, and input the downsampled original CFP image, bright feature image, and dark feature image into a cascade of multiple adaptive multi-branch feature fusion modules for feature fusion; The global modeling module is used to perform global image modeling based on the fused features; The classification module is used to input the final fused features into the pooling layer and the fully connected layer for classification according to the global image modeling results, to determine whether there is macular edema in the CFP image and to grade it.

2. The macular edema diagnosis system based on color fundus photography according to claim 1, characterized in that: The feature extraction module extracts the features of the bright image and the dark image respectively based on different strategies, including: using residual blocks to perform multi-scale deep feature extraction on the bright image; and using a super-dimensional block processing module to perform block processing and high-dimensional mapping on the dark image, and extracting dark edge information through an activation function.

3. The macular edema diagnosis system based on color fundus photography according to claim 2, characterized in that: The process of block processing and high-dimensional mapping by the super-dimensional block processing module includes: Using a block encoder to segment the dark image into a plurality of image blocks matching the feature scales of the corresponding bright image; For each image block of the dark image, its pixel information is projected to a single high-dimensional pixel through high-dimensional mapping and a convolution operation is performed to extract features in the image block; The extracted feature information is processed by a nonlinear activation function, and the feature layer with the largest activation value is selected as the representative feature of the image block; The representative features of each image block are combined to form a dark feature image; the final output feature map contains enhanced dark edge information.

4. The macular edema diagnosis system based on color fundus photography according to claim 1, characterized in that: The processing process of the multi-branch feature fusion module includes: Select twice the highest dimension branch from the downsampled original CFP image, bright feature image, and dark feature image as the fusion dimension, upgrade the feature dimensions of these three branches to the fusion dimension through a learnable convolutional layer, and then input them into the first adaptive multi-branch feature fusion module for feature fusion processing, and downsample the output fusion features to the buffer block as the pre-fusion features of the next adaptive multi-branch feature fusion module; The next adaptive multi-branch feature fusion module continues the feature fusion process with the pre-fusion feature, the bright feature image and the dark feature image after further downsampling as three branches; Repeat the above steps until the last adaptive multi-branch feature fusion module in the cascade structure completes the fusion process.

5. The macular edema diagnosis system based on color fundus photography according to claim 1 or 4, characterized in that: The adaptive multi-branch feature fusion module includes: Three convolutional layer modules, feature concatenation module, batch normalization module, activation function module, and output module; The original CFP image, the bright feature image and the dark feature image are respectively input into three convolution layer modules for convolution processing; the convolution processing results are all output into the feature stitching module for stitching; the spliced ​​features are input into the batch normalization module after convolution processing; the output result of the batch normalization module is input into the activation function module for nonlinear activation processing, and the activated features are output by the output module.

6. The macular edema diagnosis system based on color fundus photography according to claim 1, characterized in that: The global modeling module flattens the fused feature matrix into a one-dimensional vector and inputs it into multiple Mamba layers based on the state space model; the Mamba layer is a deep learning module for global modeling of the image.

7. A method for determining macular edema based on color fundus photography, characterized in that: Applied to the diagnostic system as claimed in claim 1, the method comprises: Performing dark and bright enhancement processing on the original CFP image; the bright enhancement processing is used to enhance the retinal blood vessels in the image, and the dark enhancement processing is used to enhance the edge features of the macular area in the image; Based on the convolutional neural network, the features of the enhanced dark and bright images are extracted respectively; The original CFP image is downsampled to the same size as the bright and dark feature matrices using depthwise separable convolution, and the downsampled original CFP image, bright feature image, and dark feature image are input into a cascade of multiple adaptive multi-branch feature fusion modules for feature fusion; Based on the fused features, global image modeling is performed to enhance the global features. The globally enhanced features are input into the pooling layer and the fully connected layer for classification to determine whether there is macular edema in the CFP image and to grade it.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for determining macular edema based on color fundus photography as described in claim 7 is implemented.

9. A computer program product, characterized in that The computer program product includes computer program codes, and when the computer program codes are executed on a computer, the computer is enabled to implement the method for determining macular edema based on color fundus photography as claimed in claim 7.

10. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the method for determining macular edema based on color fundus photography as described in claim 7.

Images